@rhertzog I recently looked into the postgresql-common package included in Debian / Kali and manually reproduced the setup steps currently performed by msfdb except using the postgresql-common tools. When creating a new cluster one must provide the PostgreSQL version. If more than one version is installed would you prefer to use the latest version?

In your preferred use case, how is Metasploit configured to use pre-existing PostgreSQL instance?

When creating a new cluster one must provide the PostgreSQL version. If more than one version is installed would you prefer to use the latest version?

I guess so. Usually when there are two versions, it's because an upgrade is needed/planned and we are better creating the metasploit cluster using the latest version. That said a point can be made that if the ugprade did not happen yet, we might be better using the default version (i.e. the one which runs on port 5432). The difference is rather minimal and both approaches are OK.

But somehow it depends on how you envision the use of the cluster... do you want a single metasploit cluster on the system shared by all users or do you want to setup a cluster for each metasploit user?
Right now the database configuration in Kali is stored system wide (/usr/share/metasploit-framework/config/database.yml) and thus shared by all users. But if we really want to support multiple users, the better design would be to have a dedicated database for each user but all managed in a shared cluster (instead of having multiple clusters).

In your preferred use case, how is Metasploit configured to use pre-existing PostgreSQL instance?

In Debian/Kali, if you install postgresql (which we do by way of dependencies) you can assume that there will be an instance configured to run on the default port 5432 and that it will be started with "systemctl start postgresql". From there on, our current msfdb just creates a dedicated user and database in that instance (by relying on the fact that the "postgres" unix user is the super user for that instance).

See https://git.kali.org/gitweb/?p=packages/metasploit-framework.git;a=blob;f=debian/extra/msfdb;h=a480a03faaf46d44f3787a4f1dccb61b8e50fa97;hb=refs/heads/kali/master for how we handled that.

I've documented my initial investigation for the steps necessary to get msfdb using the postgresql-common tools.

@mkienow-r7 @busterb What's the status of this? Is this something that you planned to work on or are you waiting on some external contribution?

I looked into your wiki page and I'm not sure that using the postgresql-common tools to install in a user-owned directory is a good idea. The tools are unlikely to avoid conflicts on port numbers between the system-wide clusters and the user-defined clusters...

I understand that you designed this new msfdb code that way due to the case where you bundle everything including postgresql. But the reality in Linux distributions is that postgresql is a system-wide service and we should be able to use the already available system-wide setup without too much fuss.

We don't care about creating new clusters, we care about creating a dedicated database in the currently running cluster and just using it. And possibly starting/stopping the postgresql service because it might not be running.

@rhertzog I appreciate your feedback. My hope with the user-defined cluster was to avoid conflicts with a system-wide cluster, and allow users without elevated privileges to perform operations necessary to get a database up and running. In your vision for msfdb using a system-wide cluster, would the end user running msfdb have privileges to create new databases in the existing cluster? Would the user have permissions to run pg_ctlcluster in order to start and stop it? Would you like the user to be able to pass the cluster to msfdb similar to postgresql-common's pg_wrapper --cluster option formatted as version/name?

What kind of conflict are you fearing? A cluster is just a place where you can host databases. Databases have names that uniquely identify them.

If you assume a system-wide cluster, then no the user doesn't have the right to create a database and he doesn't have the right to start/stop the service. In general the database should be created during the package installation procedure and the service should be running by default. That said "msfdb" could be smart and just rely on sudo when it wants to perform operations that the current user is not allowed to do.

I'm not opposed to "msfdb" having code/logic to handle setting up PostgreSQL in a user account but it should be able to also work with a (pre-existing) system-wide cluster. Either the database is already setup and the user passes only the credentials required to connect, or it's not and the script is able to create the postgresql user and the database.

Any progress on this issue ? Kali users are not benefiting from the latest features due to our inability to use your new msfdb. If you have no plans to fix this, then we should look into some alternate plan but right now we were waiting on a solution on your side.

My vote would be to simply adjust msfdb to only work when invoked as a user with sufficient privileges to start and create a database on Kali, and to simply use the native Debian tools if they are available. That should make it work very similar to how it does today on Kali and other environments.

While this approach doesn't bring the ability to run per-user databases or other more complex use cases, it's important to note that was never the explicit design goal of the omnibus packages we produce in the first place. Rather, the additional complexity of supporting Windows, macOS, and many different Linux flavors, not needing to create a dedicated postgresql user on every kind of OS and making sure it was secure was a design benefit. If postgresql ran as a root user natively, we might have allowed that path as well.

I'll take a look at moving this forward, thanks!

WIP branch for msfdb changes: https://github.com/busterb/metasploit-framework/tree/msfdb-improvements

I ran into a couple of issues related to how gems are updated in Kali specifically. It seems that ruby-librex is installed globally, which is not fully-hidden from how /usr/share/metasploit-framework/ruby loads its gemset. This then causes problems loading a few other dependencies.

It seems that these packages: beef-xss kali-linux-full kali-linux-large ruby-librex ruby-msfrpc-client need an update to riby-msfrpc-client since it actually does not need ruby-librex anymore at all. Could we ask that Kali update to the latest https://rubygems.org/gems/msfrpc-client and remove the dependency on the old global rex gem?

In the mean time, I can also add some workarounds like we do with other tools to try to fixup the gem search path in msfdb as well.

@busterb Sorry for the delay, I just uploaded an updated ruby-msfrpc-client in kali-dev. It should reach kali-rolling shortly. That said it will not remove ruby-librex from the system unless the user runs "apt autoremove"...

I made a quick try of your new msfdb and it looks promising but it doesn't seem to work for me yet.

root@kali:~# MSF_LOCAL_LIB=/usr/share/metasploit-framework/lib ./msfdb init
[+] Detected Debian-flavored PostgreSQL installation
[+] Starting database
[+] Creating database users
Password: 
Password: 
[+] Stopping database
[+] Starting database
[+] Creating initial database schema
[?] Initial MSF web service account username? [root]: 
[?] Initial MSF web service account password? (Leave blank for random password): 
[+] Generating SSL key and certificate for MSF web service
No MSF web service configuration found at /root/msf-ws.ru, not starting

It was not clear to me what I had to input on the password prompt... first time I entered two different passwords and it did not complain. Then I retried (used "reinit") with the same password and the password I saw in /usr/share/metasploit-framework/config/database.yml was not the one I typed.

Then I looked up the list of databases (with "su - postgres -c 'psql -l'") and I didn't see any "msf" database at all. Not sure why since I haven't seen any error message. And obviously msfconsole complained about the lack of the database.

[-] * WARNING: No database support: FATAL:  password authentication failed for user "msf"
FATAL:  password authentication failed for user "msf"

I'm happy to do more tests or provide supplementary defails if needed.

Did anything happen with this, it seems to have just died mid discussion. A pull of the latest master isn't working on a standard Kali installation so it doesn't seem to have been fixed.

@rhertzog msfdb shouldn't be prompting you for a password after the "Creating database users" step, so something is not working as expected. The script generates a database password and is using createdb to create the database with the generated password. Do you experience this same behavior with the msfdb on master?

@digininja Yes, it appears work on this issue hasn't had any recent progress. Hopefully the team will get back to this issue again soon.

@mkienow-r7 My understanding is that the work in @busterb repository has nothing to do with what's in master as it was supposed to use the system wide postgresql so I'm not sure why you are asking me to try what's in master. When I did my test, it was specifically testing the code in @busterb 's msfdb-improvements branch.

I'll gladly do more tests if required but it's not clear that we are on the same page here (and it's been a while so I don't remember everything in details).

Hi!

This issue has been left open with no activity for a while now.

We get a lot of issues, so we currently close issues after 60 days of inactivity. It’s been at least 30 days since the last update here.
If we missed this issue or if you want to keep it open, please reply here. You can also add the label "not stale" to keep this issue open!

As a friendly reminder: the best way to see this issue, or any other, fixed is to open a Pull Request.

I'd like this bug to be kept open until it's fixed. In Kali, we are still using our own fork of msfdb due to this and we're missing some features.

Although not entirely related to this ticket, if implementing this change requires a rewrite of the current functionality of msfdb - then it's potentially worth noting that I also ran into issues when I wanted to run msfdb against an empty docker postgres container.

For context - as a first time user I would have anticipated being able to provide a connection string to msfdb, or for it to use a provided postgres instance on the host machine. This would mimic the behavior of a normal rails db:create db:migrate workflow. Just a consideration for when this ticket is looked at next! 👍

Sharing what BlackArch is doing ~ https://github.com/BlackArch/msfdb

We added support for pg_ctlcluster over here #15054

There's also docker/remote database support by providing a connection string:

docker run -p 9000:5432 -e POSTGRES_PASSWORD=mysecretpassword postgres:14.2

bundle exec ruby ./msfdb reinit --connection-string=postgresql://postgres:mysecretpassword@localhost:9000/postgres

Closing for now, feel free to comment/raise an issue if the functionality still isn't quite there yet