sshd, a n00bs tail …
What better way to start a blog than to share how I built the OpenBSD server that hosts WordPress on it. Or more specifically, how I started by setting up sshd securely. This will be a multi-part series of posts that covers the major pain points I went through. Plus, I have all that learning from my previous build to post that I, er, didn’t .. write down. God damn it. Well, this time it will be different.
I should point out that this is not a “How To” on building and configuring a OpenBSD server. This series is about getting WordPress installed on a OpenBSD server in a reasonably secured fashion. Visit the OpenBSD.org site to learn more about building a OpenBSD server. I also use a iMac and am quite familiar with using terminal windows. If your using MS Windows, my condolences, they make some fine products, their operating systems however, was never one of my favorites.
NOTE: I HIGHLY recommend acquiring a password vault like 1Password. You will be needing to save many crucial logins and passwords before we are done with this series.
Now, where to house the new Server? My house was definitely out.
1. I had no protection from power outages (and I live in a high wind area)
2. The monthly costs of having a business line through my ISP was high yet they had a sketchy service history
3. No redundancy in the event of hardware failure
4. I would have to buy hardware
5. Having to keep the server up 24/7
So I decided to rent space and build a virtual private server (VPS) on Vultr.com which provide the following benefits:
1. Power outages were no longer an issue
2. The current monthly charges are around $8.00/mo.
3. It is a VPS, hardware issues should be non-existent from where I’m sitting
4. I didn’t have to buy the Data center grade hardware
5. Server up time is “always” with some maintenance windows and or I touch it in un-professional ways.
The best part is Vultr.com host OpenBSD 6.1 OS out of the box! Easy Peasy, Weasy. They even had an ISO available that had 6.2, which I used, that took me through the traditional setup procedure.
I went to GoDaddy.com and reserved my domain, then off to Vultr.com to set up my Vultr.com account and setup my OpenBSD server in like 2 minutes using Vultr.com website. A very convenient experience.
But before we start setting up WordPress lets make our server a little more secure so no one trashes our hard earned creation by securing SSH access. Here are our goals, simply put, we want to prevent a malicious user trying to attack our box through SSH by using the following techniques.
1. Every *nix box has the administrative user “root”. We want to prevent the user “root” from logging into our box to make guessing an admin user easy. Users can only use a low level account to login with which means they have to guess the account name and are not an admin if they gain access. There is a good probability that the user they guessed cannot elevate their permissions to root level.
2. All SSH sessions use port 22 to connect the server, Script Kiddies scan IPs for a port 22 that responds to brute force SSH login requests. We will make SSH listen to another port to keep login scans on port 22 from filling our logs.
3. SSH provides us the ability to use encrypted RSA keys. RSA keys uniquely identify a user by having the public key on our VPS hosting our website and the private key on the users local server. Both keys share the same passphrase. With this setup, a user must have the appropriate private key on the box they are establishing a connection with and the correct passphrase. It is known as two factor authorization.
4. Password only connections will not be allowed. We will deny all connections that do not utilize a RSA key for access.
Ssh stands for Secure SHell and has two main programs. The server side secure shell daemon (sshd) and the client side secure shell (ssh) you connect to the server with.
SSHD
Being a daemon (or service for you windows users) sshd and is located on /etc/rc.d/sshd and is started at boot time if enabled, which it is by default. On startup sshd uses it’s config file /etc/ssh/sshd_config to tell it how to run and what security to impose on each session that is connecting to the Openbsd server – which is currently minimal. It is this configuration file we will be updating to secure our connection to our OpenBSD.
SSH
To connect the a remote sshd we use the ssh client program which is located on /usr/bin/ssh on a *nix desktop. We will be using three terminal sessions in three separate windows today
1. A “VPS” in which we will be starting SSHD sessions in debug mode
2. A “Client” where we will be connecting to the SSHD debug sessions to test settings and making changes to the sshd debug configuration file.
3. A “Maintenance” widow and is used in the event that your production sshd_config gets set incorrectly and no longer allows connections, this is your failsafe. it can also be used to make any changes to your sshd_config.test file.
You should be able to connect as “root” to your server at this point. Do so now using your “Maintenance” window. You will need to add a low level user to OpenBSD to use as the ssh login later. We shall call our low level user “bob” in this tutorial, you can call him anything you like. Make sure you add that user to the “wheel” group when prompted for default login group. The wheel group contains the users who are allowed to elevate themselves to “root”, the almighty, an probably other things I have yet to discover. “root” can do anything, including dropping your server like a hot rock and giving you the opportunity to re-build it from scratch; always type wisely when using root privileges .
Lets give bob a password…
passwd bob
…enter and confirm a password for bob. Make sure both “bob” and root have secure passwords which you of course added to your password vault. You’ll thank me later.
Time to setup our testing environment. Making changes to our new servers /etc/ssh/sshd_config file directly is dangerous without testing those changes first, you can lock yourself out of your own server. To avoid that awkwardness, we are going to make a test /etc/ssh/sshd_config.test file. We can then test our changes to the sshd before we commit to them by using the -f switch to point to a different config file. We will then test each change and verify we get the result we want and we do not loose access to our server. After we are satisfied that all of our changes meet our needs, we will swap out the sshd_config file with the sshd_config.test and restart the servers sshd daemon. At that point our changes will be live and in production.
Open a ssh connection using you Client window to your VPS as “bob” and su to root to test to see if you have a valid user that you can ssh in with and verify that your user is in the “wheel” group. We will need root privileges to make some of our changes later.
Lets create a test sshd configuration file to try our new changes in and to prevent those changes from locking us out of our server. We will call this test sshd config file sshd_config.test. To make the sshd_config.test, simply use “cp” to make a copy of sshd_config to a new name.
cp /etc/ssh/sshd_config /etc/ssh/sshd_config.test
Now we want to start a new test ssh daemon using the /etc/ssh/sshd_config.test but use a new port as two sshd processes cannot share the same port. We are using the file switch (-f) so we use our new test config file and debug switch (-d[dd]) so we can see the connection process. Multiple “d” increase the verbosity of the debug output. The maximum is 3. The port switch (-p) will keep us from trying to use the default port 22 and instead use port 44433. After testing, If you type ctrl+c it will kill the sshd process and clean up the connections for you. You should ssh as bob in the Client window to your server and then su to root privileges.
<begin test>
In the VPS window end your sshd session (type ctl+c) if you have one. In the Client window exit your ssh connection (type exit+<Enter>) if you have one. Back in the VPS window, as root, start your debug ssh daemon:
/usr/sbin/sshd -f /etc/ssh/sshd_config.test -p 44433 -d
In the Client window connect to your VPS on the test port.
ssh bob@12.345.67.899 -p44433
if you successfully connect to the debug sshd, stop your sshd by going to the VPS terminal window it is running in and type ctrl+c thus shutting down the sshd using the /etc/ssh/sshd_config.test. This will terminate your connection in the Client window too.
<end test/>
You will want to repeat the above test after you make each change or every couple of changes to your sshd_config.test file. Using the above setup guarantees you will not “lock” yourself out of your own server.
Let address out first goal – Prevent root from logging in. Re-start VPS sshd using sshd_config.test as shown in the <test> above. In the Client window connect to your server as bob and elevate bob to root. When OpenBSD is distributed, all if its configuration files are set with the defaults listed but commented out using the “#” comment character. Though it looks like these options are not in use, they are the defaults. Using your favorite editor open the sshd_config.test and perform the following:
Uncomment “#PermitRootLogin prohibit-password” (# is the comment character; delete it) set it to “PermitRootLogin no” this prevents root from using ssh to login directly to your OS even if they have a RSA key.
Save your your changes to the sshd_config.test. Time to test. Re-start your sshd in your VPS window.
/usr/sbin/sshd -f /etc/ssh/sshd_config.test -p 44433 -d
… but connect as the root user in the Client window.
ssh root@12.345.67.899 -p44433
You should get “permission denied” this time. Re-attempt your Client login using bob.
ssh bob@12.345.67.899 -p44433
You should get ssh welcome screen. You have now prevented a malicious user from trying to login in with the one known user on most *nix boxes, “root”. It should be noted that your Maintenance window is still up and connected as root. This will still be the case but all future connections will prevent root as a login user.
Lets address our second goal – make sshd listen to another port to keep login scans on port 22 from filling our logs. In the Client window connect to your server as bob and elevate bob to root. Using your favorite editor open the sshd_config.test and perform the following:
Un-comment the “#Port 22” port setting (# is the comment character; delete it) and set the port from 22 to another number to keep script kiddies from trying gain access through the default ssh port and filling your logs with those annoying attempts. You can use “netstat -nat | grep LISTEN” to see which ports is currently in use and pick a different one. Aim for a high number like i did in the example to avoid conflicts. The port setting should look like this
Port 44433
Save your your changes to the sshd_config.test. Time to test. Re-start your sshd in your VPS window. But we do not need the “-p” switch this time as it has been set in the sshd_config.test file.
/usr/sbin/sshd -f /etc/ssh/sshd_config.test -d
… Connect using your Client window as bob. You still need to force the port to 44433 from the client.
ssh bob@12.345.67.899 -p44433
You should get ssh welcome screen. You now have prevented IP scans for ssh port 22 from filling your logs.
Lets address our third goal – setting up two factor authentication based on RSA keys. RSA keys are an authorization mechanism where the private key sits on your local server and the other public key is installed on all the servers you wish to authenticate too. RSA keys can have a passphrase (yes, phrases such as “You’ve Cat to be Kitten Me Right Meow!”) added to make them a two factor authorization mechanism which I recommend for this application. We will use RSA keys to stop anyone from logging in through ssh without a private key on their server. If they manage to acquire a copy of your private key they will still need the passphase. The private key can be transferred to another computer(s) so you can use this key on multiple computers to access the remote servers if you desire.
On your local computer use your Maintenance window move to your user .ssh directory (ex: CD ~/.ssh) or /home/<your login>/.ssh
To make a RSA_ID key we will need to invoke ssh-keygen and pass a filename(-f) switch to tell ssh-keygen what file name to use. Type the following (you can use another file name other than “bob_rsa_id”):
ssh-keygen -f bob_rsa_id
When prompted for a RSA_ID passphrase type one in (and store that in your password vault, see I told you you would thanks me latter…). This will generate a public and a private key of 2048 bits. bob_rsa_id is your private key that need to stay in your .ssh directory, bob_rsa_id.pub is your public key and is placed on any server you want to authenticate your local server too. Now we copy your public key to the VPS . The command to do this is scp (secure copy). Scp uses the following format -scp <source> <target> so we need to be in the .ssh directory when you do the copy.
Using your Maintenance window type the following:
scp bob_rsa_id.pub bob@12.345.67.899:~/.ssh
This copies the public key bob_rsa_id.pub from your local server to the VPS into the .ssh directory under the home directory(~) of the low level user account bob. You will get prompted for bobs password before the copy happens so have it handy. When your copy is finished, ssh onto the VPS and make sure your in your “.ssh” directory (cd ~/.ssh). Type “ls -la”
ls -la
Verify your _rsa_id.pub is there. Now we are going to push the contents of the _rsa_id.pub into the authorized_keys file. Type “cat bob_rsa_id.pub >> authorized_keys”. Make sure you use ‘>>’ not ‘>’ as ‘>’ will over write the file. We want to append to the bottom of the file.
cat bob_rsa_id.pub >> authorized_keys
Type “cat authorizes_keys” and verify the bob_rsa_id.pub contents are there.
cat authorizes_keys
Time to test. Start your sshd with the sshd_config.test file in the VPS window.
/usr/sbin/sshd -f /etc/ssh/sshd_config.test -d
In the Client window we will open a connection to the test sshd a little differently this time. We use the “-i” switch to tell ssh we want to use a specific RSA private key to authenticate with.
ssh bob@12.345.67.899 -p44433 -i bob_rsa_id
When prompted for a RSA_ID passphrase you should see “bob_rsa_id” in the request. Use the passphrase you used when generating the RSA key. If you are successful on the first login attempt, you should now be logged in as bob on your VPS using your RSA two factor authentication. It should be pointed out that if your login attempt is not successful for any reason the login security protocol will default back to keyboard interactive and allow you to log in using “bob”’s password at this point. We will fix that security hole next.
On to our our fourth goal – Password only connections will not be allowed. Elevate bob to root privileges and open /etc/ssh/sshd_config.test in your text editor. Some of these values are already set as the default though they commented out, uncomment them anyway to show clear intent for the next Administrator that comes along.
Uncomment “#PubkeyAuthentication yes”, verify it is set to “yes” this should be the default.
Uncomment “#PasswordAuthentication yes” and set it to PasswordAuthentication no
Uncomment “#PermitEmptyPasswords no” this should be the default.
Uncomment “#ChallengeResponseAuthentication yes” and set it to ChallengeResponseAuthentication no
Save your your changes to the sshd_config.test. Time to test. Re-start your sshd in your VPS window.
/usr/sbin/sshd -f /etc/ssh/sshd_config.test -d
In the Client window we will open a connection to the test sshd. Again we use the “-i” switch to tell ssh we want to use a specific RSA private key to authenticate with.
ssh bob@12.345.67.899 -p44433 -i bob_rsa_id
When prompted for a RSA_ID passphrase you should see “bob_rsa_id” in the request. Use the passphrase you used when generating the RSA key. You will have three attempts to get this right and you should not see a “fall back” to keyboard interactive password prompt. Log “bob” back out (type “exit”) and re-attempt with a bogus passphrase. After the third attempt should should get a “Permission denied (publickey).” And be returned to your local system prompt. SSH sessions on your VPS is now secured to only users who have a RSA key and a proper passphrase to use it.
Ok, now the moment of truth, it’s time to go live with your changes to the sshd_config file. Remember that maintenance window we discussed earlier that you were logged in as “root”? Check to make sure that is still open and working. It is our failsafe remember. To make our changes to the sshd_config.test file go live, we need to replace the sshd_config file with the sshd_config.test file. But first we want to save our default sshd_config file incase something goes sideways and we need to restore it. We will the “Move” command here. First we make a copy of sshd_config and call it sshd_config_old.
mv /etc/ssh/sshd_config /etc/ssh/sshd_config_old
Now we rename sshd_config.test to sshd_config.
mv /etc/ssh/sshd_config.test /etc/ssh/sshd_config
run “rcctl sshd restart” to pick up the new configuration changes. Your tested changes are now running confidently in production. Time to test. Make sure your VPS window is not running a debug session of sshd. Ctrl+c in that window sill stop it if it is. In your Client window make sure your not logged in. Typing “exit+<return>” until your back to your local machine prompt. Now using your client window simply log in as “bob” using your RSA key.
ssh bob@12.345.67.899 -p44433 -i bob_rsa_id
When prompted for a RSA_ID passphrase you should see “bob_rsa_id” in the request. Use the passphrase you used when generating the RSA key. If it was successful, great, you have completed your goals. If it was not a successful login you can use your Maintenance window to open up your sshd_config to see if you missed a step and ultimately rename your sshd_config back to sshd_config.test and sshd_config_old back to sshd_config to get you back to where you were before starting this adventure. However I doubt you will need to do that as you tested your changes through out this process. Remember to backup your RSA public and private keys!
Congratulations! You have made your Openbsd ssh sessions much more secure than they were and we covered some import concepts as well. Today we learned how to
• prevent root logins
• move the port sshd listens on to another less common port
• setup and use RSA keys to prevent unauthorized access to your server
• Remove the ability to use a keyboard interactive login/password to gain access
Not a bad couple of hours worth of work. But this is the first step in getting WordPress installed on Openbsd. We have much more fun to look forward too.