Skip to content

Latest commit

 

History

History
670 lines (425 loc) · 23.7 KB

server-security.md

File metadata and controls

670 lines (425 loc) · 23.7 KB
Raw

Server Security

This is very incomplete and covers some of the basics. These assume a single server setup, some of the recommendations would change considerably if you could delegate to other servers in your cluster.

Securing your server

Create a new server and set your SSH key. Then ssh root@IP_ADDRESS

Root Password

First things, first change your root password:

passwd

Set a password generated from 1Password.

Updates and automatic updates:

You should upgrade right now and keep things upgraded:

apt-get update
apt-get upgrade
apt-get install unattended-upgrades

Next edit /etc/apt/apt.conf.d/10periodic so it looks like:

APT::Periodic::Update-Package-Lists "1";
APT::Periodic::Download-Upgradeable-Packages "1";
APT::Periodic::AutocleanInterval "7";
APT::Periodic::Unattended-Upgrade "1";

Creating a user

Now we want a user. We'll call this user deploy:

useradd -D -s /bin/bash
useradd deploy --shell /bin/bash
mkdir /home/deploy
mkdir /home/deploy/.ssh
chmod 700 /home/deploy/.ssh

Now that you have a user you want to setup an authorized key saying that your computer is allowed to login to the server without using a password. This is actually much safer than using a password because passwords can be guessed. Keys are much longer and encrypt all of you communications between your machine and the server.

nano /home/deploy/.ssh/authorized_keys

The nano editor will open and you need to get some information from your local computer. Open another tab or another terminal window and in the new Terminal type:

cat ~/.ssh/id_rsa.pub

You should see something like:

$ cat ~/.ssh/id_rsa.pub
ssh-rsa ssflsjdjsfljslkdfjyUvMIvhJUYs7nIaiLBnUcs03XuOeHiw1JGh1M/ovbKc9YO4SJsl9CYxpyDeh9jSyvdNhNdeUSg7PBSSyAYpVXeK6WXN9LnqKOWRu8n5rXGNSycM2tenaADiS/xtMkHmIFYOE/QFQF+AmgdklfjsdfjlskjdflksjfdlkjslkjsIP/gxSsEDzOGSIxYqJwpCzV+/nwub72ElzVcW9EJu2HJHGYIUHK80980989089089089090OyFubRb7+V6Db4xc+x2HirdyMW7hZceDfTpB0xX1GXd4PYZ023gfvNkGbIj/aOSDeJtDIJyQhzCbIXY+CiLkU556e0rg2Nw== [email protected]

Note that this is your public key. You also have a private key which you should not give away ever (it is in the file id_rsa without the .pub extension).

Copy the contents (including ssh-rsa and [email protected]) and paste it back in the nano window in the other terminal. Once pasted, click Ctrl+X type "Y" for "yes" and hit enter.

Now we need to lock down that file so nobody can change it (and add their own keys). Once we have changed the mode so it is only readable, we will change the owner to our new deploy user:

chmod 400 /home/deploy/.ssh/authorized_keys
chown deploy:deploy /home/deploy -R

Super user access (optional)

We'll give the user a password which we will need when doing important things.

passwd deploy

Next we need to change how the sudo command works. The sudo command is short for "super-user do" and lets users run commands as if they were the root (most powerful) user. To change this we'll use a command called visudo. This command opens an editor called vi which works a little differently than the editors you are used to. Instead we'll force it to use nano:

EDITOR=nano visudo

Once open you should see a lot of commands and you want to comment some of them out:

%admin ALL=(ALL) ALL

and

%sudo ALL=(ALL) ALL

We are going to comment these lines out by adding a "# " to the start of the lines:

# %admin ALL=(ALL) ALL

and

# %sudo ALL=(ALL) ALL

Next we'll add the line:

deploy ALL=(ALL) ALL

This says, as long as we are logged in and know the password we can run ALL commands as the superuser.

Technically we shouldn't need this for our server as everything will be setup. Giving users sudo is very serious. If they have super user privileges they can change anything about the server. Sometimes when setting up a server you'll see people use NOPASSWD:ALL indicating that you don't even need a password. This is very dangerous and is usually just a sign of laziness. For us, it is good to have some access so that we can switch to the super user later.

Finally we want to be able to use SSH as this user:

groupadd ssh-user
usermod -a -G ssh-user deploy

Local mail and smtp via Postfix

These instructions are taken directly from https://www.digitalocean.com/community/tutorials/how-to-install-and-configure-postfix-as-a-send-only-smtp-server-on-ubuntu-14-04. You'll want to note your Droplet's hostname:

hostname -f

On Ubuntu the easiest way to get mail up an running is to install mailutils:

apt-get install mailutils

When prompted for the Postfix Configuration choose Internet Site and choose OK. Next you will be asked for the System Mail Name which should be the same as your droplet's hostname.

Next you'll need to edit /etc/postfix/main.cf. Change

inet_interfaces = all

To:

inet_interfaces = localhost

Then restart postfix:

service postfix restart

Test that sending mail works:

echo "This is the body" | mail -s "The subject" [email protected]

To forward system mail you'll need to add an alias to /etc/aliases. You can add aliases for existing users and non-existing users:

# See man 5 aliases for format
postmaster:    root
root:          [email protected]
ops:           [email protected]

Save the file and enable the aliases:

newaliases

An alternate email host

What if the box you are creating is actually api.example.com but you want the email it is sending out to appear to be from example.com. You can change this (with the caveat that your RDNS will not be right). In /etc/postfix/main.cf change:

myhostname = api.example.com

To:

myhostname = example.com

And change:

mydestination = api.example.com, localhost.example.com, localhost

To:

mydestination = example.com, localhost

Then change /etc/mailname to

example.com

Testing email

Test that sending mail works:

echo "This is the body" | mail -s "The subject" root

You can whitelist emails from the domain in your email client (so they are not sent to spam) or you can correctly identify the server using a PTR record, SPF record and by using OpenDKIM

Make sure the PTR record (the name of your droplet) matches your fully qualified domain name (FQDN). For example, if you domain is example.com, then your mailing host above should be example.com and your droplet should be called example.com. Your PTR record is managed by your hosting provider (not your DNS).

Then add an SPF record to your DNS:

SPF @ "v=spf1 a mx ptr include:_spf.google.com ip4:1.2.3.4 -all"

This says that email sent from your A record, MX record, or PTR and from google.com should be allowed. It also allows your primary IP as a sender. With the setup described in this section, you could actually remove the MX (it isn't sending directly).

Some DNS companies don't allow a specific "SPF" record. If you can add both an SPF record and a TXT record with the same value.

You can check that you have set this up correctly at https://www.unlocktheinbox.com/dnstools/spf/

OpenDKIM

For OpenDKIM, start by installing the package:

apt-get install opendkim opendkim-tools

Add the following options to /etc/opendkim.conf:

AutoRestart             Yes
AutoRestartRate         10/1h
SyslogSuccess           Yes
LogWhy                  Yes

Canonicalization        relaxed/simple

ExternalIgnoreList      refile:/etc/opendkim/TrustedHosts
InternalHosts           refile:/etc/opendkim/TrustedHosts
KeyTable                refile:/etc/opendkim/KeyTable
SigningTable            refile:/etc/opendkim/SigningTable

Mode                    sv
PidFile                 /var/run/opendkim/opendkim.pid
SignatureAlgorithm      rsa-sha256

UserID                  opendkim:opendkim

Socket                  inet:12301@localhost

Add the milter by adding the following to /etc/default/opendkim:

SOCKET="inet:12301@localhost"

Add to /etc/postfix/main.cf:

milter_protocol = 6
milter_default_action = accept
smtpd_milters = inet:localhost:12301
non_smtpd_milters = inet:localhost:12301

Specify Trusted hosts in /etc/opendkim/TrustedHosts:

127.0.0.1
localhost
192.168.0.1/24

*.example.com

# Not sure if we need these
# example.com
# 172.17.0.0/16

In /etc/opendkim/KeyTable:

mail._domainkey.example.com example.com:mail:/etc/opendkim/keys/example.com/mail.private

In /etc/opendkim/SigningTable:

*@example.com mail._domainkey.example.com    
*@api.example.com mail._domainkey.example.com

Note: if you have a subdomain but are sending as the root domain you can have multiple values here (on multiple lines) pointing to the same key.

Make a key dir and key (note, if you have a subdomain you can still sign but the TXT record may be impacted):

mkdir -p /etc/opendkim/keys/example.com
cd /etc/opendkim/keys/example.com
opendkim-genkey -s mail -d example.com
chown opendkim:opendkim mail.private
cat mail.txt

Copy the p value and create a TXT DNS entry. Domain:

TXT mail._domainkey "v=DKIM1; k=rsa; p=YOUR_P_VALUE_HERE"

Restart:

service postfix restart
service opendkim restart

You can test the setup by sending mails at http://www.mail-tester.com.

Unfortunately testing via mail-tester.com is limited to 3 per day. To check that your DKIM record is correct instead go to http://dkimcore.org/c/keycheck. For your selector choose mail (this is the prefix for mail._domainkey) and then enter your root domain:

If you are sending via a third party domain you can explore ATPSDomains. Or you can set them up manually. For example Mandrill allows customization: https://mandrill.zendesk.com/hc/en-us/articles/205582387-How-do-I-set-up-sending-domains-

Once you have completed your SPF and DKIM setup and have successfully tested them, you can add a _dmarc host (another TXT record) that helps ensure your SPF and DKIM are respected.

TXT _dmarc "v=DMARC1; p=none"

For example, on Cloudflare you might have entries like this:

Sample DNS config

Troubleshooting

Send yourself an email and view the raw source (in Mail.app it is View | Message | Raw Source)

Delivered-To: [email protected]
Received: by 10.31.59.7 with SMTP id i7csp785441vka;
        Sun, 8 Jan 2017 12:25:30 -0800 (PST)
X-Received: by 10.200.55.5 with SMTP id o5mr14642120qtb.248.1483907130557;
        Sun, 08 Jan 2017 12:25:30 -0800 (PST)
Return-Path: <[email protected]>
Received: from admin.example.com (admin.example.com. [1.2.3.4])
        by mx.google.com with ESMTP id u67si45157657qkb.264.2017.01.08.12.25.30
        for <[email protected]>;
        Sun, 08 Jan 2017 12:25:30 -0800 (PST)
Received-SPF: pass (google.com: domain of [email protected] designates 1.2.3.4 as permitted sender) client-ip=1.2.3.4;
Authentication-Results: mx.google.com;
       dkim=pass [email protected];
       spf=pass (google.com: domain of [email protected] designates 1.2.3.4 as permitted sender) [email protected]
Received: by admin.example.com (Postfix, from userid 1001)
	id 2E730152A5D; Sun,  8 Jan 2017 15:23:34 -0500 (EST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=example.com; s=mail;
	t=1483907130; bh=fdkeB/A0FkbVP2k4J4pNPoeWH6vqBm9+b0C3OY87Cw8=;
	h=Subject:To:Date:From:From;
	b=GnitLFExKeciSLFD0W8wl7yBhsNlZ9M7rX8TuCAuVeMDB1YEL5J8ti15WD4wuZdLF
	 DES3UVAZ0klBjfGXxdAgnLwVfh3UoRam4cP6dgGI08AwyFKHmenUWNUi4jUbxLAckC
	 JWcfqxU6vfksO4famHKuKNR4kpjz0Tz+VuDdmaxg=
Subject: The Subject
To: <[email protected]>
X-Mailer: mail (GNU Mailutils 2.99.98)
Message-Id: <[email protected]>
Date: Sun,  8 Jan 2017 15:23:34 -0500 (EST)
From: [email protected]

Test

Here you can see if your DKIM and SPF records passed. If there is no DKIM record then it might not be signing correctly. This generally happens if there is not a SigningTable entry for the "from address" domain.

If it is signed, but there is dkim=temperror (no key for signature) then it is likely that the d= value does not accurately reflect the TXT record in your DNS. For example if you have d=admin.example.com but your _domainkey does not include the subdomain you'll see this error.

If you can receive email on the reply-to address you can also test by sending an email to [email protected].

Forwarding mail

From How to setup MX forwarding

In /etc/postfix/main.cf add the following:

virtual_alias_domains = example.com 
virtual_alias_maps = hash:/etc/postfix/virtual

If you followed the previous section you'll actually need to change your inet_interfaces line:

inet_interfaces = all

Then in /etc/postfix/virtual:

# you can add specific virtual emails here for specific forwarding
@example.com   [email protected]

Then restart the service:

postmap /etc/postfix/virtual    
service postfix reload

Now that your server can forward email you'll want to ensure the port is open (see the Firewall section below) and that the MX domain records are pointing to your server.

Test that your server is configured correctly (and is not an open relay) using a service like http://mxtoolbox.com/diagnostic.aspx

Running mail from a Docker container

To send email from within a Docker container (like Dokku), you need postfix to bind to the interface and then either setup TLS for authentication or list the container network inside your mynetworks directive.

Docker creates its own network interface docker0. Modify /etc/postfix/main.cf and make sure you have:

inet_interfaces = all

Or that you have the interface listed:

inet_interfaces = eth0, docker0, lo

One this is set you can simply list your docker IP range:

mynetworks = 127.0.0.0/8 172.17.0.0/16 [::ffff:127.0.0.0]/104 [::1]/128

You can find your docker range from ifconfig and look at the docker0 interface. Also, you can docker ps, then grab a container name and docker inspect <CONTAINER>.

Watching the logs

Next get logwatch:

apt-get install logwatch

Make a cache dir:

mkdir /var/cache/logwatch

Make a default conf:

cp /usr/share/logwatch/default.conf/logwatch.conf /etc/logwatch/conf/

If you want to send email directly to an alternate user you can edit the cron configuration /etc/cron.daily/00logwatch:

/usr/sbin/logwatch --mailto [email protected]

You can test this by executing the cron manually:

/etc/cron.daily/00logwatch

Autoban malicious users

If someone is trying to do something bad and you can detect it, you should kick them out (or slow them down):

apt-get install fail2ban

Create /etc/fail2ban/jail.local:

[DEFAULT]
ignoreip  = 127.0.0.1
bantime   = 86400
destemail = [email protected]
banaction = iptables-multiport
action    = %(action_)s

# JAILS
[ssh]
enabled   = true
maxretry  = 3

[pam-generic]
enabled   = true
banaction = iptables-allports

[ssh-ddos]
enabled   = true

[apache]
enabled = true

[nginx-http-auth]
enabled = true

[postfix]
enabled  = true
maxretry = 1

[dovecot-pop3imap]
enabled = true
filter = dovecot-pop3imap
action = iptables-multiport[name=dovecot-pop3imap, port="pop3,imap,993,995", protocol=tcp]
logpath = /var/log/mail.log
maxretry = 20
findtime = 1200
bantime = 1200

Rootkits

In addition to fail2ban you'll want to check for rootkits:

apt-get install lynis
apt-get install rkhunter

Securing SSH

You'll need to secure your /etc/ssh/sshd_config:

# Package generated configuration file
# See the sshd_config(5) manpage for details

# What ports, IPs and protocols we listen for
Port 22

# Never use protocol 1
Protocol 2

# HostKeys for protocol version 2
HostKey /etc/ssh/ssh_host_rsa_key
HostKey /etc/ssh/ssh_host_ed25519_key

# Secure defaults; we do not support older encryption. We must support Kex algorithms
# that rely on the moduli (diffie-hellman-group-exchange-sha256) because OSX won't use
# curve25519 by default (yet)
KexAlgorithms [email protected],diffie-hellman-group-exchange-sha256
Ciphers [email protected],[email protected],[email protected],aes256-ctr,aes192-ctr,aes128-ctr
MACs [email protected],[email protected],[email protected],[email protected]

# Privilege Separation is turned on for security
UsePrivilegeSeparation yes

# Lifetime and size of ephemeral version 1 server key
KeyRegenerationInterval 3600
ServerKeyBits 2048

# Causes rekeys for forward secrecy intra-session, may affect large uploads or downloads
RekeyLimit 256M

# Logging
SyslogFacility AUTH
LogLevel INFO

# Authentication:
LoginGraceTime 30
MaxAuthTries 3
PermitRootLogin no
StrictModes yes

# Force logout idle sessions (15 minutes)
ClientAliveInterval 900
ClientAliveCountMax 0

# Don't read the user's ~/.rhosts and ~/.shosts files
IgnoreRhosts yes
# For this to work you will also need host keys in /etc/ssh_known_hosts
RhostsRSAAuthentication no
# similar for protocol version 2
HostbasedAuthentication no
# Uncomment if you don't trust ~/.ssh/known_hosts for RhostsRSAAuthentication
#IgnoreUserKnownHosts yes

# To enable empty passwords, change to yes (NOT RECOMMENDED)
PermitEmptyPasswords no

# Change to yes to enable challenge-response passwords (beware issues with
# some PAM modules and threads)
ChallengeResponseAuthentication no

# Change to no to disable tunnelled clear text passwords
PasswordAuthentication no

# Turn off other forms of authentication
RSAAuthentication no # Do I need this?
PubkeyAuthentication yes

# We are not allowing tunneling
AllowTcpForwarding no
GatewayPorts no
PermitTunnel no

# Not using X11
X11Forwarding no
X11DisplayOffset 10
PrintMotd no
PrintLastLog yes
TCPKeepAlive yes
#UseLogin no

# Limit the number of concurrent unauthenticated sessions
MaxStartups 2
MaxSessions 3

# Legal warning to tell people to stay out
Banner /etc/issue.net

# Don't allow client to pass env, even locale environment variables
PermitUserEnvironment no

Subsystem sftp /usr/lib/openssh/sftp-server

# Set this to 'yes' to enable PAM authentication, account processing,
# and session processing. If this is enabled, PAM authentication will
# be allowed through the ChallengeResponseAuthentication and
# PasswordAuthentication.  Depending on your PAM configuration,
# PAM authentication via ChallengeResponseAuthentication may bypass
# the setting of "PermitRootLogin without-password".
# If you just want the PAM account and session checks to run without
# PAM authentication, then enable this but set PasswordAuthentication
# and ChallengeResponseAuthentication to 'no'.
UsePAM yes

# Take DNS out of the equation
UseDNS no

# Only allow specific users to ssh.
# Add the group `sudo groupadd ssh-user`
# Then add a user to the group `sudo usermod -a -G ssh-user <username>`
AllowGroups ssh-user

Fix your moduli. If /etc/ssh/moduli exists:

awk '$5 > 2000' /etc/ssh/moduli > "${HOME}/moduli"
wc -l "${HOME}/moduli" # make sure there is something left
mv "${HOME}/moduli" /etc/ssh/moduli

Otherwise create it (takes about 10 minutes on a Digital Ocean 5$/mo box):

ssh-keygen -G "${HOME}/moduli" -b 4096
ssh-keygen -T /etc/ssh/moduli -f "${HOME}/moduli"
rm "${HOME}/moduli"

Create /etc/issue.net:

Unauthorized access to this system is forbidden and will be
prosecuted by law. By accessing this system, you agree that your
actions may be monitored if unauthorized usage is suspected.

Regenerate your keys (optional?):

cd /etc/ssh
rm ssh_host_*key*
ssh-keygen -t ed25519 -f ssh_host_ed25519_key < /dev/null
# Don't paste this whole block run the next command separately
ssh-keygen -t rsa -b 4096 -f ssh_host_rsa_key < /dev/null

This will cause a known host error when SSHing in the next time if you have added the previous keys to your known hosts.

To restart SSH, keep your current session open and run

service ssh restart

Then from another console try to SSH in again. If successful you have not lost your login.

Firewall

You want to restrict most external access to this machine. For now we want to allow only three services: SSH, and HTTP/HTTPs. To do this we could edit the iptables directly. It is much easier, however, to use the Uncomplicated Firewall (ufw) utility.

ufw default deny incoming

Because we specified our SSH port as 22, to allow it, run:

ufw allow 22

Note, this allows any machine on the internet to connect to this port and attempt to authenticate via SSH. To lock this down to just your machine:

ufw allow from YOURIP to any port 22

Note, this means that you won't be able to SSH from a coffee-shop, the office, etc. You could specify multiple addresses by running the command with each of the different IP addresses.

ufw allow 443
ufw allow 80
ufw allow 25
ufw enable

Footnotes

TODO

  • TODO: force chroot in SSH?
  • TODO: ntp (local mirror)
  • TODO: IDS (OSSEC)
  • TODO: logrotate
  • TODO: logstash
  • TODO: nagios?
  • TODO: monitoring

NTP

See http://ubuntuforums.org/showthread.php?t=862620

driftfile /var/lib/ntp/ntp.drift
server 0.pool.ntp.org
server 1.pool.ntp.org
server 2.pool.ntp.org
server 3.pool.ntp.org
server 127.127.1.0
fudge 127.127.1.0 stratum 10

# Could use north america specific servers instead
server 0.north-america.pool.ntp.org
server 1.north-america.pool.ntp.org
server 2.north-america.pool.ntp.org
server 3.north-america.pool.ntp.org

OSSEC

https://www.digitalocean.com/community/tutorials/how-to-install-and-configure-ossec-security-notifications-on-ubuntu-14-04

logstash

Jump servers / Bastion servers