I recently rebuilt one of my cloud VMs because it was aging and I wasn't happy with it anymore. After rebuilding it, I decided to use it to also run my own caching recursive DNS service that includes domain blacklisting (blocking known malware, phishing, and ad domains). Since this would have to be public-facing, I knew (at some point) someone would get curious and try to attack it. I have previous experience with just sshd being public, watching as countless bots scan and potential attackers probe for weaknesses.
I've always run SELinux and fail2ban, but I recently decided to take it a step further with hardened configuration files and custom fail2ban filters. fail2ban has some great builtin filters and they'll help protect a system against basic attacks with minimal configuration. However, what makes fail2ban really awesome are custom filters and actions. These enhance the functionality of fail2ban to give you the most protection possible. Nowadays, cyber criminals are after everyone, not just businesses.
Disclaimer: The file locations in this post are focused towards RHEL-based systems (i.e. CentOS, Fedora, etc.). I also have no experience (yet!) with firewalld; my current expertise is limited to iptables. However, the concepts and configurations should easily translate to any modern Linux system.
What's fail2ban?
fail2ban is an awesome IDS/IPS (Intrusion Detection System/Intrusion Prevention System) that uses regular expressions to scan logs then subsequently perform certain operations. Namely, it interfaces very well with iptables. Out of the box, it has support for a wide range of services and builtin regular expressions to help defend against the most common types of attacks. It also has support for custom filters and actions, making it highly extensible. fail2ban, leveraged with sound security practices, can provide assistance with protecting your system and services.
sshd (or, "Secure Shell Daemon")
sshd is the best service to use for performing remote system management. In some cases, it's the only way you can get into a box. Thus, taking advantage of the protection that fail2ban can bring to your system should be paramount. There are a few steps I would recommend making in addition to activating the sshd IDS/IPS protection in fail2ban.
First, set your default iptables INPUT policy to DROP and save the rules. The default policy provided by the vendor is ACCEPT, which is the equivalent of "accept all, deny none" unless you have a rule specifically blocking a port and/or protocol. Setting your default INPUT policy to DROP basically says, "deny all, accept none". This allows you to setup your iptables rules to only allow specific traffic. This protection is a basic rule that everyone should employ - least privilege methodology. Your system can still be compromised even if you have nothing listening; if you do and you forget to configure it to only listen on localhost, it immediately becomes an attack vector. This method also allows you to setup a new service and configure it before activating a rule, thereby allowing traffic to hit it.
Before setting your default policy to DROP, make sure you also have a rule that allows for STATEFUL,ESTABLISHED connections. This rule is typically included by default on RHEL-based systems. If you're not running on a RHEL-based system, make sure you have this rule at the top of your INPUT table.
Remember: Firewall rules are processed based on their order in the table.
Next, make sure you have a rule allowing SSH traffic on port 22. Once you have the aforementioned rules added, you can then feel safe to set your default INPUT policy to DROP. Don't lock yourself out of your own box! If you already have fail2ban installed, their rules should be first while the service is running. Simply stop the fail2ban service, add your rules (don't forget to save!), then start it again.
Second, secure your sshd service by hardening your sshd_config file. This file is typically found at /etc/ssh/sshd_config. Start by denying root login via SSH. Set PermitRootLogin to "no":
PermitRootLogin no
You should never remotely login to your box as root. Only allow logins by an underprivileged account that has permission (if necessary) to escalate to root or access the sudo command. Next, set some strong Key Exchange Algorithms, data encryption Ciphers, and Message Authentication Ciphers. This will protect you from attackers and the Big Bad (cough, cough - NSA). It's actually surprising how many attackers don't have a modern SSH client that supports newer ciphers. It's likely due to their targets being systems that aren't using modern cryptography. As clients are updated with modern algorithms, deprecated ones are removed. Nonetheless, any modern macOS and Linux operating system should support these ciphers. I haven't tested with Windows and PuTTY, though. If you'd like to ensure your session is encrypted using strong stuff, add the following text below the "Protocol 2" line:
# Specify Algos and Ciphers (so we're not vulnerable to default and weak cryptos)
KexAlgorithms diffie-hellman-group-exchange-sha256
Ciphers aes256-ctr,aes192-ctr,aes128-ctr
MACs hmac-sha2-512,hmac-sha2-256
Increase sshd's logging level. Set the LogLevel to VERBOSE:
LogLevel VERBOSE
This allows you to more closely audit the activity of sshd and key authentication. This is also required for a custom fail2ban filter that I will discuss later.
Disable password authentication and only utilize key-based authentication. It's simple:
PasswordAuthentication no
Make sure you generate a key pair (I recommend 2048-bit RSA or greater); store the public key in ~/.ssh/authorized_keys and chmod it to 0400. Only leave password authentication enabled if you're using a form of two-factor (key + account password).
Whether using key-based or password-based authentication, a valid user shouldn't have any reason to take longer than a minute to login. Set the LoginGraceTime to a conservative value; I have mine at one minute:
LoginGraceTime 1m
Let's also protect against resource consumption DoS attacks and reduce the number of chances an attacker has per connection. This is as simple as reducing MaxAuthTries:
MaxAuthTries 3
The default is set to six. As long as you have fail2ban blocking an IP after three authentication failures, you should be safe.
Finally, with regards to sshd_config, I set the following to reduce the attack surface on the server and client:
GSSAPIAuthentication no
X11Forwarding no
I don't use GSS and I don't want it available to a potential attacker. I don't use X11 on my remote server, so I close that open channel (which can be an attack vector on the client).
Third, custom sshd filter in fail2ban. Even though disabling password authentication and setting strong algorithms should seem like enough, it's not. While attackers may not reach a preauth or auth stage, they still consume resources by establishing a connection. To further prevent a resource consumption DoS, I setup the following file in /etc/fail2ban/filter.d/sshd-badmac.conf:
# This rule augments the default fail2ban sshd rules by adding an additional filter to block potential attackers.
# There really aren't any legitimate reasons why an authorized person would connect using unsupported macs, kexs, or algos.
[INCLUDES]
# Read common prefixes. If any customizations available -- read them from
# common.local
before = common.conf
[Definition]
_daemon = sshd
# these are the log entries we're looking for
#Mar 28 14:30:21 myserver sshd[30255]: Connection from 123.45.67.89 port 98765
#Mar 28 14:30:22 myserver sshd[30256]: fatal: no matching mac found: client hmac-md5,hmac-sha1 server hmac-sha2-512,hmac-sha2-256
failregex = ^(?P<__prefix>%(__prefix_line)s)Connection fromport \d+(?: on \S+ port \d+)? %(__prefix_line)sfatal: no matching mac found: .+$
ignoreregex =
[Init]
maxlines = 2
journalmatch = _SYSTEMD_UNIT=sshd.service + _COMM=sshd
And activate it in /etc/fail2ban/jail.d/jail.local:
[sshd-badmac]
enabled = true
filter = sshd-badmac
port = ssh
logpath = /var/log/secure
bantime = 86400
findtime = 3600
maxretry = 1
action = iptables[name=SSHBADMAC, port=ssh, protocol=tcp]
Ba'da bing, ba'da boom. Hoohaw! Three steps to secure your sshd service and protect your system from attackers.
bind (or, "named")
In my opinion, bind is the best DNS server available. It's highly configurable, fast, and performs well under a heavy load. However, there are some security considerations to take in if you're going to host it on a public system. To make this work, you're going to have to allow queries from subnets outside of your host's public subnet. By default, bind allows recursive queries as long as the client is within the same subnet as the host. If you wish to take it a step further, you can block non-domestic IPs based on the ranges available from ARIN (American Registry of Internet Numbers). Even if you employ this technique, domestic attackers may still attempt to DoS your DNS server, compromise it, or even rack up your bandwidth charge because they're bored. There are a few steps you can take to protect your public-facing DNS service whether you're running a resolver or recursive name server.
First, we need to harden our bind configuration to rate-limit queries. I have averaged usage by a few users that I share my DNS service with and a reasonable user shouldn't be making more than 10 queries per second. Start by adding this to your bind config (in the options section), typically found at /etc/named.conf:
rate-limit {
responses-per-second 10;
log-only yes;
};
Make sure you also enable DNSSEC:
dnssec-enable yes;
dnssec-validation yes;
dnssec-lookaside auto;
Also be sure to deny transfers:
allow-transfer { none; };
Make sure you aren't replying authoritatively for domains that you do not resolve for:
auth-nxdomain no;
Finally, block your version number from being exposed to attackers, thereby making their job easier. Blocking the version number prevents an attacker from looking up existing vulnerabilities and attacking your service. This makes their job harder and they will likely pass you up. To do this, set the following option:
version "unknown";
Second, setup your bind logging to log everything you'd ever need to know about bind's activity. This will also come in handy with setting the rate limiting I mentioned earlier. The fail2ban custom filter utilizes the query log to block potential malicious activity. Here's what my logging stanzas look like:
logging {
channel default {
file "/var/log/named-auth.log";
severity info;
print-time yes;
print-category yes;
print-severity yes;
};
channel default_debug {
file "data/named.run";
severity dynamic;
};
category lame-servers { null; };
category default { default; };
channel "querylog" { file "/var/log/named-query.log"; print-time yes; };
category queries { querylog; };
};
Third, setup a custom fail2ban filter to block users that are rate limited. I setup the following filter in /etc/fail2ban/filter.d/named-throttling.conf:
# fail2ban filter for hosts attempting to cause a DoS
# this works in conjunction with the bind query throttling. messages are printed when
# hosts start abusing the system.
# example of what we're looking for to block
# 28-Mar-2017 22:53:22.624 client 12.34.56.78#33283: query: nccih.nih.gov IN ANY +E (98.76.54.32)
# 28-Mar-2017 22:53:22.624 client 12.34.56.78#33283: would slip response to 12.34.56.0/24 for nccih.nih.gov IN ANY (00111ec3)
# 28-Mar-2017 22:53:22.624 client 12.34.56.78#33283: query: nccih.nih.gov IN ANY +E (98.76.54.32)
# 28-Mar-2017 22:53:22.624 client 12.34.56.78#33283: would drop response to 12.34.56.0/24 for nccih.nih.gov IN ANY (00111ec3)
[Definition]
# Daemon name
_daemon=named
# Shortcuts for easier comprehension of the failregex
__pid_re=(?:\[\d+\])
__daemon_re=\(?%(_daemon)s(?:\(\S+\))?\)?:?
__daemon_combs_re=(?:%(__pid_re)s?:\s+%(__daemon_re)s|%(__daemon_re)s%(__pid_re)s?:)
# hostname daemon_id spaces
# this can be optional (for instance if we match named native log files)
__line_prefix=(?:\s\S+ %(__daemon_combs_re)s\s+)?
failregex = ^%(__line_prefix)s?\s*client#\S+( \([\S.]+\))?: would (slip|drop) response .+$
ignoreregex =
[Init]
maxlines = 1
I then activated the filter in my /etc/fail2ban/jail.d/jail.local file with the following:
[named-throttling]
enabled = true
filter = named-throttling
port = domain
logpath = /var/log/named-query.log
bantime = 120
findtime = 60
maxretry = 1
action = iptables[name=NAMEDTHROTTLE, port=domain, protocol=udp]
This ensures that anyone who triggers bind's rate limiting is blocked for two minutes. As I stated earlier, any reasonable user shouldn't have more than 10 queries per second. So, if you're not abusing the service, you shouldn't have to worry. Keep in mind, I set this up for my friends and family to use, as well as myself. So, based on your usage, you may need to tweak the bind rate limiting.
There you have it, folks. My thoughts on how to secure your system with hardened configurations and custom filters. I have some advanced tips I'll likely cover in future articles on iptables, sshd, bind, and fail2ban; they just didn't fit here.
Versions
CentOS: release 6.8bind/named: bind-9.8.2-0.47.rc1.el6_8.4.x86_64
sshd: openssh-5.3p1-118.1.el6_8.x86_64
fail2ban: fail2ban-0.9.6-1.el6.1.noarch
Links
CentOS: https://www.centos.org/RHEL: https://www.redhat.com/en/technologies/linux-platforms/enterprise-linux
EPEL (where you can grab fail2ban): https://fedoraproject.org/wiki/EPEL
bind/named: https://www.isc.org/downloads/bind/
sshd: https://www.openssh.com/
fail2ban: https://www.fail2ban.org/wiki/index.php/Main_Page
Until next time ...
