Protect your system with fail2ban and firewalld blacklists

If you run a server with a public-facing SSH access, you might have experienced malicious login attempts. This article shows how to use two utilities to keep the intruder out of our systems.

To protect against repeated ssh login attempts, we’ll look at fail2ban. And if you don’t travel much, and perhaps stay in one or two countries, you can configure firewalld to only allow access from the countries you choose.

First let’s work through a little terminology for those not familiar with the various applications we’ll need to make this work:

fail2ban: Daemon to ban hosts that cause multiple authentication errors.

fail2ban will monitor the SystemD journal to look for failed authentication attempts for whichever jails have been enabled. After the number of failed attempts specified it will add a firewall rule to block that specific IP address for an amount of time configured.

firewalld: A firewall daemon with D-Bus interface providing a dynamic firewall.

Unless you’ve manually decided to use traditional iptables, you’re already using firewalld on all supported releases of Fedora and CentOS.

Assumptions

  • The host system has an internet connection and is either fully exposed directly, through a DMZ (both REALLY bad ideas unless you know what you’re doing), or has a port being forwarded to it from a router.
  • While most of this might apply to other systems, this article assumes a current version of Fedora (31 and up) or RHEL/CentOS 8. On CentOS you must enable the Fedora EPEL repo with
    sudo dnf install epel-release

Install & Configuration

Fail2Ban

More than likely whichever FirewallD zone is set already allows SSH access but the sshd service itself is not enabled by default. To start it manually and without permanently enabling on boot:

$ sudo systemctl start sshd

Or to start and enable on boot:

$ sudo systemctl enable --now sshd

The next step is to install, configure, and enable fail2ban. As usual the install can be done from the command line:

$ sudo dnf install fail2ban

Once installed the next step is to configure a jail (a service you want to monitor and ban at whatever thresholds you’ve set). By default IPs are banned for 1 hour (which is not near long enough). The best practice is to override the system defaults using *.local files instead of directly modifying the *.config files. If we look at my jail.local we see:

# cat /etc/fail2ban/jail.local
[DEFAULT]

# "bantime" is the number of seconds that a host is banned.
bantime  = 1d

# A host is banned if it has generated "maxretry" during the last "findtime"
findtime  = 1h

# "maxretry" is the number of failures before a host get banned.
maxretry = 5

Turning this into plain language, after 5 attempts within the last hour the IP will be blocked for 1 day. There’s also options for increasing the ban time for IPs that get banned multiple times, but that’s the subject for another article.

The next step is to configure a jail. In this tutorial sshd is shown but the steps are more or less the same for other services. Create a configuration file inside /etc/fail2ban/jail.d. Here’s mine:

# cat /etc/fail2ban/jail.d/sshd.local
[sshd]
enabled = true

It’s that simple! A lot of the configuration is already handled within the package built for Fedora (Hint: I’m the current maintainer). Next enable and start the fail2ban service.

$ sudo systemctl enable --now fail2ban

Hopefully there were not any immediate errors, if not, check the status of fail2ban using the following command:

$ sudo systemctl status fail2ban

If it started without errors it should look something like this:

$ systemctl status fail2ban
● fail2ban.service - Fail2Ban Service
Loaded: loaded (/usr/lib/systemd/system/fail2ban.service; disabled; vendor preset: disabled)
Active: active (running) since Tue 2020-06-16 07:57:40 CDT; 5s ago
Docs: man:fail2ban(1)
Process: 11230 ExecStartPre=/bin/mkdir -p /run/fail2ban (code=exited, status=0/SUCCESS)
Main PID: 11235 (f2b/server)
Tasks: 5 (limit: 4630)
Memory: 12.7M
CPU: 109ms
CGroup: /system.slice/fail2ban.service
└─11235 /usr/bin/python3 -s /usr/bin/fail2ban-server -xf start
Jun 16 07:57:40 localhost.localdomain systemd[1]: Starting Fail2Ban Service…
Jun 16 07:57:40 localhost.localdomain systemd[1]: Started Fail2Ban Service.
Jun 16 07:57:41 localhost.localdomain fail2ban-server[11235]: Server ready

If recently started, fail2ban is unlikely to show anything interesting going on just yet but to check the status of fail2ban and make sure the jail is enabled enter:

$ sudo fail2ban-client status
Status
|- Number of jail:	1
`- Jail list:	sshd

And the high level status of the sshd jail is shown. If multiple jails were enabled they would show up here.

To check the detailed status a jail, just add the jail to the previous command. Here’s the output from my system which has been running for a while. I have removed the banned IPs from the output:

$ sudo fail2ban-client status sshd
Status for the jail: sshd
|- Filter
|  |- Currently failed:	8
|  |- Total failed:	4399
|  `- Journal matches:	_SYSTEMD_UNIT=sshd.service + _COMM=sshd
`- Actions
   |- Currently banned:	101
   |- Total banned:	684
   `- Banned IP list:   ...

Monitoring the fail2ban log file for intrusion attempts can be achieved by “tailing” the log:

$ sudo tail -f /var/log/fail2ban.log

Tail is a nice little command line utility which by default shows the last 10 lines of a file. Adding the “-f” tells it to follow the file which is a great way to watch a file that’s still being written to.

Since the output has real IPs in it, a sample won’t be provided but it’s pretty human readable. The INFO lines will usually be attempts at a login. If enough attempts are made from a specific IP address you will see a NOTICE line showing an IP address was banned. After the ban time has been reached you will see an NOTICE unban line.

Lookout for several WARNING lines. Most often this happens when a ban is added but fail2ban finds the IP address already in its ban database, which means banning may not be working correctly. If recently installed the fail2ban package it should be setup for FirewallD rich rules. The package was only switched from “ipset” to “rich rules” as of fail2ban-0.11.1-6 so if you have an older install of fail2ban it may still be trying to use the ipset method which utilizes legacy iptables and is not very reliable.

FirewallD Configuration

Reactive or Proactive?

There are two strategies that can be used either separately or together. Reactive or proactive permanent blacklisting of individual IP address or subnets based on country of origin.

For the reactive approach once fail2ban has been running for a while it’s a good idea to take a look at how “bad is bad” by running sudo fail2ban-client status sshd again. There most likely will be many banned IP addresses. Just pick one and try running whois on it. There can be quite a bit of interesting information in the output but for this method, only the country of origin is of importance. To keep things simple, let’s filter out everything but the country.

For this example a few well known domain names will be used:

$ whois google.com | grep -i country
Registrant Country: US
Admin Country: US
Tech Country: US
$ whois rpmfusion.org | grep -i country
Registrant Country: FR
$ whois aliexpress.com | grep -i country
Registrant Country: CN

The reason for the grep -i is to make grep non-case sensitive while most entries use “Country”, some are in all lower case so this method matches regardless.

Now that the country of origin of an intrusion attempt is known the question is, “Does anyone from that country have a legitimate reason to connect to this computer?” If the answer is NO, then it should be acceptable to block the entire country.

Functionally the proactive approach it not very different from the reactive approach, however, there are countries from which intrusion attempts are very common. If the system neither resides in one of those countries, nor has any customers originating from them, then why not add them to the blacklist now rather than waiting?

Blacklisting Script and Configuration

So how do you do that? With FirewallD ipsets. I developed the following script to automate the process as much as possible:

#!/bin/bash
# Based on the below article
# https://www.linode.com/community/questions/11143/top-tip-firewalld-and-ipset-country-blacklist

# Source the blacklisted countries from the configuration file
. /etc/blacklist-by-country

# Create a temporary working directory
ipdeny_tmp_dir=$(mktemp -d -t blacklist-XXXXXXXXXX)
pushd $ipdeny_tmp_dir

# Download the latest network addresses by country file
curl -LO http://www.ipdeny.com/ipblocks/data/countries/all-zones.tar.gz
tar xf all-zones.tar.gz

# For updates, remove the ipset blacklist and recreate
if firewall-cmd -q --zone=drop --query-source=ipset:blacklist; then
    firewall-cmd -q --permanent --delete-ipset=blacklist
fi

# Create the ipset blacklist which accepts both IP addresses and networks
firewall-cmd -q --permanent --new-ipset=blacklist --type=hash:net \
    --option=family=inet --option=hashsize=4096 --option=maxelem=200000 \
    --set-description="An ipset list of networks or ips to be dropped."

# Add the address ranges by country per ipdeny.com to the blacklist
for country in $countries; do
    firewall-cmd -q --permanent --ipset=blacklist \
        --add-entries-from-file=./$country.zone && \
        echo "Added $country to blacklist ipset."
done

# Block individual IPs if the configuration file exists and is not empty
if [ -s "/etc/blacklist-by-ip" ]; then
    echo "Adding IPs blacklists."
    firewall-cmd -q --permanent --ipset=blacklist \
        --add-entries-from-file=/etc/blacklist-by-ip && \
        echo "Added IPs to blacklist ipset."
fi

# Add the blacklist ipset to the drop zone if not already setup
if firewall-cmd -q --zone=drop --query-source=ipset:blacklist; then
    echo "Blacklist already in firewalld drop zone."
else
    echo "Adding ipset blacklist to firewalld drop zone."
    firewall-cmd --permanent --zone=drop --add-source=ipset:blacklist
fi

firewall-cmd -q --reload

popd
rm -rf $ipdeny_tmp_dir

This should be installed to /usr/local/sbin and don’t forget to make it executable!

$ sudo chmod +x /usr/local/sbin/firewalld-blacklist

Then create a configure file: /etc/blacklist-by-country:

# Which countries should be blocked?
# Use the two letter designation separated by a space.
countries=""

And another configuration file /etc/blacklist-by-ip, which is just one IP per line without any additional formatting.

For this example 10 random countries were selected from the ipdeny zones:

# ls | shuf -n 10 | sed "s/\.zone//g" | tr '\n' ' '
nl ee ie pk is sv na om gp bn

Now as long as at least one country has been added to the config file it’s ready to run!

$ sudo firewalld-blacklist 
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   142  100   142    0     0   1014      0 --:--:-- --:--:-- --:--:--  1014
100  662k  100  662k    0     0   989k      0 --:--:-- --:--:-- --:--:--  989k
Added nl to blacklist ipset.
Added ee to blacklist ipset.
Added ie to blacklist ipset.
Added pk to blacklist ipset.
Added is to blacklist ipset.
Added sv to blacklist ipset.
Added na to blacklist ipset.
Added om to blacklist ipset.
Added gp to blacklist ipset.
Added bn to blacklist ipset.
Adding ipset blacklist to firewalld drop zone.
success

To verify that the firewalld blacklist was successful, check the drop zone and blacklist ipset:

$ sudo firewall-cmd --info-zone=drop
drop (active)
  target: DROP
  icmp-block-inversion: no
  interfaces: 
  sources: ipset:blacklist
  services: 
  ports: 
  protocols: 
  masquerade: no
  forward-ports: 
  source-ports: 
  icmp-blocks: 
  rich rules: 

$ sudo firewall-cmd --info-ipset=blacklist | less
blacklist
  type: hash:net
  options: family=inet hashsize=4096 maxelem=200000
  entries: 

The second command will output all of the subnets that were added based on the countries blocked and can be quite lengthy.

So now what do I do?

While it will be a good idea to monitor things more frequently at the beginning, over time the number of intrusion attempts should decline as the blacklist grows. Then the goal should be maintenance rather than active monitoring.

To this end I created a SystemD service file and timer so that on a monthly basis the by country subnets maintained by ipdeny are refreshed. In fact everything discussed here can be downloaded from my pagure.io project:

https://pagure.io/firewalld-blacklist

Aren’t you glad you read the whole article? Now just download the service file and timer to /etc/systemd/system/ and enable the timer:

$ sudo systemctl daemon-reload
$ sudo systemctl enable --now firewalld-blacklist.timer

Fedora Project community

18 Comments

  1. K. de Jong

    “If you run a server with a public-facing SST access”, do you mean SSH? If not, please add the acronym in full or a hyperlink for more info.

  2. Pieter

    Thank you for this great article and the firewalld-blacklist repo!

  3. Ron Olson

    Great article! FYI If you’re doing this from CentOS, you need to first run

    sudo dnf install epel-release

    to get the packages.

  4. Leo L. B.

    fail2ban doesnt protect against distributed bruteforce attacks coming from several hosts at the same time, which is what the bruteforce attempts you get over SSH are when you run a public facing server.

    Maybe you could talk about Port Knocking and how it can stop port scanning and therefore these distributed bruteforce attacks completely?

    Good article nonetheless!

    • Joao Rodrigues

      If you really want to stop worrying about brute force attacks, the only way is to completely disable password authentication and rely on public keys or client certificate authentication.

  5. Elio Qoshi

    Bonus points if you could change ‘Blacklist’ to a less discriminatory term, such as “Blocklist” or ‘Deny List’ 🙂

      • Although this article was planned and written some time before publication, and before people were starting to have discussions in IT about the use of these terms, that doesn’t make it any less of an issue. The editors are aware of the concern, and are having a discussion currently about new guidelines for standard usage, including using “allowlist/denylist.” Work is already underway on those guidelines. One thing to note in that discussion — there is plenty of input from many quarters already, but much of the input is not coming from editors. As with other open source communities, the people who do the work get to make the decisions on what to do here, and the editors seem to be largely in agreement about fixing this.

  6. Randy

    BLOCKlists 🙄 c’mon

  7. Luya Tshimbalanga

    Small correction. “Systemd” instead of “SystemD”.

  8. Glad to bе one of many visitants on tһis awing site
    :D.

  9. francesco

    Suddenly it looks like fail2ban doesn’t support IPv6 (I’m a Debian user, so I’m not sure about Fedora…)

    Here you can find more info:
    https://github.com/fail2ban/fail2ban/issues/39

    That’s why I uncomment the line:
    ListenAddress 0.0.0.0
    in the sshd config file (/etc/ssh/sshd_config on my system).

    sshd will then listen only on IPv4.

    • francesco

      I’m sorry, I didn’t notice that the issue was closed a few years ago… so fail2ban supports IPv6 too, now.

  10. Brooks Kelley

    Thank you for posting this article. How timely too!

    I had just upgraded from Fedora 29 to 32 ( yes, I know, big leap ).

    I have been used to using IPTABLES to exclude countries from accessing my SSH server. I was not looking forward to the learning curve for NFTABLES to do what I used to do for my SSH server with IPTABLES. I finally gave up and just secured my SSH server to not being accessible from the cloud when I saw this article posted.

    I have limited now my access to the US and CN. I also set the failure rate to 3 instead of 5. In addition, I extended the attempts time to two hours and the ban to two days.

    After installing the script and running it, I VPNed into France and tried to access my SSH server using my FQDN. No response! 😎

    Before adding the country blacklist though, I just ran fail2bain for a couple of days. I had somewhere around 345 bans in effect by the time I installed your scripts for firewalld and ipsets.

    What I would love to see is the article you promised following up on how if someone shows up again on the BAN list, you can extend their BANTIME even further or even permanently.

    I would really love to know how to do that. My server is a private server and I don’t mind making it as hard as possible to gain access via SSH. I have turned off direct root access. Of course, I could go further with keys and may do so. But, I would rather install some sort of two factor authentication over the use of keys.

    Thank you again for posting this!

  11. Marcwa

    I cam across this and in the first shot it was not working for me using Fedora 32.
    I noticed, that i added the Country-Lists with uppercase characters in /etc/blacklist-by-country.

    So i adjusted the Code of firewalld-blacklist a little bit, that it converts the country code to lowercase characters.

    To do so, just change on line from:
    “–add-entries-from-file=./$country.zone && \”
    to
    “–add-entries-from-file=./${country,,}.zone && \”

    Now it is working perfectly! Thank you very much for this guide.

  12. Brooks Kelley

    Last night, I had to shut down my server. It’s okay. It is just a toy I use to learn on trying out server software. Our home network’s LAN was experiencing a data storm and devices weren’t working. Afterwards, things returned to normal on the network. I went into the cable router and shutdown port forwarding to my server.

    Upon examining the fail2ban.log file this morning, I found the fail2ban service restarted around 9 in the morning, which is odd since I was leaving the system alone. Twenty-five minutes later everything shut down in the log file for the remainder of the day. Of course, this left my system vulnerable for the rest of the day to a brute force attack. By the evening, my network was in disarray. I suspect a vulnerability of fail2ban has been found.

    I went to their website and was shocked to find that it hasn’t been updated since 2015.

    The worse part is that fail2ban shuts down logging on ‘secure’ so I couldn’t see what was going on with my ssh server independently of my fail2ban service after it failed.

    This is the last of my fail2ban log file. The abuser is from China. …

    2020-07-04 09:25:30,175 fail2ban.actions [767]: NOTICE [sshd] Restore Ban 106.12.88.165
    2020-07-04 09:26:30,212 fail2ban.utils [767]: ERROR 7f974829f630 — exec: ports=”ssh”; for p in $(echo $ports | tr “, ” ” “); do firewall-cmd –add-rich-rule=”rule family=’ipv4′ source address=’106.12.88.165′ port port=’$p’ protocol=’tcp’ reject type=’icmp-port-unreachable'”; done
    2020-07-04 09:26:30,213 fail2ban.utils [767]: ERROR 7f974829f630 — timed out after 60 seconds.
    2020-07-04 09:26:30,413 fail2ban.utils [767]: ERROR ports=”ssh”; for p in $(echo $ports | tr “, ” ” “); do firewall-cmd –add-rich-rule=”rule family=’ipv4′ source address=’106.12.88.165′ port port=’$p’ protocol=’tcp’ reject type=’icmp-port-unreachable'”; done — failed with [Errno 3] No such process
    2020-07-04 09:26:30,413 fail2ban.utils [767]: ERROR 7f974829f630 — stderr: ‘ERROR:dbus.proxies:Introspect error on :1.7:/org/fedoraproject/FirewallD1: dbus.exceptions.DBusException: org.freedesktop.DBus.Error.NoReply: Did not receive a reply. Possible causes include: the remote application did not send a reply, the message bus security policy blocked the reply, the reply timeout expired, or the network connection was broken.’
    2020-07-04 09:26:30,413 fail2ban.utils [767]: ERROR 7f974829f630 — killed with SIGTERM (return code: -15)
    2020-07-04 09:26:30,414 fail2ban.actions [767]: ERROR Failed to execute ban jail ‘sshd’ action ‘firewallcmd-rich-rules’ info ‘ActionInfo({‘ip’: ‘106.12.88.165’, ‘family’: ‘inet4’, ‘fid’: <function Actions.ActionInfo. at 0x7f9748fcfe50>, ‘raw-ticket’: <function Actions.ActionInfo. at 0x7f9748fd0550>})’: Error banning 106.12.88.165

  13. Tim

    If you want to block visitors by country using ipset, you can find the instructions in https://blog.ip2location.com/knowledge-base/how-to-block-ip-addresses-from-a-country-using-ipset/

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.

The opinions expressed on this website are those of each author, not of the author's employer or of Red Hat. Fedora Magazine aspires to publish all content under a Creative Commons license but may not be able to do so in all cases. You are responsible for ensuring that you have the necessary permission to reuse any work on this site. The Fedora logo is a trademark of Red Hat, Inc. Terms and Conditions

%d bloggers like this: