Firewalld and how to preserve the original source IP when forwarding to internal IP

Using firewalld and the forwarding options (IP or port forward) might work not as expected if the default setup is left on the system. Consider the simple example:

main menu
Internet <-> router <-> local network

The purpose is to forward a port to a server in the local network, which should be easy enough. Let the forwarding port be 80 and the server should receive the original source IP. To archive this task the system administrator should do the following on the router with firewalld service. Here is one of the simplest methods:

  • When the router’s external IP/interface and the router’s internal IP/interface are in the same firewalld zone. The zone is named “public” in the CentOS world.

The solution uses the masquerade rule added with a rich rule (–add-rich-rule), not the masquerade option of the zone (–add-masquerade).
The default configuration will assign the external interface and the internal interface, which may be a virtual one, in the same firewalld zone such as “public”. When this happens, activating the masquerade option will break the source IP and it will be replaced by the Netfilter with the internal IP address of the router and the internal server will see all incoming connections on the forwarded port as if they were coming from the internal router IP. All different IPs coming to this port will be replaced with the router’s internal IP and forwarded to the server’s internal.

The router’s external IP/interface and the router’s internal IP/interface are in the same firewalld zone.

This solution is demonstrated with a virtual interface – bridge br0, but it may be a network interface. By default, when the bridge is created, it will be added to the default zone, which is “public” in CentOS world. Use –get-active-zones to check the active zones and the assigned interfaces.

[root@srv ~]# firewall-cmd --get-active-zones
public
  interfaces: eth0 br0
[root@srv ~]# firewall-cmd --list-all
public (active)
  target: default
  icmp-block-inversion: no
  interfaces: br0 eth0
  sources: 
  services: cockpit dhcpv6-client http https ssh
  ports: 10022/tcp
  protocols: 
  forward: yes
  masquerade: no
  forward-ports: 
  source-ports: 
  icmp-blocks: 
  rich rules:

If the options forward and masquerade are activated (i.e. yes on the above output) and a forward rule to an internal local IP (some server IP connected to the bridge br0) is introduced to the firewall, the local server will receive all connection attempts to the forwarded port, but the source IP will be overwritten with the gateway IP of the internal (local network). For example, the bridge br0 has IP 192.168.0.1 and the eth0 has Internet IP 1.1.1.1. Forwarding port 1.1.1.1:80 to a server behind the bridge br0 with IP 192.168.0.100:
Keep on reading!

firewalld and podman (or docker) – no internet in the container and could not resolve host

If you happen to use CentOS 8 you have already discovered that Red Hat (i.e. CentOS) switch to podman, which is a fork of docker. So probably the following fix might help to someone, which does not use CentOS 8 or podman. For now, podman and docker are 99.99% the same.
So creating and starting a container is easy and in most cases one command only, but you may stumble on the error your container could not resolve or could not connect to an IP even there is a ping to the IP!
The service in the container may live a happy life without Internet access but just the mapped ports from the outside world. Still, it may happen to need Internet access, let’s say if an update should be performed.
Here is how to fix podman (docker) missing the Internet access in the container:

  • No ping to the outside world. The chances you are missing
    sysctl -w net.ipv4.ip_forward=1
    

    And do not forget to make it permanent by adding the “net.ipv4.ip_forward=1” to /etc/sysctl.conf (or a file “.conf” in /etc/sysctl.d/).

  • ping to the outside IP of the container is available, but no connection to any service is available! Probably the NAT is not enabled in your podman docker configuration. In the case with firewalld, at least, you must enable the masquerade option of the public zone
    firewall-cmd --zone=public --add-masquerade
    firewall-cmd --permanent --zone=public --add-masquerade
    

    The second command with “–permanent” is to make the option permanent over reboots.

The error – Could not resolve host (Name or service not known) despite having servers in /etc/resolv.conf and ping to them!

One may think having IPs in /etc/resolv.conf and ping to them in the container should give the container access to the Internet. But the following error occurs:

[root@srv /]# yum install telnet
Loaded plugins: fastestmirror, ovl
Determining fastest mirrors
 * base: artfiles.org
 * extras: centos.mirror.net-d-sign.de
 * updates: centos.bio.lmu.de
http://mirror.fra10.de.leaseweb.net/centos/7.7.1908/os/x86_64/repodata/repomd.xml: [Errno 14] curl#6 - "Could not resolve host: mirror.fra10.de.leaseweb.net; Unknown error"
Trying other mirror.
http://artfiles.org/centos.org/7.7.1908/os/x86_64/repodata/repomd.xml: [Errno 14] curl#6 - "Could not resolve host: artfiles.org; Unknown error"
Trying other mirror.
^C

Exiting on user cancel
[root@srv /]# ^C
[root@srv /]# ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
64 bytes from 8.8.8.8: icmp_seq=1 ttl=56 time=5.05 ms
64 bytes from 8.8.8.8: icmp_seq=2 ttl=56 time=5.06 ms
^C
--- 8.8.8.8 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1000ms
rtt min/avg/max/mdev = 5.050/5.055/5.061/0.071 ms
[root@srv ~]# cat /etc/resolv.conf 
nameserver 8.8.8.8
nameserver 8.8.4.4
[root@srv /]# ping google.com
ping: google.com: Name or service not known

The error 2 – Can’t connect to despite having ping to the IP!

[root@srv /]# ping 2.2.2.2
PING 2.2.2.2 (2.2.2.2) 56(84) bytes of data.
64 bytes from 2.2.2.2: icmp_seq=1 ttl=56 time=9.15 ms
64 bytes from 2.2.2.2: icmp_seq=2 ttl=56 time=9.16 ms
^C
[root@srv2 /]# mysql -h2.2.2.2 -uroot -p
Enter password: 
ERROR 2003 (HY000): Can't connect to MySQL server on '2.2.2.2' (113)
[root@srv2 /]#

Despite having ping the MySQL server on 2.2.2.2 and despite the firewall on 2.2.2.2 allows outside connections the container could not connect to it. And testing other services like HTTP, HTTPS, FTP and so on resulted in “unable to connect“, too. Simply because the NAT (aka masquerade is not enabled in the firewall).