Firewalld and how to preserve the original source IP when forwarding to internal IP

Using firewalld and the forwarding options (IP or port forward) might work not as expected if the default setup is left on the system. Consider the simple example:

The purpose is to forward a port to a server in the local network, which should be easy enough. Let the forwarding port be 80 and the server should receive the original source IP. To archive this task the system administrator should do the following on the router with firewalld service. Here is one of the simplest methods:

• When the router’s external IP/interface and the router’s internal IP/interface are in the same firewalld zone. The zone is named “public” in the CentOS world.

The solution uses the masquerade rule added with a rich rule (–add-rich-rule), not the masquerade option of the zone (–add-masquerade).
The default configuration will assign the external interface and the internal interface, which may be a virtual one, in the same firewalld zone such as “public”. When this happens, activating the masquerade option will break the source IP and it will be replaced by the Netfilter with the internal IP address of the router and the internal server will see all incoming connections on the forwarded port as if they were coming from the internal router IP. All different IPs coming to this port will be replaced with the router’s internal IP and forwarded to the server’s internal.

The router’s external IP/interface and the router’s internal IP/interface are in the same firewalld zone.

This solution is demonstrated with a virtual interface – bridge br0, but it may be a network interface. By default, when the bridge is created, it will be added to the default zone, which is “public” in CentOS world. Use –get-active-zones to check the active zones and the assigned interfaces.

[root@srv ~]# firewall-cmd --get-active-zones
public
  interfaces: eth0 br0
[root@srv ~]# firewall-cmd --list-all
public (active)
  target: default
  icmp-block-inversion: no
  interfaces: br0 eth0
  sources: 
  services: cockpit dhcpv6-client http https ssh
  ports: 10022/tcp
  protocols: 
  forward: yes
  masquerade: no
  forward-ports: 
  source-ports: 
  icmp-blocks: 
  rich rules:

If the options forward and masquerade are activated (i.e. yes on the above output) and a forward rule to an internal local IP (some server IP connected to the bridge br0) is introduced to the firewall, the local server will receive all connection attempts to the forwarded port, but the source IP will be overwritten with the gateway IP of the internal (local network). For example, the bridge br0 has IP 192.168.0.1 and the eth0 has Internet IP 1.1.1.1. Forwarding port 1.1.1.1:80 to a server behind the bridge br0 with IP 192.168.0.100:

[root@srv ~]# firewall-cmd --get-active-zones
public
  interfaces: eth0 br0
[root@srv ~]# firewall-cmd --list-all
public (active)
  target: default
  icmp-block-inversion: no
  interfaces: br0 eth0
  sources: 
  services: cockpit dhcpv6-client http https ssh
  ports: 10022/tcp
  protocols: 
  forward: yes
  masquerade: no
  forward-ports: 
  source-ports: 
  icmp-blocks: 
  rich rules:
[root@srv ~]# firewall-cmd --permanent --zone=public --add-rich-rule="rule family="ipv4" source address="192.168.0.0/24" masquerade"
success
[root@srv ~]# firewall-cmd --permanent --zone=public --add-forward-port=port=80:proto=tcp:toport=80:toaddr=192.168.0.100
success
[root@srv ~]# firewall-cmd --reload
success
[root@srv ~]# firewall-cmd --zone=public --list-all
public (active)
  target: default
  icmp-block-inversion: no
  interfaces: br0 eth0
  sources: 
  services: cockpit dhcpv6-client http https ssh
  ports: 10022/tcp
  protocols: 
  forward: yes
  masquerade: no
  forward-ports: 
        port=80:proto=tcp:toport=80:toaddr=192.168.0.100
  source-ports: 
  icmp-blocks: 
  rich rules:
        rule family="ipv4" source address="192.168.0.0/24" masquerade

First, add the masquerade only for source addresses with 192.168.0.0/24 (i.e. originating from the local network) and then add a port forward for all connections on port 80 to go to internal IP 192.168.0.100 on the same port 80. Such configuration will ensure, the internal server will receive the original source IP of the connections to the port 80, not overwritten IP of the internal network gateway 192.168.0.1. In the server logs of the internal server, connections from the Internet will be saved with the real Internet remote IPs:

197.237.33.172 - - [09/Jun/2023:15:53:37 +0200] "GET /images/1547.png HTTP/1.1" 304 0 "-" "okhttp/3.10.0" "-"
154.73.111.146 - - [09/Jun/2023:15:53:37 +0200] "GET /images/17266.png HTTP/1.1" 200 112534 "-" "okhttp/3.10.0" "-"
105.161.222.89 - - [09/Jun/2023:15:53:37 +0200] "GET /images/19196.png HTTP/1.1" 200 22418 "-" "okhttp/3.10.0" "-"
105.161.222.89 - - [09/Jun/2023:15:53:37 +0200] "GET /images/1025.png HTTP/1.1" 200 32119 "-" "okhttp/3.10.0" "-"

On the contrary, if just enable masquerade with “–add-masquerade” and use the same forward rule as above:

[root@srv ~]# firewall-cmd --get-active-zones
public
  interfaces: eth0 br0
[root@srv ~]# firewall-cmd --list-all
public (active)
  target: default
  icmp-block-inversion: no
  interfaces: br0 eth0
  sources: 
  services: cockpit dhcpv6-client http https ssh
  ports: 10022/tcp
  protocols: 
  forward: yes
  masquerade: no
  forward-ports: 
  source-ports: 
  icmp-blocks: 
  rich rules:
[root@srv ~]# firewall-cmd --permanent --add-masquerade
success
[root@srv ~]# firewall-cmd --permanent --zone=public --add-forward-port=port=80:proto=tcp:toport=80:toaddr=192.168.0.100
success
[root@srv ~]# firewall-cmd --reload
success
[root@srv ~]# firewall-cmd --zone=public --list-all
public (active)
  target: default
  icmp-block-inversion: no
  interfaces: br0 eth0
  sources: 
  services: cockpit dhcpv6-client http https ssh
  ports: 10022/tcp
  protocols: 
  forward: yes
  masquerade: yes
  forward-ports: 
        port=80:proto=tcp:toport=80:toaddr=192.168.0.100
  source-ports: 
  icmp-blocks: 
  rich rules:

The servers logs will look:

192.168.0.1 - - [09/Jun/2023:15:55:21 +0200] "GET /images/1547.png HTTP/1.1" 304 0 "-" "okhttp/3.10.0" "-"
192.168.0.1 - - [09/Jun/2023:15:55:21 +0200] "GET /images/17266.png HTTP/1.1" 200 112534 "-" "okhttp/3.10.0" "-"
192.168.0.1 - - [09/Jun/2023:15:55:21 +0200] "GET /images/19196.png HTTP/1.1" 200 22418 "-" "okhttp/3.10.0" "-"
192.168.0.1 - - [09/Jun/2023:15:55:21 +0200] "GET /images/1025.png HTTP/1.1" 200 32119 "-" "okhttp/3.10.0" "-"

No original source IP, which may be a big problem for the software behind the router. And if it is easy to fix it with web proxy services, it is not so easy for all other services like ssh, SMTP, IMAP, and so on. In fact, letting the original source IP to the local server means the server may enforce additional firewall policies.

There are more solutions than this one, but they are much more complicated such as two separate zones for the external and internal networks, which must include rules for ingress and egress packets rules to transfer packets between zones (use policy objects in firewalld) or another way is to add direct rules in iptables format to use SNAT and DNAT targets (though firewalld may use nftables as backend!).
This is a typical use case when using LXC with a bridge device with internal IPs to the containers. Here is how to configure the network – Run LXC Ubuntu 22.04 LTS container with bridged network under CentOS Stream 9, Run LXC Ubuntu 22.04 LTS container with bridged network under CentOS Stream 9, Howto do QEMU full virtualization with bridged networking and more.