firewalld | Any IT here? Help Me!

Firewalld and how to preserve the original source IP when forwarding to internal IP

Using firewalld and the forwarding options (IP or port forward) might work not as expected if the default setup is left on the system. Consider the simple example:

main menu — Internet <-> router <-> local network

The purpose is to forward a port to a server in the local network, which should be easy enough. Let the forwarding port be 80 and the server should receive the original source IP. To archive this task the system administrator should do the following on the router with firewalld service. Here is one of the simplest methods:

When the router’s external IP/interface and the router’s internal IP/interface are in the same firewalld zone. The zone is named “public” in the CentOS world.

The solution uses the masquerade rule added with a rich rule (–add-rich-rule), not the masquerade option of the zone (–add-masquerade).
The default configuration will assign the external interface and the internal interface, which may be a virtual one, in the same firewalld zone such as “public”. When this happens, activating the masquerade option will break the source IP and it will be replaced by the Netfilter with the internal IP address of the router and the internal server will see all incoming connections on the forwarded port as if they were coming from the internal router IP. All different IPs coming to this port will be replaced with the router’s internal IP and forwarded to the server’s internal.

The router’s external IP/interface and the router’s internal IP/interface are in the same firewalld zone.

This solution is demonstrated with a virtual interface – bridge br0, but it may be a network interface. By default, when the bridge is created, it will be added to the default zone, which is “public” in CentOS world. Use –get-active-zones to check the active zones and the assigned interfaces.

[root@srv ~]# firewall-cmd --get-active-zones
public
  interfaces: eth0 br0
[root@srv ~]# firewall-cmd --list-all
public (active)
  target: default
  icmp-block-inversion: no
  interfaces: br0 eth0
  sources: 
  services: cockpit dhcpv6-client http https ssh
  ports: 10022/tcp
  protocols: 
  forward: yes
  masquerade: no
  forward-ports: 
  source-ports: 
  icmp-blocks: 
  rich rules:

If the options forward and masquerade are activated (i.e. yes on the above output) and a forward rule to an internal local IP (some server IP connected to the bridge br0) is introduced to the firewall, the local server will receive all connection attempts to the forwarded port, but the source IP will be overwritten with the gateway IP of the internal (local network). For example, the bridge br0 has IP 192.168.0.1 and the eth0 has Internet IP 1.1.1.1. Forwarding port 1.1.1.1:80 to a server behind the bridge br0 with IP 192.168.0.100:
Keep on reading!

firewalld and podman (or docker) – no internet in the container and could not resolve host

If you happen to use CentOS 8 you have already discovered that Red Hat (i.e. CentOS) switch to podman, which is a fork of docker. So probably the following fix might help to someone, which does not use CentOS 8 or podman. For now, podman and docker are 99.99% the same.
So creating and starting a container is easy and in most cases one command only, but you may stumble on the error your container could not resolve or could not connect to an IP even there is a ping to the IP!
The service in the container may live a happy life without Internet access but just the mapped ports from the outside world. Still, it may happen to need Internet access, let’s say if an update should be performed.
Here is how to fix podman (docker) missing the Internet access in the container:

No ping to the outside world. The chances you are missing
```
sysctl -w net.ipv4.ip_forward=1
```
And do not forget to make it permanent by adding the “net.ipv4.ip_forward=1” to /etc/sysctl.conf (or a file “.conf” in /etc/sysctl.d/).
ping to the outside IP of the container is available, but no connection to any service is available! Probably the NAT is not enabled in your podman docker configuration. In the case with firewalld, at least, you must enable the masquerade option of the public zone
```
firewall-cmd --zone=public --add-masquerade
firewall-cmd --permanent --zone=public --add-masquerade
```
The second command with “–permanent” is to make the option permanent over reboots.

The error – Could not resolve host (Name or service not known) despite having servers in /etc/resolv.conf and ping to them!

One may think having IPs in /etc/resolv.conf and ping to them in the container should give the container access to the Internet. But the following error occurs:

[root@srv /]# yum install telnet
Loaded plugins: fastestmirror, ovl
Determining fastest mirrors
 * base: artfiles.org
 * extras: centos.mirror.net-d-sign.de
 * updates: centos.bio.lmu.de
http://mirror.fra10.de.leaseweb.net/centos/7.7.1908/os/x86_64/repodata/repomd.xml: [Errno 14] curl#6 - "Could not resolve host: mirror.fra10.de.leaseweb.net; Unknown error"
Trying other mirror.
http://artfiles.org/centos.org/7.7.1908/os/x86_64/repodata/repomd.xml: [Errno 14] curl#6 - "Could not resolve host: artfiles.org; Unknown error"
Trying other mirror.
^C

Exiting on user cancel
[root@srv /]# ^C
[root@srv /]# ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
64 bytes from 8.8.8.8: icmp_seq=1 ttl=56 time=5.05 ms
64 bytes from 8.8.8.8: icmp_seq=2 ttl=56 time=5.06 ms
^C
--- 8.8.8.8 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1000ms
rtt min/avg/max/mdev = 5.050/5.055/5.061/0.071 ms
[root@srv ~]# cat /etc/resolv.conf 
nameserver 8.8.8.8
nameserver 8.8.4.4
[root@srv /]# ping google.com
ping: google.com: Name or service not known

The error 2 – Can’t connect to despite having ping to the IP!

[root@srv /]# ping 2.2.2.2
PING 2.2.2.2 (2.2.2.2) 56(84) bytes of data.
64 bytes from 2.2.2.2: icmp_seq=1 ttl=56 time=9.15 ms
64 bytes from 2.2.2.2: icmp_seq=2 ttl=56 time=9.16 ms
^C
[root@srv2 /]# mysql -h2.2.2.2 -uroot -p
Enter password: 
ERROR 2003 (HY000): Can't connect to MySQL server on '2.2.2.2' (113)
[root@srv2 /]#

Despite having ping the MySQL server on 2.2.2.2 and despite the firewall on 2.2.2.2 allows outside connections the container could not connect to it. And testing other services like HTTP, HTTPS, FTP and so on resulted in “unable to connect“, too. Simply because the NAT (aka masquerade is not enabled in the firewall).

Receive multicast packets on CentOS 7 (and other linux distros)

There are so many web pages and blogs post for multicast traffic under linux and how to enable it, but in most of them something always is missing and if you follow them probably you’ll end up with not working setup and you’ll have to search the Internet again – some do need tuning of the linux kernel variables not to drop packets, some need tuning the firewall to allow protocols.
Here we present a real working example of a server under CentOS 7, our server has two network ports:

eno1 – local unicast traffic with local IP
eno2 – multicast traffic

We have multicast TV streams, which we can use through our second network interface and we want to use ffmpeg to encode the video. We have “Multicast Group:port” for every stream, which is like “IP:PORT” and in our case the port is always the same 5000. Here are the steps you need to do if you want to receive these streams.

STEP 1) Set networking and make the configuration permanent.

In CentOS 7 the network of eno2, set a local IP, in fact it does not matter the exact IP, we used 10.10.10.0/24 local network.

/etc/sysconfig/network-scripts/ifcfg-eno2

TYPE=Ethernet
BOOTPROTO=none
DEFROUTE=yes
IPV4_FAILURE_FATAL=no
IPV6INIT=yes
IPV6_AUTOCONF=yes
IPV6_DEFROUTE=yes
IPV6_FAILURE_FATAL=no
NAME=eno2
UUID=2481b907-5e6e-45f9-ab96-7091e4e7d6d1
ONBOOT=yes
HWADDR=0c:c4:7a:44:87:a5
IPADDR0=10.10.10.152
PREFIX0=24
IPV6_PEERDNS=yes
IPV6_PEERROUTES=yes
NM_CONTROLLED=no

The important lines are highlighted. Configure the network adapter on boot and set 10.10.10.152.
Second and very important add a static route for the multicast traffic for the network interface, which is supposed to have the multicast streams (in our case “eno2”). Use file:

/etc/sysconfig/network-scripts/route-eno2

224.0.0.0/4 dev eno2

STEP 2) Kernel variables tuning

[srv@local ~]# for i in /proc/sys/net/ipv4/conf/*/rp_filter ; do echo 0 > "$i";   done
[srv@local ~]# echo "0" > /proc/sys/net/ipv4/icmp_echo_ignore_broadcasts

Turn off “Reverse Path Filtering” for all interfaces (rp_filter) and ping for the multicast address to work properly we need to disable icmp_echo_ignore_broadcasts. By default kernel drops these packets so you need this tuning or your application will not receive a single udp packet.
You could save the above two lines in

/etc/rc.local

and do not forget to set proper access rights:

[srv@local ~]# chmod 755 /etc/rc.local

Or you can use

/etc/sysctl.conf

Add the following lines in it:

net.ipv4.conf.default.rp_filter=0
net.ipv4.conf.all.rp_filter=0
net.ipv4.conf.eno2.rp_filter=0
net.ipv4.icmp_echo_ignore_broadcasts=0

STEP 3) Allow UDP traffic (and/or IGMP) if you have firewall

Here the port is the “Multicast Group:port” of the IP your application will join (look the example below)

firewalld – the default with CentOS 7:

[srv@local ~]# firewall-cmd --new-zone=multicast --permanent
[srv@local ~]# firewall-cmd --zone=multicast --add-interface=eno2 --permanent
[srv@local ~]# firewall-cmd --zone=multicast --add-protocol=igmp --permanent
[srv@local ~]# firewall-cmd --zone=multicast --add-protocol=icmp --permanent
[srv@local ~]# firewall-cmd --zone=multicast --add-source=224.0.0.0/4 --permanent
[srv@local ~]# firewall-cmd --zone=multicast --add-source=10.10.10.0/24 --permanent
[srv@local ~]# firewall-cmd --zone=multicast --add-port=5000/udp --permanent
[srv@local ~]# firewall-cmd --reload

We make a new zone for the multicast streams and add our interface, the sources and the port we use. We added IGMP, because in some use cases it is needed (in our it could work without IGMP added)

iptables – if you have disabled firewall and still you wanted to have a firewall:
```
# allow multicast addresses
[srv@local ~]# iptables -A INPUT -p udp -d 5000 -j ACCEPT
[srv@local ~]# iptables -A INPUT -s 224.0.0.0/4 -j ACCEPT
[srv@local ~]# iptables -A INPUT -p igmp -d 224.0.0.0/4 -j ACCEPT
[srv@local ~]# iptables -A INPUT -p icmp --icmp-type 0 -j ACCEPT
```
Probably it is a good idea to see if you have current rules (with “iptables -L -v -n” or even “iptables-save”) and to see if you should use “-A” (above) or “-I” to insert the rules above the DROP rule(s).

* Example with ffmpeg joining to a multicast group

[srv@local ~]# ffmpeg -i 'udp://239.100.10.5:5000'
ffmpeg version 2.8.6 Copyright (c) 2000-2016 the FFmpeg developers
  built with gcc 5.3.0 (Gentoo 5.3.0 p1.0, pie-0.6.5)
  configuration: --prefix=/usr --libdir=/usr/lib64 --shlibdir=/usr/lib64 --mandir=/usr/share/man --enable-shared --cc=x86_64-pc-linux-gnu-gcc --cxx=x86_64-pc-linux-gnu-g++ --ar=x86_64-pc-linux-gnu-ar --optflags='-march=native -O2 -msse3 -fomit-frame-pointer -pipe' --disable-static --enable-avfilter --enable-avresample --disable-stripping --enable-nonfree --enable-version3 --enable-nonfree --disable-indev=alsa --disable-indev=oss --disable-outdev=alsa --disable-outdev=oss --enable-version3 --enable-bzlib --disable-runtime-cpudetect --disable-debug --disable-doc --disable-gnutls --enable-gpl --enable-hardcoded-tables --enable-iconv --disable-lzma --enable-network --enable-openssl --enable-postproc --disable-libsmbclient --enable-ffplay --enable-sdl --disable-vaapi --disable-vdpau --enable-xlib --disable-libxcb --disable-libxcb-shm --disable-libxcb-xfixes --enable-zlib --disable-libcdio --disable-libiec61883 --disable-libdc1394 --enable-libcaca --disable-openal --disable-opengl --disable-libv4l2 --disable-libpulse --enable-libopencore-amrwb --enable-libopencore-amrnb --disable-libfdk-aac --enable-libopenjpeg --disable-libbluray --disable-libcelt --disable-libgme --enable-libgsm --disable-libmodplug --disable-libopus --disable-libquvi --disable-librtmp --disable-libssh --disable-libschroedinger --disable-libspeex --enable-libvorbis --enable-libvpx --disable-libzvbi --disable-libbs2b --disable-libflite --disable-frei0r --disable-libfribidi --enable-fontconfig --disable-ladspa --disable-libass --enable-libfreetype --disable-libsoxr --enable-pthreads --enable-libvo-aacenc --disable-libvo-amrwbenc --enable-libmp3lame --disable-libaacplus --enable-libfaac --disable-libsnappy --enable-libtheora --disable-libtwolame --disable-libwavpack --disable-libwebp --enable-libx264 --disable-libx265 --enable-libxvid --enable-x11grab --disable-amd3dnow --disable-amd3dnowext --disable-fma4 --disable-xop --cpu=host
  libavutil      54. 31.100 / 54. 31.100
  libavcodec     56. 60.100 / 56. 60.100
  libavformat    56. 40.101 / 56. 40.101
  libavdevice    56.  4.100 / 56.  4.100
  libavfilter     5. 40.101 /  5. 40.101
  libavresample   2.  1.  0 /  2.  1.  0
  libswscale      3.  1.101 /  3.  1.101
  libswresample   1.  2.101 /  1.  2.101
  libpostproc    53.  3.100 / 53.  3.100
[mpeg2video @ 0xf2fa80] Invalid frame dimensions 0x0.
    Last message repeated 10 times
Input #0, mpegts, from 'udp://239.100.10.5:5000':
  Duration: N/A, start: 87846.990933, bitrate: 5659 kb/s
  Program 5 
    Metadata:
      service_name    : ?TVtest
      service_provider: ?ss
    Stream #0:0[0x33]: Video: mpeg2video (Main) ([2][0][0][0] / 0x0002), yuv420p(tv), 720x576 [SAR 64:45 DAR 16:9], 5467 kb/s, 25 fps, 25 tbr, 90k tbn, 50 tbc
    Stream #0:1[0x34](bul): Audio: mp2 ([3][0][0][0] / 0x0003), 48000 Hz, stereo, s16p, 192 kb/s
At least one output file must be specified

As you can see to join the multicast group and ffmpeg to start encoding you use “udp://239.100.10.5:5000” for input stream parameter. Here ffmpeg joins the group and receives packets successfully.

* Generic example to receive UDP multicast stream

You may use the following example in any linux distro like Ubuntu, CentOS 7, Gentoo, OpenSuse and many others to receive muticast streams

[srv@local ~]# #ifconfig or ip - use one of them
[srv@local ~]# #ifconfig
[srv@local ~]# ifconfig eno1 10.10.10.152/24 up
[srv@local ~]# route add -net 224.0.0.0 netmask 240.0.0.0 eno2
[srv@local ~]# #or ip
[srv@local ~]# ip addr add 10.10.10.152/24 dev eno2
[srv@local ~]# ip link set eno2 up
[srv@local ~]# #allow multicast packets to the server
[srv@local ~]# for i in /proc/sys/net/ipv4/conf/*/rp_filter ; do echo 0 > "$i";   done
[srv@local ~]# echo "0" > /proc/sys/net/ipv4/icmp_echo_ignore_broadcasts
[srv@local ~]# #if you have firewall use the following for the iptables OR firewalld after that - use only one of them!
[srv@local ~]# iptables -A INPUT -p udp -d 5000 -j ACCEPT
[srv@local ~]# iptables -A INPUT -s 224.0.0.0/4 -j ACCEPT
[srv@local ~]# iptables -A INPUT -p igmp -d 224.0.0.0/4 -j ACCEPT
[srv@local ~]# iptables -A INPUT -p icmp --icmp-type 0 -j ACCEPT
[srv@local ~]# #firewalld setup
[srv@local ~]# firewall-cmd --new-zone=multicast --permanent
[srv@local ~]# firewall-cmd --zone=multicast --add-interface=eno2 --permanent
[srv@local ~]# firewall-cmd --zone=multicast --add-protocol=igmp --permanent
[srv@local ~]# firewall-cmd --zone=multicast --add-protocol=icmp --permanent
[srv@local ~]# firewall-cmd --zone=multicast --add-source=224.0.0.0/4 --permanent
[srv@local ~]# firewall-cmd --zone=multicast --add-source=10.10.10.0/24 --permanent
[srv@local ~]# firewall-cmd --zone=multicast --add-port=5000/udp --permanent
[srv@local ~]# firewall-cmd --reload