iptables | Any IT here? Help Me!

Cron missing path – executing docker/podman – adding network: failed to locate iptables

If you have ever happened to execute some complex scripts using the cron system you were inevitable to discover the Linux environment was different than the login or ssh shell. The different environment tends to lead to a missing or different PATH environment! Here is what happens with podman starting a container from a cron script:

time="2020-04-19T20:45:20Z" level=error msg="Error adding network: failed to locate iptables: exec: \"iptables\": executable file not found in $PATH"
time="2020-04-19T20:45:20Z" level=error msg="Error while adding pod to CNI network \"podman\": failed to locate iptables: exec: \"iptables\": executable file not found in $PATH"
Error: unable to start container "onedrive-cli": error configuring network namespace for container d297cf80db20441d4258a1acc7d810444795d1ca8730ab242d9fe8a13eaa697d: failed to locate iptables: exec: "iptables": executable file not found in $PATH

The iptables executable is missing because the PATH variable is different than the login or ssh shell one. Executing the commands or the script under ssh or login will result in no error and a proper podman (docker) execution!

A similar problem could have happened with another software trying to execute iptables or another tool, which is not found in the cron’s PATH environment because cron’s environment is very limited and

To ensure the PATH is like the user’s (root) environment just source the “profile” or “.bashrc” file of the current user before the execution of the script or in the first lines of it.
This would do the trick.

. /etc/profile

Or user’s custom

. ~/.bashrc

Or the default OS bashrc

. /etc/bashrc

The dot may be replaced by “source”:

source /etc/bashrc

All (environment) variables will be available after the source command.

Here is the difference:
The environment without the sourcing profile/bashrc file:

 
LANG=en_US.UTF-8
XDG_SESSION_ID=19118
USER=root
PWD=/root
HOME=/root
SHELL=/bin/sh
SHLVL=1
LOGNAME=root
DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/0/bus
XDG_RUNTIME_DIR=/run/user/0
PATH=/usr/bin:/bin
_=/usr/bin/env

Sourcing the “/etc/profile” file:

LANG=en_US.UTF-8
HISTCONTROL=ignoredups
HOSTNAME=srv.example.com
XDG_SESSION_ID=19165
USER=root
PWD=/root
HOME=/root
MAIL=/var/spool/mail/root
SHELL=/bin/bash
SHLVL=1
LOGNAME=root
DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/0/bus
XDG_RUNTIME_DIR=/run/user/0
PATH=/usr/local/sbin:/usr/sbin:/usr/bin:/bin
HISTSIZE=1000
LESSOPEN=||/usr/bin/lesspipe.sh %s
_=/usr/bin/env

Multiple additional envrinment varibles, which could be important for user’s scripts executed by the cron.

And in CentOS 8 the iptables happens to be in “/usr/sbin/iptables” – a path /usr/sbin not included in the default cron environment PATH variable!
Of course, the PATH environment may be edited in the cron scheduler with crontab (by just setting the PATH with a path) till the next path missing in it and included in the user’s path! It’s just better to ensure the two environments are the same every time by sourcing the environment configuration file such as /etc/profile or user’s bashrc (or the default on in /etc/bashrc?).

Save iptables rules over reboots on Ubuntu 16 and Ubuntu 18 – persistent iptables rules

Moving towards the firewalld software and especially the systemd some good old init scripts got missing! For example, one of those good scripts is the init script for iptables firewall, which allows saving iptables rules and during boot, it loads them again. With the init iptables script we have persistence of the iptables rules. Meanwhile, we can always call the init script with “save” argument to update the currently saved rules. Many different Linux distributions have this init script – “/etc/init.d/iptables”, but in systemd world, it has been removed and replaced with nothing (probably, because you are encouraged to use firewalld, which is not a bad thing!).

There are two packages “iptables-persistent” and “netfilter-persistent”, which work together to have iptables persistence over reboots. The rules are saved and restored automatically during system startup.

First, install “iptables-persistent” and “netfilter-persistent” with

sudo apt install netfilter-persistent iptables-persistent

During the iptables–persistent installation the setup asks the user to save the current iptables rules. Hit “Yes” if you want to save the current iptables rules, which will be automatically loaded the next time the system starts up.

main menu — Configuring iptables-persistent setup

So it is safe to install it on a live system – the current iptables rules won’t be deleted.
Second, ensure the boot script to restore the iptables rules is enabled

sudo systemctl enable netfilter-persistent

Additional information

Saving the current state of the iptables rules:

myuser@myubuntupc:~$ sudo /usr/sbin/netfilter-persistent save
run-parts: executing /usr/share/netfilter-persistent/plugins.d/15-ip4tables save
run-parts: executing /usr/share/netfilter-persistent/plugins.d/25-ip6tables save

Restore the original state of the iptables rules:

sudo systemctl restart netfilter-persistent

And all commands you can do – start, stop, restart, reload, flush, save. You can use the script directly (it is not mandatory to use systemctl to restart, i.e. restore rules and etc.)

myuser@myubuntupc:~$ sudo /usr/sbin/netfilter-persistent
Usage: /usr/sbin/netfilter-persistent (start|stop|restart|reload|flush|save)

The script netfilter-persistent executes 2 other scripts as plugins:

/usr/share/netfilter-persistent/plugins.d/15-ip4tables
/usr/share/netfilter-persistent/plugins.d/25-ip6tables

The iptables rules are saved respectively in files

/etc/iptables/rules.v4
/etc/iptables/rules.v6

And you can always edit them manually or save/restore with iptables-save and iptables-restore redirecting the output to the above files.

It’s normal the state of the “active (exited)”. The service is “enabled” as you can see (by default the setup automatically enables the service on Ubuntu, but always check it to be sure, it’s the firewall!).

myuser@myubuntupc:~$ sudo systemctl status netfilter-persistent
● netfilter-persistent.service - netfilter persistent configuration
   Loaded: loaded (/lib/systemd/system/netfilter-persistent.service; enabled; vendor preset: enabled)
   Active: active (exited) since Thu 2019-01-17 20:44:08 EST; 14min ago
 Main PID: 666 (code=exited, status=0/SUCCESS)
   CGroup: /system.slice/netfilter-persistent.service

Jan 17 20:44:08 myubuntupc systemd[1]: Starting netfilter persistent configuration...
Jan 17 20:44:08 myubuntupc netfilter-persistent[666]: run-parts: executing /usr/share/netfilter-persistent/plugins.d/15-ip4tables start
Jan 17 20:44:08 myubuntupc netfilter-persistent[666]: run-parts: executing /usr/share/netfilter-persistent/plugins.d/25-ip6tables start
Jan 17 20:44:08 myubuntupc systemd[1]: Started netfilter persistent configuration.

nginx remote logging to UDP rsyslog server (CentOS 7)

This article will present to you all the configuration needed to remotely save access logs of an Nginx web server. All the configuration from the client and server sides is included. The client and the server use CentOS 7 Linux distribution and the configuration could be used under different Linux distribution. Probably only Selinux rules are kind of specific to the CentOS 7 and the firewalld rules are specific for those who use it as a firewall replacing the iptables. Here is the summary of what to expect:

Client-side – nginx configuration
Server-side – rsyslog configuration to accept UDP connections
Server-side – selinux and firewall configuration

The JSON formatted logs may be sent to a Elasticsearch server, for example. Here is how to do it – send access logs in json to Elasticsearch using rsyslog

STEP 1) Client-side – the Nginx configuration.

Nginx configuration is pretty simple just a single line with the log template and the IP (and port if not default 514) of the rsyslog server. For the record, this is the official documentation https://nginx.org/en/docs/syslog.html. In addition it worth mentioning there could be multiple access_log directives in a single section to log simultaneously on different targets (and the templates may be different or the same). So you can set the access log output of a section locally and remotely.
Nginx configuration (probably /etc/nginx/nginx.conf or whatever is the organization of your Nginx configuration files.)

server {
     .....
     access_log      /var/log/nginx/example.com_access.log main;
     access_log      syslog:server=10.10.10.2:514,facility=local7,tag=nginx,severity=info main3;
     .....
}

The “main” and “main3” are just names of the logging templates defined earlier (you may check rsyslog remote logging – prevent local messages to appear to see an interesting Nginx logging template).
The error log also could be remotely logged:

error_log syslog:server=10.10.10.3 debug;

STEP 2) Server-side – rsyslog configuration to accept UDP connections.

Of course, if you have not installed the rsyslog it’s high time you installed it with (for CentOS 7):

yum install -y rsyslog

To enable rsyslog to listen for UDP connections your rsyslog configuration file (/etc/rsyslog.conf) must include the following:

$ModLoad imudp
$UDPServerRun 514

Most of the Linux distributions have these two lines commented so you just need to uncomment them by removing the “#” from the beginning of the lines. If the lines are missing just add them under section “MODULES” (it should be near the first lines of the rsyslog configuration file).
Change the 514 with the number you like for the UDP listening port.
Write the client’s incoming lines of information to a different location and prevent merging with the local log messages – rsyslog remote logging – prevent local messages to appear. Include as a first rule under the rules’ section starting with “RULES” of the rsyslog configuration file (/etc/rsyslog.conf):

# Remote logging
$template HostIPtemp,"/mnt/logging/%FROMHOST-IP%.log"
if ($fromhost-ip != "127.0.0.1" ) then ?HostIPtemp
& stop

Logs only of remote hosts are going to be saved under /mnt/logging/.log.
Keep on reading!

Receive multicast packets on CentOS 7 (and other linux distros)

There are so many web pages and blogs post for multicast traffic under linux and how to enable it, but in most of them something always is missing and if you follow them probably you’ll end up with not working setup and you’ll have to search the Internet again – some do need tuning of the linux kernel variables not to drop packets, some need tuning the firewall to allow protocols.
Here we present a real working example of a server under CentOS 7, our server has two network ports:

eno1 – local unicast traffic with local IP
eno2 – multicast traffic

We have multicast TV streams, which we can use through our second network interface and we want to use ffmpeg to encode the video. We have “Multicast Group:port” for every stream, which is like “IP:PORT” and in our case the port is always the same 5000. Here are the steps you need to do if you want to receive these streams.

STEP 1) Set networking and make the configuration permanent.

In CentOS 7 the network of eno2, set a local IP, in fact it does not matter the exact IP, we used 10.10.10.0/24 local network.

/etc/sysconfig/network-scripts/ifcfg-eno2

TYPE=Ethernet
BOOTPROTO=none
DEFROUTE=yes
IPV4_FAILURE_FATAL=no
IPV6INIT=yes
IPV6_AUTOCONF=yes
IPV6_DEFROUTE=yes
IPV6_FAILURE_FATAL=no
NAME=eno2
UUID=2481b907-5e6e-45f9-ab96-7091e4e7d6d1
ONBOOT=yes
HWADDR=0c:c4:7a:44:87:a5
IPADDR0=10.10.10.152
PREFIX0=24
IPV6_PEERDNS=yes
IPV6_PEERROUTES=yes
NM_CONTROLLED=no

The important lines are highlighted. Configure the network adapter on boot and set 10.10.10.152.
Second and very important add a static route for the multicast traffic for the network interface, which is supposed to have the multicast streams (in our case “eno2”). Use file:

/etc/sysconfig/network-scripts/route-eno2

224.0.0.0/4 dev eno2

STEP 2) Kernel variables tuning

[srv@local ~]# for i in /proc/sys/net/ipv4/conf/*/rp_filter ; do echo 0 > "$i";   done
[srv@local ~]# echo "0" > /proc/sys/net/ipv4/icmp_echo_ignore_broadcasts

Turn off “Reverse Path Filtering” for all interfaces (rp_filter) and ping for the multicast address to work properly we need to disable icmp_echo_ignore_broadcasts. By default kernel drops these packets so you need this tuning or your application will not receive a single udp packet.
You could save the above two lines in

/etc/rc.local

and do not forget to set proper access rights:

[srv@local ~]# chmod 755 /etc/rc.local

Or you can use

/etc/sysctl.conf

Add the following lines in it:

net.ipv4.conf.default.rp_filter=0
net.ipv4.conf.all.rp_filter=0
net.ipv4.conf.eno2.rp_filter=0
net.ipv4.icmp_echo_ignore_broadcasts=0

STEP 3) Allow UDP traffic (and/or IGMP) if you have firewall

Here the port is the “Multicast Group:port” of the IP your application will join (look the example below)

firewalld – the default with CentOS 7:

[srv@local ~]# firewall-cmd --new-zone=multicast --permanent
[srv@local ~]# firewall-cmd --zone=multicast --add-interface=eno2 --permanent
[srv@local ~]# firewall-cmd --zone=multicast --add-protocol=igmp --permanent
[srv@local ~]# firewall-cmd --zone=multicast --add-protocol=icmp --permanent
[srv@local ~]# firewall-cmd --zone=multicast --add-source=224.0.0.0/4 --permanent
[srv@local ~]# firewall-cmd --zone=multicast --add-source=10.10.10.0/24 --permanent
[srv@local ~]# firewall-cmd --zone=multicast --add-port=5000/udp --permanent
[srv@local ~]# firewall-cmd --reload

We make a new zone for the multicast streams and add our interface, the sources and the port we use. We added IGMP, because in some use cases it is needed (in our it could work without IGMP added)

iptables – if you have disabled firewall and still you wanted to have a firewall:
```
# allow multicast addresses
[srv@local ~]# iptables -A INPUT -p udp -d 5000 -j ACCEPT
[srv@local ~]# iptables -A INPUT -s 224.0.0.0/4 -j ACCEPT
[srv@local ~]# iptables -A INPUT -p igmp -d 224.0.0.0/4 -j ACCEPT
[srv@local ~]# iptables -A INPUT -p icmp --icmp-type 0 -j ACCEPT
```
Probably it is a good idea to see if you have current rules (with “iptables -L -v -n” or even “iptables-save”) and to see if you should use “-A” (above) or “-I” to insert the rules above the DROP rule(s).

* Example with ffmpeg joining to a multicast group

[srv@local ~]# ffmpeg -i 'udp://239.100.10.5:5000'
ffmpeg version 2.8.6 Copyright (c) 2000-2016 the FFmpeg developers
  built with gcc 5.3.0 (Gentoo 5.3.0 p1.0, pie-0.6.5)
  configuration: --prefix=/usr --libdir=/usr/lib64 --shlibdir=/usr/lib64 --mandir=/usr/share/man --enable-shared --cc=x86_64-pc-linux-gnu-gcc --cxx=x86_64-pc-linux-gnu-g++ --ar=x86_64-pc-linux-gnu-ar --optflags='-march=native -O2 -msse3 -fomit-frame-pointer -pipe' --disable-static --enable-avfilter --enable-avresample --disable-stripping --enable-nonfree --enable-version3 --enable-nonfree --disable-indev=alsa --disable-indev=oss --disable-outdev=alsa --disable-outdev=oss --enable-version3 --enable-bzlib --disable-runtime-cpudetect --disable-debug --disable-doc --disable-gnutls --enable-gpl --enable-hardcoded-tables --enable-iconv --disable-lzma --enable-network --enable-openssl --enable-postproc --disable-libsmbclient --enable-ffplay --enable-sdl --disable-vaapi --disable-vdpau --enable-xlib --disable-libxcb --disable-libxcb-shm --disable-libxcb-xfixes --enable-zlib --disable-libcdio --disable-libiec61883 --disable-libdc1394 --enable-libcaca --disable-openal --disable-opengl --disable-libv4l2 --disable-libpulse --enable-libopencore-amrwb --enable-libopencore-amrnb --disable-libfdk-aac --enable-libopenjpeg --disable-libbluray --disable-libcelt --disable-libgme --enable-libgsm --disable-libmodplug --disable-libopus --disable-libquvi --disable-librtmp --disable-libssh --disable-libschroedinger --disable-libspeex --enable-libvorbis --enable-libvpx --disable-libzvbi --disable-libbs2b --disable-libflite --disable-frei0r --disable-libfribidi --enable-fontconfig --disable-ladspa --disable-libass --enable-libfreetype --disable-libsoxr --enable-pthreads --enable-libvo-aacenc --disable-libvo-amrwbenc --enable-libmp3lame --disable-libaacplus --enable-libfaac --disable-libsnappy --enable-libtheora --disable-libtwolame --disable-libwavpack --disable-libwebp --enable-libx264 --disable-libx265 --enable-libxvid --enable-x11grab --disable-amd3dnow --disable-amd3dnowext --disable-fma4 --disable-xop --cpu=host
  libavutil      54. 31.100 / 54. 31.100
  libavcodec     56. 60.100 / 56. 60.100
  libavformat    56. 40.101 / 56. 40.101
  libavdevice    56.  4.100 / 56.  4.100
  libavfilter     5. 40.101 /  5. 40.101
  libavresample   2.  1.  0 /  2.  1.  0
  libswscale      3.  1.101 /  3.  1.101
  libswresample   1.  2.101 /  1.  2.101
  libpostproc    53.  3.100 / 53.  3.100
[mpeg2video @ 0xf2fa80] Invalid frame dimensions 0x0.
    Last message repeated 10 times
Input #0, mpegts, from 'udp://239.100.10.5:5000':
  Duration: N/A, start: 87846.990933, bitrate: 5659 kb/s
  Program 5 
    Metadata:
      service_name    : ?TVtest
      service_provider: ?ss
    Stream #0:0[0x33]: Video: mpeg2video (Main) ([2][0][0][0] / 0x0002), yuv420p(tv), 720x576 [SAR 64:45 DAR 16:9], 5467 kb/s, 25 fps, 25 tbr, 90k tbn, 50 tbc
    Stream #0:1[0x34](bul): Audio: mp2 ([3][0][0][0] / 0x0003), 48000 Hz, stereo, s16p, 192 kb/s
At least one output file must be specified

As you can see to join the multicast group and ffmpeg to start encoding you use “udp://239.100.10.5:5000” for input stream parameter. Here ffmpeg joins the group and receives packets successfully.

* Generic example to receive UDP multicast stream

You may use the following example in any linux distro like Ubuntu, CentOS 7, Gentoo, OpenSuse and many others to receive muticast streams

[srv@local ~]# #ifconfig or ip - use one of them
[srv@local ~]# #ifconfig
[srv@local ~]# ifconfig eno1 10.10.10.152/24 up
[srv@local ~]# route add -net 224.0.0.0 netmask 240.0.0.0 eno2
[srv@local ~]# #or ip
[srv@local ~]# ip addr add 10.10.10.152/24 dev eno2
[srv@local ~]# ip link set eno2 up
[srv@local ~]# #allow multicast packets to the server
[srv@local ~]# for i in /proc/sys/net/ipv4/conf/*/rp_filter ; do echo 0 > "$i";   done
[srv@local ~]# echo "0" > /proc/sys/net/ipv4/icmp_echo_ignore_broadcasts
[srv@local ~]# #if you have firewall use the following for the iptables OR firewalld after that - use only one of them!
[srv@local ~]# iptables -A INPUT -p udp -d 5000 -j ACCEPT
[srv@local ~]# iptables -A INPUT -s 224.0.0.0/4 -j ACCEPT
[srv@local ~]# iptables -A INPUT -p igmp -d 224.0.0.0/4 -j ACCEPT
[srv@local ~]# iptables -A INPUT -p icmp --icmp-type 0 -j ACCEPT
[srv@local ~]# #firewalld setup
[srv@local ~]# firewall-cmd --new-zone=multicast --permanent
[srv@local ~]# firewall-cmd --zone=multicast --add-interface=eno2 --permanent
[srv@local ~]# firewall-cmd --zone=multicast --add-protocol=igmp --permanent
[srv@local ~]# firewall-cmd --zone=multicast --add-protocol=icmp --permanent
[srv@local ~]# firewall-cmd --zone=multicast --add-source=224.0.0.0/4 --permanent
[srv@local ~]# firewall-cmd --zone=multicast --add-source=10.10.10.0/24 --permanent
[srv@local ~]# firewall-cmd --zone=multicast --add-port=5000/udp --permanent
[srv@local ~]# firewall-cmd --reload