Tune nginx proxy cache – control the cache manager how to delete cached files

In most cases you’ll never want to modify the default settings for deleting cache items with proxy_cache_path directives. The problem is in a peak the file deleting could impact your server performance and even it could kill your server leaving it unresponsive for a period of time. You cannot instruct nginx with a schedule job for deletion cached items or ban the deletion when the server is busy or loaded. The manager just traces each zone for used cache capacity versus the maximum allowed size and if the used capacity is near or bigger than the maximum allowed size (max_size) the manager process triggers deletion with the default values – the nginx manager will try to delete at least 100 files (up to 200 milliseconds) and then it will sleep for 50 milliseconds then again it will try deleting 100 files. So your file system could receive at least 1000 files per second to delete!

This could lead your server to almost unresponsive state in the peaks.

And it could be perfectly OK in off-peaks, but there is no way how to tell nginx cache manager there is a plenty free space despite you reach the cache limit so at the moment it is not the best time to delete the cache!

You can tune three parameters per cache directory (manual here: https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_cache_path):

  • manager_files – not more than this number of files to delete in one iteration. The default value is 100.
  • manager_threshold – limit the delete iteration time. The default value is 200 milliseconds and you must use nginx time syntax concatenated to the number you want, for example if you want 500 milliseconds you must use “500 ms”.
  • manager_sleep – how much time to sleep the manager before executing another delete iteration. The default value is 50 milliseconds and here you must use nginx time syntax concatenated to the number you want, for example if you want 500 milliseconds you must use “500 ms”.
        proxy_cache_path /mnt/cache levels=1:2 keys_zone=CACHESTATICS:900m inactive=710h max_size=4000g manager_files=2 manager_sleep=200ms manager_threshold=500ms;

The cache manager will delete not more than 2 files for up to 500 milliseconds and it will sleep 200 milliseconds before another delete iteration.

The best option for loaded servers

The best option for loaded servers with full cache is to balance the free space – delete small amount of files at once to be sure your server will not get loaded even the free space decreases at the peaks (so more files are cached than the nginx manager could delete – you are aware of this and the free space should be enough), but during the off peak (which normally is several times longer than the peak) the nginx manager could catch up with the deleting and it should free up some space (cached files are lesser than the deleted ones). Of course, you should tune this according to your situation.
The main idea is to delete in small amounts of files to not saturate your disks it could take longer to recover the free space, but it will not load your server in peaks. You should consider two things:

  1. Free space – enough free space and to be sure the free space is enough for the peaks, when the cache could grow above the threshold.
  2. Number of deletions per iteration – you should experiment with this. Fist you should be away how many files are added for a period of time, which includes one peak and one off-peak and then to balance the number in such a way that after the period the cache is not above the maximum size. Probably the best is to start with a 24 hours period, which includes at least one peak.

As you can see the example above only 2 files are good enough for an iteration for our case. Taking into account the 200ms sleep between the files’ deletions 10 files at most should be deleted per second. In our case it is not enough for the peak, but for the off-peak, which is 20 hours every 24 hours, is good enough to get into the maximum size limit of the cache.

Here you can learn how to verify your nginx is deleting cache files and the impact of the default settings on a busy server in a peak: how to disable effectively the deleting (purging) files from nginx proxy_cache (nginx cache manager process) Our loaded server just stopped serving files and the bandwidth decreased with 99% because nginx cache manager suddenly started deleting cached files.

how to disable effectively the deleting (purging) files from nginx proxy_cache (nginx cache manager process)

In peaks deleting files could kill your server and easily the traffic could degraded multiple times than normal if the nginx cache manager start deleting files!

The server is perfectly normal but suddenly it just get loaded and all nginx processes are in D (“Disk sleep”) state.

What could it be? What is going on with your proxy server?

Probably the cache is full!

Unfortunately there is no way to check how much is filled the cache live – just an upgrade or restart of the nginx process will trigger nginx cache loader to check all the cache files and will write the cache size on exit in the error log – but be careful the cache loading is also IO intensive operation – stats all the cache files and they could be millions images).

If you are sure the cache manager is to blame for the IO of your server (probably using this method – Check whether nginx cache manager is deleting files at the moment), you can stop it almost immediately!

Just increase the nginx cache drastically – add zero to the maximum cache size

Of course, you should have enough free space till you resolve the problem – for example more servers or manual deletion on peak-off or tune your cache deletion or any other solution….
Search for something like

        proxy_cache_path /mnt/cache levels=1:2 keys_zone=CACHESTATICS:900m inactive=710h max_size=400g

And add zero to the max_size number like:

        proxy_cache_path /mnt/cache levels=1:2 keys_zone=CACHESTATICS:900m inactive=710h max_size=4000g

The max size will increase from 400G to 4000G (4T)!
This will effectively stop the files deleting and the nginx cache manager will have slept for long time before invoking again to delete files. This could be life saving operation for your server at peak!

Here is a real graph from one of our servers – the cache manager started deleting files from the cache and the traffic dropped 99%!!!

SCREENSHOT 1) The nginx cache manager just started to delete files from the cache and this operation just killed our server completely.

You can see almost zero bandwidth! The problem was resolved when we reloaded nginx with a bigger cache max_size value. The nginx manager immediately went to sleep and no IO for deleting files. The load of the server returned to normal!

main menu
nginx cache manager start deleting files

SCREENSHOT 2) Hard drives were saturated and the disk maxed the IO time to 10 ms.

Despite the bigger READ and WRITE IOPS there was 95-99% less traffic.

main menu
Disk IO Time when cache manager is working

Then you can tune the values for deleting files from the cache – Tune nginx proxy cache – control the cache manager how to delete cached files.

Check whether nginx cache manager is deleting files at the moment

Here is a tip for the webmasters (or system admins) to discover whether the nginx using proxy_cache to cache files is deleting files at the moment! There situation where you may need to know if the loaded of a static media server is caused by the deletion of the cache manager or by the read or seek operations when serving the static files. The deletion is really slow and IO intensive operation, which could greatly impact the performance and traffic of the server.
Find the process nginx’s “cache manager process” and strace it:

[root@srv ~]# ps axuf|grep nginx
root     31582  0.0  0.0 2906768 25108 ?       Ss   Feb15   0:01 nginx: master process /usr/sbin/nginx -c /etc/nginx/nginx.conf
nginx    16008  1.9  1.3 2941188 440224 ?      S    16:39   1:33  \_ nginx: worker process
nginx    16009  1.5  1.2 2941188 398836 ?      S    16:39   1:12  \_ nginx: worker process
nginx    16010  0.5  0.7 2941984 239064 ?      S    16:39   0:26  \_ nginx: worker process
nginx    16011  0.7  0.9 2941984 299356 ?      D    16:39   0:35  \_ nginx: worker process
nginx    16012  1.2  1.1 2941188 389540 ?      D    16:39   1:01  \_ nginx: worker process
nginx    16013  2.3  1.5 2941188 487324 ?      D    16:39   1:55  \_ nginx: worker process
nginx    16014  0.0  0.6 2906772 224004 ?      S    16:39   0:01  \_ nginx: cache manager process
[root@srv ~]# strace -f -p 16014
strace: Process 16014 attached
gettid()                                = 16014
write(31, "2019/02/25 18:00:31 [info] 16014"..., 89) = 89
epoll_wait(36, [], 512, 5406)           = 0
unlink("/mnt/cache/0/39/c8ccbbc06d16debb1c8d58ceb6f99390") = 0
unlink("/mnt/cache/0/78/118924d7bf70e20fa8f790c6f9e7c780") = 0
unlink("/mnt/cache/3/ce/fab074cc670e6a80114dcbc398a63ce3") = 0
unlink("/mnt/cache/5/48/0b4e162dd7be8244815721fb7d68e485") = 0
unlink("/mnt/cache/5/56/e5eb4b38c7c8d209d0aabaf79ac02565") = 0
unlink("/mnt/cache/e/c6/207b432fa77375e4eefcaf52db250c6e") = 0
unlink("/mnt/cache/4/6d/ac0db27a03dabc79d869068db1b516d4") = 0
unlink("/mnt/cache/9/e8/91625c6e60de8e5425c4135c7dfb2e89") = 0
unlink("/mnt/cache/b/3c/f3c53000cf0cb20d55d8c09df8a733cb") = 0
unlink("/mnt/cache/f/f7/6f06423cd411b45816969fe020903f7f") = 0
unlink("/mnt/cache/f/50/c9b8ab72821a6e9bcb9c8d4b790dc50f") = 0
unlink("/mnt/cache/6/1f/74b0f1fdf1ac30db6af7793dc15671f6") = 0
unlink("/mnt/cache/0/83/caf199c1b99d438f96caec71bf2ea830") = 0
unlink("/mnt/cache/4/3d/c90f8fbbba4aaf407e386641dc2203d4") = 0
unlink("/mnt/cache/4/ad/d23cf8598020141b2bcec46d2b5cbad4") = 0
unlink("/mnt/cache/d/47/05973bc310503f36c67b7c1c24c8247d") = 0
unlink("/mnt/cache/f/11/e4fcbde8533d89105ab41f22c55e211f") = 0
unlink("/mnt/cache/2/06/29066a58e4116d24266026b4ed1e3062") = 0
epoll_wait(32, [], 512, 50)             = 0
unlink("/mnt/cache/4/6b/9a104ebdf70d00137a88d4584b2bb6b4") = 0
unlink("/mnt/cache/e/95/6d176447f57f21769d86a8f0b2a8b95e") = 0
unlink("/mnt/cache/b/b2/2f6f51163c65ae1fc06a913d6de1ab2b") = 0
unlink("/mnt/cache/a/24/2b058045a23b69de7a4442c9e6fce24a") = 0
unlink("/mnt/cache/7/60/00833e0b236ca8472f5be8227d645607") = 0
unlink("/mnt/cache/a/08/bf00eea300eff97dc4fffa61daaca08a") = 0
unlink("/mnt/cache/2/48/a291d8aca2b6f4f9471686eabe9b2482") = 0
unlink("/mnt/cache/0/e3/2d631adbc3bfdf8e44a51fa5453eee30") = 0
unlink("/mnt/cache/1/3b/08eef7c86c5ece9b5279b304dd86e3b1") = 0
unlink("/mnt/cache/b/a4/03213e4a8a1e8fb17ae698e54e70fa4b") = 0
unlink("/mnt/cache/b/a3/77f1b11811a9cda0ae93c498769f7a3b") = 0
unlink("/mnt/cache/4/01/1d50fac60681ae3263c8875775d20014") = 0
unlink("/mnt/cache/c/94/e71b96cbc65b248bd8e4540cbd69294c") = 0
unlink("/mnt/cache/1/59/99ec58e865b97e217835dd84f5f48591") = 0
unlink("/mnt/cache/4/b8/6a64825ce555b8f2440f051a7f7bcb84") = 0
unlink("/mnt/cache/7/51/fe2acbb895427ed8e406ce7e79d61517") = 0
.....
.....

You can tune the file removing from the cache with manager_files, manager_threshold and manager_sleep arguments of the proxy_cache_path.
If you came here searching information on the topic probably you should check out these articles, too: how to disable effectively the deleting (purging) files from nginx proxy_cache (nginx cache manager process) and Tune nginx proxy cache – control the cache manager how to delete cached files

mysql – Error ‘Your password does not satisfy the current policy requirements’ or zero length mysql password

We got this error when granting permissions for one of our new slave server (it could be for an ordinary MySQL server, too):

Error 'Your password does not satisfy the current policy requirements' on query. Default database: ''. Query: 'GRANT REPLICATION SLAVE ON *.* TO 'reusr'@'127.0.01''

It appeared that MySQL has activated by default a password checking plugin and our password in the GRANT (or SET PASSWORD) option didn’t meet the requirements.
So here is what you can do:

OPTION 1) Lower the password policy level

Check the policy level and lower it if it is MEDIUM or HIGH (they are there options LOW=0, MEDIUM=1 the default and HIGH=2). The policy level controls how to check and what is involved in the complexity algorithm for the passwords. More details here – https://dev.mysql.com/doc/refman/5.7/en/validate-password-options-variables.html#sysvar_validate_password_policy. Here is what you have:

[myuser@mysql1 ~]# mysql
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 13
....
....
mysql> SHOW VARIABLES LIKE 'validate%';
+--------------------------------------+--------+
| Variable_name                        | Value  |
+--------------------------------------+--------+
| validate_password_check_user_name    | OFF    |
| validate_password_dictionary_file    |        |
| validate_password_length             | 8      |
| validate_password_mixed_case_count   | 1      |
| validate_password_number_count       | 1      |
| validate_password_policy             | MEDIUM |
| validate_password_special_char_count | 1      |
+--------------------------------------+--------+
7 rows in set (0.00 sec)

So set the validate_password_policy=0 and try again your query:

mysql> set global validate_password_policy=0;
Query OK, 0 rows affected (0.00 sec)

If you still get the error your password is lower than the validate_password_length (=8 by default) so you need to change it at last to 8 characters. But what if you what zero password (or with 1,2,3 characters)? Setting validate_password_length to 0 won’t work, because there is a hard limit to 4, so you cannot set it to 0 event the set query is not reporting error when using 0 with validate_password_length.

You should uninstall the plugin.

OPTION 2) Uninstall the MySQL Validation Plugin

You can uninstall the validation plugin on-the-fly in a working server without restarting or reloading and then you can set whatever password you like.
Here is how to do it:

[myuser@mysql1 ~]# mysql
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 13
....
....
mysql> UNINSTALL PLUGIN validate_password;
Query OK, 0 rows affected (0.03 sec)

mysql> SHOW VARIABLES LIKE 'validate%';
Empty set (0.01 sec)

As you can see no “validate_password” variables are available anymore! Now set your password.
But there is a catch, if you have started the server with “–validate-password=FORCE_PLUS_PERMANENT” (you can check it with “ps axuf|grep mysqld” in the command line) you won’t be able to uninstall the plugin live even with the root MySQL user. So at the end if you do not have root permissions to restart the MySQL service without this option it might be better to change your password or skip the query if it is received by the slave in the MySQL replication bin log.
You can install the plugin again with:

[myuser@mysql1 ~]# mysql
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 13
....
....
mysql> INSTALL PLUGIN validate_password SONAME 'validate_password.so';
Query OK, 0 rows affected (0.00 sec)

And it will be available over restarts, too, because it is registered in “mysql.plugin” table.

mysql – Error ‘Column count of mysql.user is wrong. Expected 45, found 43. The table is probably corrupted’ on query.

If you

upgraded your MySQL server (from 5.6 to 5.7 or above)

or

imported a MySQL dump SQL file from older version

than your current server you may encounter when granting permissions to a user:

Error 'Column count of mysql.user is wrong. Expected 45, found 43. The table is probably corrupted' on query. Default database: ''. Query: 'GRANT REPLICATION SLAVE ON *.* TO 'replusr'@'144.76.156.182''

Do not panic probably it is not corrupted just continue reading.

There is the simple fix, just

execute mysql_upgrade

It will automatically detect what to upgrade and it will upgrade it:

[myuser@mysql1 ~]# screen -R upgrade
[myuser@mysql1 ~]# mysql_upgrade 
Checking if update is needed.
Checking server version.
Running queries to upgrade MySQL server.
Checking system database.
mysql.columns_priv                                 OK
mysql.db                                           OK
mysql.engine_cost                                  OK
mysql.event                                        OK
mysql.func                                         OK
mysql.general_log                                  OK
mysql.gtid_executed                                OK
mysql.help_category                                OK
mysql.help_keyword                                 OK
mysql.help_relation                                OK
mysql.help_topic                                   OK
mysql.host                                         OK
mysql.innodb_index_stats                           OK
mysql.innodb_table_stats                           OK
mysql.ndb_binlog_index                             OK
mysql.plugin                                       OK
mysql.proc                                         OK
mysql.procs_priv                                   OK
mysql.proxies_priv                                 OK
mysql.server_cost                                  OK
mysql.servers                                      OK
mysql.slave_master_info                            OK
mysql.slave_relay_log_info                         OK
mysql.slave_worker_info                            OK
mysql.slow_log                                     OK
mysql.tables_priv                                  OK
mysql.time_zone                                    OK
mysql.time_zone_leap_second                        OK
mysql.time_zone_name                               OK
mysql.time_zone_transition                         OK
mysql.time_zone_transition_type                    OK
mysql.user                                         OK
The sys schema is already up to date (version 1.5.1).
Found 0 sys functions, but expected 22. Re-installing the sys schema.
Upgrading the sys schema.
Checking databases.
phpmyadmin.pma__bookmark                           OK
phpmyadmin.pma__central_columns                    OK
phpmyadmin.pma__column_info                        OK
phpmyadmin.pma__designer_settings                  OK
phpmyadmin.pma__export_templates                   OK
phpmyadmin.pma__favorite                           OK
phpmyadmin.pma__history                            OK
phpmyadmin.pma__navigationhiding                   OK
phpmyadmin.pma__pdf_pages                          OK
phpmyadmin.pma__recent                             OK
phpmyadmin.pma__relation                           OK
phpmyadmin.pma__savedsearches                      OK
phpmyadmin.pma__table_coords                       OK
phpmyadmin.pma__table_info                         OK
phpmyadmin.pma__table_uiprefs                      OK
phpmyadmin.pma__tracking                           OK
phpmyadmin.pma__userconfig                         OK
phpmyadmin.pma__usergroups                         OK
phpmyadmin.pma__users                              OK
sys.sys_config                                     OK
db1.access                                         OK
db1.users                                          OK
db1.objects                                        OK
db1.isp                                            OK
db1.desc                                           OK
Upgrade process completed successfully.
Checking if update is needed.

It works when the server is up and running and it is a good idea to execute the command in a screen.
It does not need to be logged as root, but mysql_upgrade does need to have the root MySQL password. In the example above it did not asked for password, because we have it in ~/.my.cnf file.

Just to note you might upgraded a long before this error to appear!

If you do not use a certain functionality you could live up happily with the old mysql.user scheme (and all old mysql.* tables). In our case we upgraded one of our slaves and several days after when a grant command on the master was issued the replication just stopped with this error! Of course, if someone were used the command in our slave the error would have appeared there sooner.
We also had case where old MySQL SQL dump file (5.6) was imported in a newer MySQL server 5.7 and there had been no issues for weeks till the GRANT command.

perror

Th error code is 1805.

[myuser@mysql1 ~]# perror 1805
MySQL error code 1805 (ER_COL_COUNT_DOESNT_MATCH_CORRUPTED_V2): Column count of %s.%s is wrong. Expected %d, found %d. The table is probably corrupted

Supermicro server cannot enter BIOS with F2, DEL or other when UEFI mode OS is installed

If you happen to have a supermicro server (X10SLH-F) and install Linux in UEFI mode in our case CentOS 7 and you want to enter the BIOS you’ll be surprised that you cannot with the keys provided in the very same BIOS boot screen – F2, DEL. The F11 and F12 also does not work for menu selection and network boot!

Even if you manage to press the DEL key and you see on the screen “Entering BIOS setup…” – the server WON’T enter BIOS, but will continue with the UEFI BIOS boot drive!

So what to do? Ammm break temporary your system by removing (renaming or moving) the EFI directory in your efi boot partition, resetting your server and holding pressed DEL key (again) on all start up screens of the server. When the UEFI BIOS boot entry is not valid any more and there are no other boot devices (and probably because we pressed DEL key) we were able to enter in the BIOS without remote hands on the collocation side or any other intervention on the server.

[root@srv ~]# mv /boot/efi/EFI/ /boot/efi/EFI_org
[root@srv ~]# reboot

This is the path in CentOS 7 and our standard partition layout:

[root@srv ~]# df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda3         26G  4.5G    20G  19% /
devtmpfs         7.8G     0   7.8G   0% /dev
tmpfs            7.8G     0   7.8G   0% /dev/shm
tmpfs            7.8G  8.5M   7.8G   1% /run
tmpfs            7.8G     0   7.8G   0% /sys/fs/cgroup
/dev/sda2        976M   98M   812M  11% /boot
/dev/sda1        200M  9.8M   191M   5% /boot/efi
tmpfs            1.6G     0   1.6G   0% /run/user/0

DO NOT forget to remove all other (virtual) CD/DVD ROM Devices and temporary disable your network PXE Server (if you have any in the network)

Because it when the UEFI BIOS cannot find the EFI file saved in the UEFI BIOS BOOT drive it might follow the boot order before entering the BIOS!

Enter the bios by remote console on our X9 boards with UEFI bios

Apparently there is an issue with X8 and X9 supermicro boards in UEFI mode BIOS: https://www.supermicro.com/support/faqs/faq.cfm?faq=14029
So for someone it could be useful pressing and holding “ESC” + “-” or F4 to enter the UEFI BIOS, but we could not make it because of the IPMI KVM we used to manage the server.

Update supermicro X10SLM-F firmware BIOS under Linux with the SUM cli

Here is how we updated our Supermicro server X10SLM-F with the latest firmware at the moment.

Our current BIOS firmware version is 2.0

[root@srv ~]# lshw|grep -A 14 "core$"
  *-core
       description: Motherboard
       product: X10SLM-F
       vendor: Supermicro
       physical id: 0
       version: 1.02
       serial: 11111111111
       slot: To be filled by O.E.M.
     *-firmware
          description: BIOS
          vendor: American Megatrends Inc.
          physical id: 0
          version: 2.0
          date: 04/24/2014
          size: 64KiB

Keep on reading!

sed with delimiter – any other single character in replacing words or characters

It is not so known fact that this powerful unix world command

sed

could be used with other delimiter than “/” (when replacing words or characters), which is used in 100% of the time in the Internet examples.
You probably know the syntax from the manual like:

s/regexp/replacement/
              Attempt to match regexp against the pattern space.  If successful, replace that portion matched with replacement.  The replacement may  con‐
              tain  the special character & to refer to that portion of the pattern space which matched, and the special escapes \1 through \9 to refer to
              the corresponding matching sub-expressions in the regexp.

The Internet examples always use “/” as shown in the man, BUT you CAN use “#” instead of “/” for example:

s#regexp#replacement#g

or

s:regexp:replacement:g

or you can even use letter character (not a special character as “!@#$%#^”):

sRregexpRreplacementRg

and with “r” lower case (you see not so readable, but possible!)

srregexprreplacementrg

Here are the examples:

sed 's@else@elllllse@g' test.php
sed 's#else#elllllse#g' test.php
sed 's:else:elllllse:g' test.php
sed 's~else~elllllse~g' test.php
sed 's!else!elllllse!g' test.php
sed 'spelsepelllllsepg' test.php
sed 'sRelseRelllllseRg' test.php
#could be paired with "-i", too
sed 'spelsepelllllsepg' -i test.php
sed 'sRelseRelllllseRg' -i test.php

What does it mean to you? The simple implication is you are not forced to escape characters in your regexp and replacement part. Consider you want to replace part of a unix/linux path string: /home/myuser/Desktop/mydirectory1/myfile.log if you use the default “/” you MUST escape all the “/” in your string:

sed 's/\/home\/myuser\/Desktop\/mydirectory1\/myfile.log/\/home\/user\/Desktop\/mydirectory2\/myfile.log/g' test.log

versus the simpler and more readable version with “#”

sed 's#/home/myuser/Desktop/mydirectory1/myfile.log#/home/user/Desktop/mydirectory2/myfile.log#g' test.log

And consider you use an variable:

cat file.test | sed "s/\[version\]/${PKGVERSION}/g"

* additional explanations:

  • /g (or what ever character is used for delimiter like “@”, “:”, “~” and so on) is for “Apply the replacement to all matches to the regexp, not just the first.”
  • -i (used in some of the examples above) is for inline replacement – the file you add in the sed command will be modified and the replacement will be saved in the file (the default behavior is to show the modified output in the standard output – console output)

Clear or delete systemd logs

Systemd linux distros use Journald service to collect and store logs in the system. Here are a couple of tips if you have problems with the space they occupy. It is good that all systemd linux distros support it – CentOS 7, Ubuntu 16+, Fedora, OpenSuse and so on.

TIP 1) Remove the old archive logs older than 10 days with

Time based removal of old logs. It will remove the old files. This command won’t change the configuration, so it just has a temporary effect.

journalctl --vacuum-time=10d

TIP 2) Remove the old archive logs greater than 1G

Size based removal of old logs. It will reduce the size of the logs to this specified size. This command won’t change the configuration, so it just has a temporary effect.

journalctl --vacuum-size=1024M

TIP 3) Show disk usage

[root@srv ~]# journalctl --disk-usage
Archived and active journals take up 785.5M on disk.

TIP 4) Show all logs and information for them

Where are the log files, the size they occupy, the time period of the entries in them and more:

[root@srv0 ~]# journalctl --header
File Path: /run/log/journal/bb5aa69bfa2b483386443ede7a92a45a/system.journal
File ID: 7deaff4610a94c82aab85386597e825b
Machine ID: bb5aa69bfa2b483386443ede7a92a45a
Boot ID: e89507fa3f814b8a96f9d70914c120b0
Sequential Number ID: 541a59d9a68342d3980d86e565a48c7d
State: ONLINE
Compatible Flags:
Incompatible Flags: COMPRESSED-XZ
Header size: 240
Arena size: 102956816
Data Hash Table Size: 178744
Field Hash Table Size: 333
Rotate Suggested: no
Head Sequential Number: 1239405
Tail Sequential Number: 1341511
Head Realtime Timestamp: mon 2018-06-25 10:09:09 UTC
Tail Realtime Timestamp: tue 2018-06-28 00:07:10 UTC
Tail Monotonic Timestamp: 1month 2d 21h 6min 47.287s
Objects: 258136
Entry Objects: 102107
Data Objects: 129585
Data Hash Table Fill: 72.5%
Field Objects: 36
Field Hash Table Fill: 10.8%
Tag Objects: 0
Entry Array Objects: 26406
Disk usage: 98.1M

File Path: /run/log/journal/bb5aa69bfa2b483386443ede7a92a45a/system@541a59d9a68342d3980d86e565a48c7d-000000000011512d-00056f3f3ff56172.journal
File ID: a8e1f042bc144df78f37005b0b555a82
Machine ID: bb5aa69bfa2b483386443ede7a92a45a
Boot ID: e89507fa3f814b8a96f9d70914c120b0
Sequential Number ID: 541a59d9a68342d3980d86e565a48c7d
State: ARCHIVED
Compatible Flags:
Incompatible Flags: COMPRESSED-XZ
Header size: 240
Arena size: 102956816
Data Hash Table Size: 178744
Field Hash Table Size: 333
Rotate Suggested: no
Head Sequential Number: 1134893
Tail Sequential Number: 1239404
Head Realtime Timestamp: fri 2018-06-22 18:32:10 UTC
Tail Realtime Timestamp: mon 2018-06-25 10:09:09 UTC
Tail Monotonic Timestamp: 1month 7h 8min 45.991s
Objects: 263445
Entry Objects: 104512
Data Objects: 131953
Data Hash Table Fill: 73.8%
Field Objects: 34
Field Hash Table Fill: 10.2%
Tag Objects: 0
Entry Array Objects: 26944
Disk usage: 98.1M

File Path: /run/log/journal/bb5aa69bfa2b483386443ede7a92a45a/system@541a59d9a68342d3980d86e565a48c7d-00000000000fc083-00056f0d6ef70320.journal
File ID: 007272b1f181487d83116bee96c40c30
Machine ID: bb5aa69bfa2b483386443ede7a92a45a
Boot ID: e89507fa3f814b8a96f9d70914c120b0
Sequential Number ID: 541a59d9a68342d3980d86e565a48c7d
State: ARCHIVED
Compatible Flags:
Incompatible Flags: COMPRESSED-XZ
Header size: 240
Arena size: 102956816
Data Hash Table Size: 178744
Field Hash Table Size: 333
Rotate Suggested: no
Head Sequential Number: 1032323
Tail Sequential Number: 1134892
Head Realtime Timestamp: wed 2018-06-20 07:06:10 UTC
Tail Realtime Timestamp: fri 2018-06-22 18:32:10 UTC
Tail Monotonic Timestamp: 4w 2h 1min 46.980s
Objects: 263998
Entry Objects: 102570
Data Objects: 133456
Data Hash Table Fill: 74.7%
Field Objects: 37
Field Hash Table Fill: 11.1%
Tag Objects: 0
Entry Array Objects: 27933
Disk usage: 98.1M

File Path: /run/log/journal/bb5aa69bfa2b483386443ede7a92a45a/system@541a59d9a68342d3980d86e565a48c7d-00000000000e3f67-00056ee09376d287.journal
File ID: 4c268268a6e34d6297c7ae9ca01fc31b
Machine ID: bb5aa69bfa2b483386443ede7a92a45a
Boot ID: e89507fa3f814b8a96f9d70914c120b0
Sequential Number ID: 541a59d9a68342d3980d86e565a48c7d
State: ARCHIVED
Compatible Flags:
Incompatible Flags: COMPRESSED-XZ
Header size: 240
Arena size: 102956816
Data Hash Table Size: 178744
Field Hash Table Size: 333
Rotate Suggested: yes
Head Sequential Number: 933735
Tail Sequential Number: 1032322
Head Realtime Timestamp: mon 2018-06-18 01:35:09 UTC
Tail Realtime Timestamp: wed 2018-06-20 07:06:10 UTC
Tail Monotonic Timestamp: 3w 4d 14h 35min 47.248s
Objects: 261498
Entry Objects: 98588
Data Objects: 134059
Data Hash Table Fill: 75.0%
Field Objects: 40
Field Hash Table Fill: 12.0%
Tag Objects: 0
Entry Array Objects: 28809
Disk usage: 98.1M

File Path: /run/log/journal/bb5aa69bfa2b483386443ede7a92a45a/system@541a59d9a68342d3980d86e565a48c7d-00000000000cbf3b-00056eb3ea255b2a.journal
File ID: 7019a721bebe4e00b76410a6e7052c6d
Machine ID: bb5aa69bfa2b483386443ede7a92a45a
Boot ID: e89507fa3f814b8a96f9d70914c120b0
Sequential Number ID: 541a59d9a68342d3980d86e565a48c7d
State: ARCHIVED
Compatible Flags:
Incompatible Flags: COMPRESSED-XZ
Header size: 240
Arena size: 102956816
Data Hash Table Size: 178744
Field Hash Table Size: 333
Rotate Suggested: yes
Head Sequential Number: 835387
Tail Sequential Number: 933734
Head Realtime Timestamp: fri 2018-06-15 20:18:10 UTC
Tail Realtime Timestamp: mon 2018-06-18 01:35:09 UTC
Tail Monotonic Timestamp: 3w 2d 9h 4min 45.972s
Objects: 260794
Entry Objects: 98348
Data Objects: 134060
Data Hash Table Fill: 75.0%
Field Objects: 33
Field Hash Table Fill: 9.9%
Tag Objects: 0
Entry Array Objects: 28351
Disk usage: 98.1M

File Path: /run/log/journal/bb5aa69bfa2b483386443ede7a92a45a/system@541a59d9a68342d3980d86e565a48c7d-00000000000b2fac-00056e827d3375dd.journal
File ID: 61c16e8acc364d97b8c0c7cc2afcb6ec
Machine ID: bb5aa69bfa2b483386443ede7a92a45a
Boot ID: e89507fa3f814b8a96f9d70914c120b0
Sequential Number ID: 541a59d9a68342d3980d86e565a48c7d
State: ARCHIVED
Compatible Flags:
Incompatible Flags: COMPRESSED-XZ
Header size: 240
Arena size: 102956816
Data Hash Table Size: 178744
Field Hash Table Size: 333
Rotate Suggested: no
Head Sequential Number: 733100
Tail Sequential Number: 835386
Head Realtime Timestamp: wed 2018-06-13 09:20:08 UTC
Tail Realtime Timestamp: fri 2018-06-15 20:18:10 UTC
Tail Monotonic Timestamp: 3w 3h 47min 46.827s
Objects: 264298
Entry Objects: 102287
Data Objects: 134043
Data Hash Table Fill: 75.0%
Field Objects: 33
Field Hash Table Fill: 9.9%
Tag Objects: 0
Entry Array Objects: 27933
Disk usage: 98.1M

File Path: /run/log/journal/bb5aa69bfa2b483386443ede7a92a45a/system@541a59d9a68342d3980d86e565a48c7d-000000000009966a-00056e4cbbb6ebfc.journal
File ID: c90df7bdc4654ab7a4ac955c74a779a3
Machine ID: bb5aa69bfa2b483386443ede7a92a45a
Boot ID: e89507fa3f814b8a96f9d70914c120b0
Sequential Number ID: 541a59d9a68342d3980d86e565a48c7d
State: ARCHIVED
Compatible Flags:
Incompatible Flags: COMPRESSED-XZ
Header size: 240
Arena size: 102956816
Data Hash Table Size: 178744
Field Hash Table Size: 333
Rotate Suggested: no
Head Sequential Number: 628330
Tail Sequential Number: 733099
Head Realtime Timestamp: sun 2018-06-10 17:12:09 UTC
Tail Realtime Timestamp: wed 2018-06-13 09:19:10 UTC
Tail Monotonic Timestamp: 2w 4d 16h 48min 46.993s
Objects: 263295
Entry Objects: 104770
Data Objects: 131816
Data Hash Table Fill: 73.7%
Field Objects: 37
Field Hash Table Fill: 11.1%
Tag Objects: 0
Entry Array Objects: 26670
Disk usage: 98.1M

File Path: /run/log/journal/bb5aa69bfa2b483386443ede7a92a45a/system@541a59d9a68342d3980d86e565a48c7d-000000000007fd20-00056e16f3135f30.journal
File ID: 32915d651dd84728b9e4a33a97b480cf
Machine ID: bb5aa69bfa2b483386443ede7a92a45a
Boot ID: e89507fa3f814b8a96f9d70914c120b0
Sequential Number ID: 541a59d9a68342d3980d86e565a48c7d
State: ARCHIVED
Compatible Flags:
Incompatible Flags: COMPRESSED-XZ
Header size: 240
Arena size: 102956816
Data Hash Table Size: 178744
Field Hash Table Size: 333
Rotate Suggested: no
Head Sequential Number: 523552
Tail Sequential Number: 628329
Head Realtime Timestamp: fri 2018-06-08 01:02:10 UTC
Tail Realtime Timestamp: sun 2018-06-10 17:12:09 UTC
Tail Monotonic Timestamp: 2w 2d 41min 46.200s
Objects: 263287
Entry Objects: 104778
Data Objects: 131696
Data Hash Table Fill: 73.7%
Field Objects: 36
Field Hash Table Fill: 10.8%
Tag Objects: 0
Entry Array Objects: 26775
Disk usage: 98.1M
[root@srv0 ~]#

* Deleting logs

[root@srv0 ~]# journalctl --vacuum-size=128M
Deleted archived journal /var/log/journal/9be717e698354ec481abb641cf4085c3/system@79d0440cc1174c2db132b707a0567bcb-0000000000a33241-000570199db5be3c.journal (8.0M).
Deleted archived journal /var/log/journal/9be717e698354ec481abb641cf4085c3/system@79d0440cc1174c2db132b707a0567bcb-0000000000a34463-0005701a0b24d0fc.journal (8.0M).
Deleted archived journal /var/log/journal/9be717e698354ec481abb641cf4085c3/system@79d0440cc1174c2db132b707a0567bcb-0000000000a3568c-0005701a79e388c7.journal (8.0M).
Deleted archived journal /var/log/journal/9be717e698354ec481abb641cf4085c3/system@79d0440cc1174c2db132b707a0567bcb-0000000000a36899-0005701ae529f439.journal (8.0M).
Deleted archived journal /var/log/journal/9be717e698354ec481abb641cf4085c3/system@79d0440cc1174c2db132b707a0567bcb-0000000000a37ac8-0005701b50781f62.journal (8.0M).
Deleted archived journal /var/log/journal/9be717e698354ec481abb641cf4085c3/system@79d0440cc1174c2db132b707a0567bcb-0000000000a38cde-0005701bbd3b60c3.journal (8.0M).
Deleted archived journal /var/log/journal/9be717e698354ec481abb641cf4085c3/system@79d0440cc1174c2db132b707a0567bcb-0000000000a39ef6-0005701c28dcf40e.journal (8.0M).
Deleted archived journal /var/log/journal/9be717e698354ec481abb641cf4085c3/system@79d0440cc1174c2db132b707a0567bcb-0000000000a3b0f1-0005701c9416ee94.journal (8.0M).
Deleted archived journal /var/log/journal/9be717e698354ec481abb641cf4085c3/system@79d0440cc1174c2db132b707a0567bcb-0000000000a3c309-0005701cffe19015.journal (8.0M).
Deleted archived journal /var/log/journal/9be717e698354ec481abb641cf4085c3/system@79d0440cc1174c2db132b707a0567bcb-0000000000a3d521-0005701d6c79b9cb.journal (8.0M).
Deleted archived journal /var/log/journal/9be717e698354ec481abb641cf4085c3/system@79d0440cc1174c2db132b707a0567bcb-0000000000a3e748-0005701dd7c60e79.journal (8.0M).
Deleted archived journal /var/log/journal/9be717e698354ec481abb641cf4085c3/system@79d0440cc1174c2db132b707a0567bcb-0000000000a3f92d-0005701e48413eb8.journal (8.0M).
Deleted archived journal /var/log/journal/9be717e698354ec481abb641cf4085c3/system@79d0440cc1174c2db132b707a0567bcb-0000000000a40b0a-0005701ec83a0c74.journal (8.0M).
Deleted archived journal /var/log/journal/9be717e698354ec481abb641cf4085c3/system@79d0440cc1174c2db132b707a0567bcb-0000000000a41ce2-0005701f3fdcadc6.journal (8.0M).
Deleted archived journal /var/log/journal/9be717e698354ec481abb641cf4085c3/system@79d0440cc1174c2db132b707a0567bcb-0000000000a42f04-0005701fabfefbca.journal (8.0M).
Deleted archived journal /var/log/journal/9be717e698354ec481abb641cf4085c3/system@79d0440cc1174c2db132b707a0567bcb-0000000000a4410e-000570201757a256.journal (120.0M).
Deleted archived journal /var/log/journal/9be717e698354ec481abb641cf4085c3/system@79d0440cc1174c2db132b707a0567bcb-0000000000a60a8f-0005702ac025bb0c.journal (120.0M).
Deleted archived journal /var/log/journal/9be717e698354ec481abb641cf4085c3/system@79d0440cc1174c2db132b707a0567bcb-0000000000a7d254-000570359f2bd7eb.journal (120.0M).
Deleted archived journal /var/log/journal/9be717e698354ec481abb641cf4085c3/system@79d0440cc1174c2db132b707a0567bcb-0000000000a99a34-000570405b5e812e.journal (120.0M).
Deleted archived journal /var/log/journal/9be717e698354ec481abb641cf4085c3/system@79d0440cc1174c2db132b707a0567bcb-0000000000ab639b-0005704b05553c7d.journal (120.0M).
Deleted archived journal /var/log/journal/9be717e698354ec481abb641cf4085c3/system@79d0440cc1174c2db132b707a0567bcb-0000000000ad2c3e-00057055cb403fae.journal (120.0M).
Deleted archived journal /var/log/journal/9be717e698354ec481abb641cf4085c3/system@79d0440cc1174c2db132b707a0567bcb-0000000000aef5c8-000570607686fd78.journal (120.0M).
Vacuuming done, freed 960.0M of archived journals from /var/log/journal/9be717e698354ec481abb641cf4085c3.

Replace an old IP with new one in all files of all sub-directories recursively

Here is a quick Linux tip for those who want to replace their old IP with new one for all files in a given directory and all its sub-directories recursively:

find [path-to-directory] -type f -print0 | xargs -0 sed -i 's[old-IP-escape-dot]/[new-IP]/g'

Quick example:

find /etc/nginx/ -type f -print0 | xargs -0 sed -i 's/192\.168\.10\.124/10.10.10.224/g'

As you can see the directory is “/etc/nginx” and replace it with the directory where are your (configuration) files are. We are replacing old IP=192.168.10.124 with the new one 10.10.10.224, so after the execution of the above line you’ll get modified files with IP 10.10.10.224.
You must escape the dot “.” in the IP!