logrotate under CentOS 7 had worked perfectly till one day when a web server log stopped being rotated! All the other logs were rotated successfully. Apparently, there was no reason for this, and the logrotate status file was pointing to the very web server log file was rotated successfully during the night, but it wasn’t really!
The problem appeared to be the presence of an uncompressed file, despite the compressed file of the same file being generated.
Apparently, the logrotate or the compressing process was interrupted and the uncompressed file was left in place. After that, every time rotation happened it just stopped on this step for this log file. So the solution is to remove the compressed file and then the normal rotation will continue as it should be.
The logrotate status file is /var/lib/logrotate/logrotate.status and it is just a text file with the time of the last rotation for each log file:
"/var/log/nginx/error.log" 2021-11-2-9:33:26
"/var/log/nginx/media.static.log" 2021-11-2-9:33:26
"/var/log/php-fpm/www.access.log" 2021-11-2-9:33:26
"/var/log/yum.log" 2021-11-2-9:31:49
"/var/log/httpd/*log" 2021-11-2-9:0:0
"/var/log/wtmp" 2021-11-2-9:33:26
"/var/log/chrony/*.log" 2021-9-25-3:0:0
....
But the file has been one big file for the last 30 days:
[root@srv nginx]# ls -altr |tail -n 15
-rw-r-----. 1 nginx adm 1199 Oct 2 02:42 media.static.error.log-20211002.gz
-rw-r-----. 1 nginx adm 1919 Oct 2 02:54 error.log-20211002.gz
-rw-r-----. 1 nginx adm 39174314 Oct 2 03:16 media.static.log-20211002.gz
-rw-r-----. 1 nginx adm 307922 Oct 2 03:16 access.log-20211002.gz
-rw-r-----. 1 nginx adm 1461 Oct 3 02:16 media.static.error.log-20211003.gz
-rw-r-----. 1 nginx adm 1892 Oct 3 02:42 error.log-20211003.gz
-rw-r-----. 1 nginx adm 518079 Oct 3 03:38 access.log-20211003.gz
-rw-r-----. 1 nginx adm 1666744662 Oct 3 03:38 media.static.log-20211003
drwxr-xr-x. 2 root root 20480 Oct 4 03:31 .
-rw-r-----. 1 nginx adm 23642112 Oct 4 03:31 media.static.log-20211003.gz
-rw-r-----. 1 nginx adm 633643 Nov 2 06:29 media.static.error.log
-rw-r-----. 1 nginx adm 680493 Nov 2 09:30 error.log
drwxr-xr-x. 16 root root 4096 Nov 2 09:31 ..
-rw-r-----. 1 nginx adm 279561983 Nov 2 09:43 access.log
-rw-r-----. 1 nginx adm 29415439068 Nov 2 09:43 media.static.log
Debug the problem
Executing the logrotate manually will just report that the log files does not need rotation based on the time in the logrotate status file is /var/lib/logrotate/logrotate.status and in the status file it says the file has been rotated on “2021-11-2-9:33:26”!
But executing with force option reveals the real problem:
There is an uncompressed file and a compressed one with the same only with the suffix .gz.
So the uncompressed log file was wrong left in place after compression (or probably the compression or rotation process was interrupted). After the event (probably interruption of the rotating process) every day the logrotate outputted error when rotating this very same log file, but it had been written in the logrorate status file as such the rotating was successful with the latest date and time.
[root@srv nginx]# logrotate -vf /etc/logrotate.conf
....
rotating log /var/log/nginx/media.static.log, log->rotateCount is 1000
dateext suffix '-20211102'
glob pattern '-[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9]'
compressing log with: /bin/gzip
set default create context to system_u:object_r:httpd_log_t:s0
error: error creating output file /var/log/nginx/media.static.log-20211003.gz: File exists
rotating pattern: /var/log/php-fpm/*log forced from command line (4 rotations)
empty log files are not rotated, old logs are removed
....
The solution
First, remove the compressed file force logrotate to rotate the logs.
[root@srv nginx]# rm media.static.log-20211003.gz
[root@srv nginx]# logrotate -vf /etc/logrotate.conf
reading config file /etc/logrotate.conf
including /etc/logrotate.d
reading config file chrony
reading config file nginx
reading config file php-fpm
....
rotating log /var/log/nginx/media.static.log, log->rotateCount is 1000
dateext suffix '-20211003'
glob pattern '-[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9]'
compressing log with: /bin/gzip
set default create context to system_u:object_r:httpd_log_t:s0
destination /var/log/nginx/media.static.log-20211003 already exists, skipping rotation
....
No error, just skipping the rotation (the renaming of the file) and compressing it.
