nginx with php fpm (fastcgi) and the warning – an upstream response is buffered to a temporary file /var/cache/nginx/fastcgi_temp

As the web grows and the technology advances the page size of the web sites also grows or just some times you might want to output a big chunk of data from your application server – PHP-FPM (but it could be any of another ruby, python, C, Django and more), for example.
Here is a fast configuration tip (note this is not the proxy-related warning!):

The default nginx buffers per CGI connection are too small

Here is what to do in your nginx configuration file:
First, look for a line “include /etc/nginx/fastcgi_params;” or similar and add or edit if they exist after this line:

        fastcgi_buffer_size 16k;
        fastcgi_buffers 32 16k;

Check out more for the buffers here http://nginx.org/en/docs/http/ngx_http_fastcgi_module.html#fastcgi_buffers
The warning should stop if it does not stop you can try raising them. It could consume more memory but could lower the IO usage of your disks and improve the performance of your site or whatever backend works!

Here is the warning in our nginx error logs. We got this warning when using php-fpm and the php output size was 325965 bytes (~320K).

2019/04/04 09:56:05 [warn] 24451#24451: *44269838 an upstream response is buffered to a temporary file /var/cache/nginx/fastcgi_temp/0/12/0019966120 while reading upstream, client: 10.10.10.10, server: srv17.srv.en, request: "GET /api/20140102/product HTTP/1.1", upstream: "fastcgi://127.0.0.1:9000", host: "srv17.srv.en"
2019/04/04 09:56:07 [warn] 24451#24451: *44269849 an upstream response is buffered to a temporary file /var/cache/nginx/fastcgi_temp/2/12/0019966122 while reading upstream, client: 10.10.10.11, server: srv17.srv.en, request: "GET /api/20140102/product HTTP/1.1", upstream: "fastcgi://127.0.0.1:9000", host: "srv17.srv.en"
2019/04/04 09:56:09 [warn] 24450#24450: *44269856 an upstream response is buffered to a temporary file /var/cache/nginx/fastcgi_temp/7/12/0019966127 while reading upstream, client: 10.10.10.12, server: srv17.srv.en, request: "GET /api/20140102/product HTTP/1.1", upstream: "fastcgi://127.0.0.1:9000", host: "srv17.srv.en"

php-fpm memcached – Unknown: Failed to write session data (memcached) when using persistent connections

Several reports in the Internet with such error:

[22-Mar-2019 21:35:20 Africa/Tunis] PHP Warning:  Unknown: Failed to write session data (memcached). Please verify that the current setting of session.save_path is correct (PERSISTENT=2 10.10.10.10:11211) in Unknown on line 0
[22-Mar-2019 21:35:21 Africa/Lagos] PHP Warning:  Unknown: Failed to write session data (memcached). Please verify that the current setting of session.save_path is correct (PERSISTENT=2 10.10.10.10:11211) in Unknown on line 0
[22-Mar-2019 13:35:21 America/Los_Angeles] PHP Warning:  Unknown: Failed to write session data (memcached). Please verify that the current setting of session.save_path is correct (PERSISTENT=2 10.10.10.10:11211) in Unknown on line 0

But no real solutions worked for us. Here is our setup:

Several web servers servers (using php-fpm – fastcgi) connect to a single memcached server to share sessions.

The PHP configuration (php.ini) is as simple as that:

....
session.save_handler = memcached
session.save_path = "PERSISTENT=2 10.10.10.10:11211"
....

This is how you can enable PHP sessions to be stored in the memcached server with IP 10.10.10.10 and tell the driver to use persistent connections. Note we use the binary PHP module memcached, not memcache, which is the old driver.

The persistence is handled by the PHP module, not the library, which it wraps and it is per process not per all PHP fpm (fastcgi) processes.
Apparently, sometimes this functionality just get corrupted and many of those errors begin to appear in the logs – most of the time a reload of the php-fpm after a long period of time. Even a restart did not help, so the real solution, which always works for us is

to change the ID of the “PERSISTENT=”

so just change to:

session.save_path = "PERSISTENT=234 10.10.10.10:11211"

No matter of the ID just some new one, not used previous time! And everything continues smooth!

PHP missing xml extension – Fatal error: Uncaught Error: Class DOMDocument not found

We upgraded one of our servers and at first did not notice this error. Soon logs began to fill with this error:

 Fatal error: Uncaught Error: Class 'DOMDocument' not found in /var/www/htdocs/site1/wp-content/plugins/all-in-one-seo-pack/aioseop_class.php:4501 Stack trace: #0 /var/www/htdocs/site1/wp-content/plugins/all-in-one-seo-pack/aioseop_class.php(4056): All_in_One_SEO_Pack->get_prev_next_links(Object(WP_Post)) #1 /var/www/htdocs/site1/wp-includes/class-wp-hook.php(286): All_in_One_SEO_Pack->wp_head('') #2 /var/www/htdocs/site1/wp-includes/class-wp-hook.php(310): WP_Hook->apply_filters(NULL, Array) #3 /var/www/htdocs/site1/wp-includes/plugin.php(465): WP_Hook->do_action(Array) #4 /var/www/htdocs/site1/wp-includes/general-template.php(2668): do_action('wp_head') #5 /var/www/htdocs/site1/wp-content/themes/twentysixteen/header.php(21): wp_head() #6 /var/www/htdocs/site1/wp-includes/template.php(704): require_once('/var/www/h...') #7 /var/www/htdocs/site1/wp-includes/template.php(653): load_template('/var/www/h...', true) #8 /var/www/htdocs/site1/wp-incl in /var/www/htdocs/site1/wp-content/plugins/all-in-one-seo-pack/aioseop_class.php on line 4501

Apparently, we missed to install all PHP extensions as before and one of our WordPress plugins (All in One SEO Pack) began to throw this nasty error and blank pages, where the error is supposed to be shown!

The fix is easy enough just install PHP-XML

CentOS 7

  • default PHP:
    yum -y install php-xml
    
  • Using ius-release repo:
    yum -y install php72u-xml
    

Ubuntu 16/17/18

sudo apt install -y php-xml php-dom

Gentoo

Just add to your current USE the following “xml” in your /etc/portage/make.conf and emerge (be sure “xml” is in USE emerge output, not “-xml”!):

srv1 ~ # emerge -va --nodeps php

These are the packages that would be merged, in order:

[ebuild   R    ] dev-lang/php-7.2.10:7.2::gentoo  USE="acl apache2 bcmath berkdb bzip2 calendar cgi cli ctype curl exif fileinfo filter fpm ftp gd gdbm hash iconv ipv6 json mhash mysql mysqli nls opcache pcntl pdo phar posix readline session sharedmem simplexml snmp soap sockets sqlite ssl tokenizer truetype unicode xml xmlreader xmlrpc xmlwriter zip zlib -argon2 -cdb -cjk -coverage -debug -embed -enchant -firebird -flatfile -gmp -imap -inifile -intl -iodbc -kerberos -ldap -ldap-sasl -libedit -libressl -lmdb -mssql -oci8-instant-client -odbc -phpdbg -postgres -qdbm -recode (-selinux) -session-mm -sodium -spell -systemd -sysvipc -test -threads -tidy -tokyocabinet -wddx -webp -xpm -xslt -zip-encryption" 0 KiB

Bonus

Here is how our WordPress page looks like with the error:

 <!DOCTYPE html>
<html lang="en-US" class="no-js">
<head>
	<meta charset="UTF-8">
	<meta name="viewport" content="width=device-width, initial-scale=1">
	<link rel="profile" href="http://gmpg.org/xfn/11">
		<link rel="pingback" href="https://ahelpme.com/xmlrpc.php">
		<script>(function(html){html.className = html.className.replace(/\bno-js\b/,'js')})(document.documentElement);</script>
<title>How to compile xmr-stak (2.4.5) under CentOS 7 for CPU mining cryptocurrencies | Any IT here? Help Me! - Part 2</title>

<!-- All in One SEO Pack 2.12 by Michael Torbert of Semper Fi Web Design[387,517] -->
<br />
<b>Fatal error</b>:  Uncaught Error: Class 'DOMDocument' not found in /var/www/htdocs/site1/wp-content/plugins/all-in-one-seo-pack/aioseop_class.php:4501
Stack trace:
#0 /var/www/htdocs/site1/wp-content/plugins/all-in-one-seo-pack/aioseop_class.php(4056): All_in_One_SEO_Pack-&gt;get_prev_next_links(Object(WP_Post))
#1 /var/www/htdocs/site1/wp-includes/class-wp-hook.php(286): All_in_One_SEO_Pack-&gt;wp_head('')
#2 /var/www/htdocs/site1/wp-includes/class-wp-hook.php(310): WP_Hook-&gt;apply_filters(NULL, Array)
#3 /var/www/htdocs/site1/wp-includes/plugin.php(465): WP_Hook-&gt;do_action(Array)
#4 /var/www/htdocs/site1/wp-includes/general-template.php(2668): do_action('wp_head')
#5 /var/www/htdocs/site1/wp-content/themes/twentysixteen/header.php(21): wp_head()
#6 /var/www/htdocs/site1/wp-includes/template.php(704): require_once('/var/www/h...')
#7 /var/www/htdocs/site1/wp-includes/template.php(653): load_template('/var/www/h...', true)
#8 /var/www/htdocs/site1/wp-incl in <b>/var/www/htdocs/site1/wp-content/plugins/all-in-one-seo-pack/aioseop_class.php</b> on line <b>4501</b><br />

And here is what you can expect in your web server logs (nginx + php-fpm setup)

2019/02/23 03:36:14 [error] 27441#27441: *56837 FastCGI sent in stderr: "PHP message: PHP Fatal error:  Uncaught Error: Class 'DOMDocument' not found in /var/www/htdocs/site1/root/wp-content/plugins/all-in-one-seo-pack/aioseop_class.php:4501
Stack trace:
#0 /var/www/htdocs/site1/root/wp-content/plugins/all-in-one-seo-pack/aioseop_class.php(4056): All_in_One_SEO_Pack->get_prev_next_links(Object(WP_Post))
#1 /var/www/htdocs/site1/root/wp-includes/class-wp-hook.php(286): All_in_One_SEO_Pack->wp_head('')
#2 /var/www/htdocs/site1/root/wp-includes/class-wp-hook.php(310): WP_Hook->apply_filters(NULL, Array)
#3 /var/www/htdocs/site1/root/wp-includes/plugin.php(465): WP_Hook->do_action(Array)
#4 /var/www/htdocs/site1/root/wp-includes/general-template.php(2668): do_action('wp_head')
#5 /var/www/htdocs/site1/root/wp-content/themes/twentysixteen/header.php(21): wp_head()
#6 /var/www/htdocs/site1/root/wp-includes/template.php(704): require_once('/var/www/h...')
#7 /var/www/htdocs/site1/root/wp-includes/template.php(653): load_template('/var/www/h...', true)
#8 /" while reading response header from upstream, client: 66.249.79.133, server: ahelpme.com, request: "GET /linux/fedora/review-of-freshly-installed-fedora-29-kde-plasma-desktop-kde-gui/2/ HTTP/1.1", upstream: "fastcgi://192.168.0.12:9000", host: "ahelpme.com"

And keep in mind if your HTTP 405 codes get in your access logs:

10.10.10.2 - - [22/Feb/2019:05:20:34 +0000] "GET /xmlrpc.php HTTP/1.1" 405 53 "-" "Mozilla/5.0 (X11; Linux i686; rv:2.0.1) Gecko/20100101 Firefox/4.0.1" "-"
10.10.10.2 - - [22/Feb/2019:05:20:38 +0000] "GET /xmlrpc.php HTTP/1.1" 405 53 "-" "Mozilla/5.0 (X11; Linux i686; rv:2.0.1) Gecko/20100101 Firefox/4.0.1" "-"

PHP module compilation error – libtool: Version mismatch error

If you happen to get this error trying to build your own a PHP module (probably, because you need a newer version than the included one in the disto you use and because it is fairly easy):

libtool: Version mismatch error.  This is libtool 2.4.6, but the
libtool: definition of this LT_INIT comes from an older release.
libtool: You should recreate aclocal.m4 with macros from libtool 2.4.6
libtool: and run autoconf again.
make: *** [Makefile:181: geoip.lo] Error 63

Here is what you can do to recreate the configuration files for the build process:

phpize
aclocal
libtoolize --force
autoheader
autoconf
./configure
make

This sequence of commands (probably without the phpize, because it is PHP specific) could save you in similar situations with similar error in other source code not related to PHP.
Keep on reading!

Persistent connections for PHP sessions stored using Memecached

If you have a highly loaded PHP application server you probably came to conclusion to use memcached to store the PHP sessions, because files are too slow and generate many IOs. And when the session are enabled to be stored in memcached server the server could be overloaded with so many connections to the memcached instance(s) that the server could delay the creation of new connections to the memcached server and the server is in trouble even more than before with the file stored sessions.
The solution is to use persistent connections to the memcached server, so when a connection to the memcached is needed it will check to reuse an existing idle connection stored in a pool of connections so no new connection is created each time PHP needs something from the cache like the sessions. In fact PHP accesses sessions from the cache at least once for every request to the server and it could generate a pretty big number of opened connections.

On top of that there are two PHP modules, which offer objects or functions to use memcached server cache in your PHP code. The old module is called “memcache” (link) and the new one is “memcached” (link) – just to note these are the names of the PHP modules not to be mistaken by the server caching system “memcached”.

To configure the php-fpm or mod_php to store sessions in memcached caching server:

  1. use PHP “memcached”, this is the new and better supported module for using a memcached cache server. It supports the newer versions of PHP like 7+.
  2. use the following configuration added to your php.ini or the module ini file included in your php.ini (it really depends on your configuration, but if you are not sure put phpinfo() function in a PHP file and load through your web server – you’ll see the physical location of your php.ini file);
session.save_path = "PERSISTENT=1 localhost:11212"

Replace localhost with the server IP or leave it if the memcached server is on the localhost. This single configuration could decrease the usage of opened sockets from several tens of thousands (literally 30 000 – 100 000 and above) to couple of hundreds, which could have a great impact on system performance! (You’ll see it like a system usage time in your server graphs…)

*Worth mentioning the PHP configuration (php.ini file) must be edited to use memcached server for storing sessions:

session.save_handler = memcached

PHP posix_kill missing in CentOS7

So you think you PHP code is running ok and your script for killing bad guys is perfect. Let’s assume you made a script to kill processes according to some criteria, put you script on your good all CentOS 7 server, but suddenly you encounter the error:

PHP Fatal error: Call to undefined function posix_kill() in ./kill-process.php on line 64
Killed process id: 21899
PHP Fatal error: Call to undefined function posix_kill() in ./kill-process.php on line 64
Killed process id: 21899
PHP Fatal error: Call to undefined function posix_kill() in ./kill-process.php on line 64

So you just missed a php plugin and there it is: “php-process”. Just install it with yum!

yum install -y php-process

And the error will disappear and the next time you want to kill some bad process you’ll sure do it!