nginx with php fpm (fastcgi) and the warning – an upstream response is buffered to a temporary file /var/cache/nginx/fastcgi_temp

As the web grows and the technology advances the page size of the web sites also grows or just some times you might want to output a big chunk of data from your application server – PHP-FPM (but it could be any of another ruby, python, C, Django and more), for example.
Here is a fast configuration tip (note this is not the proxy-related warning!):

The default nginx buffers per CGI connection are too small

Here is what to do in your nginx configuration file:
First, look for a line “include /etc/nginx/fastcgi_params;” or similar and add or edit if they exist after this line:

        fastcgi_buffer_size 16k;
        fastcgi_buffers 32 16k;

Check out more for the buffers here http://nginx.org/en/docs/http/ngx_http_fastcgi_module.html#fastcgi_buffers
The warning should stop if it does not stop you can try raising them. It could consume more memory but could lower the IO usage of your disks and improve the performance of your site or whatever backend works!

Here is the warning in our nginx error logs. We got this warning when using php-fpm and the php output size was 325965 bytes (~320K).

2019/04/04 09:56:05 [warn] 24451#24451: *44269838 an upstream response is buffered to a temporary file /var/cache/nginx/fastcgi_temp/0/12/0019966120 while reading upstream, client: 10.10.10.10, server: srv17.srv.en, request: "GET /api/20140102/product HTTP/1.1", upstream: "fastcgi://127.0.0.1:9000", host: "srv17.srv.en"
2019/04/04 09:56:07 [warn] 24451#24451: *44269849 an upstream response is buffered to a temporary file /var/cache/nginx/fastcgi_temp/2/12/0019966122 while reading upstream, client: 10.10.10.11, server: srv17.srv.en, request: "GET /api/20140102/product HTTP/1.1", upstream: "fastcgi://127.0.0.1:9000", host: "srv17.srv.en"
2019/04/04 09:56:09 [warn] 24450#24450: *44269856 an upstream response is buffered to a temporary file /var/cache/nginx/fastcgi_temp/7/12/0019966127 while reading upstream, client: 10.10.10.12, server: srv17.srv.en, request: "GET /api/20140102/product HTTP/1.1", upstream: "fastcgi://127.0.0.1:9000", host: "srv17.srv.en"

php-fpm memcached – Unknown: Failed to write session data (memcached) when using persistent connections

Several reports in the Internet with such error:

[22-Mar-2019 21:35:20 Africa/Tunis] PHP Warning:  Unknown: Failed to write session data (memcached). Please verify that the current setting of session.save_path is correct (PERSISTENT=2 10.10.10.10:11211) in Unknown on line 0
[22-Mar-2019 21:35:21 Africa/Lagos] PHP Warning:  Unknown: Failed to write session data (memcached). Please verify that the current setting of session.save_path is correct (PERSISTENT=2 10.10.10.10:11211) in Unknown on line 0
[22-Mar-2019 13:35:21 America/Los_Angeles] PHP Warning:  Unknown: Failed to write session data (memcached). Please verify that the current setting of session.save_path is correct (PERSISTENT=2 10.10.10.10:11211) in Unknown on line 0

But no real solutions worked for us. Here is our setup:

Several web servers servers (using php-fpm – fastcgi) connect to a single memcached server to share sessions.

The PHP configuration (php.ini) is as simple as that:

....
session.save_handler = memcached
session.save_path = "PERSISTENT=2 10.10.10.10:11211"
....

This is how you can enable PHP sessions to be stored in the memcached server with IP 10.10.10.10 and tell the driver to use persistent connections. Note we use the binary PHP module memcached, not memcache, which is the old driver.

The persistence is handled by the PHP module, not the library, which it wraps and it is per process not per all PHP fpm (fastcgi) processes.
Apparently, sometimes this functionality just get corrupted and many of those errors begin to appear in the logs – most of the time a reload of the php-fpm after a long period of time. Even a restart did not help, so the real solution, which always works for us is

to change the ID of the “PERSISTENT=”

so just change to:

session.save_path = "PERSISTENT=234 10.10.10.10:11211"

No matter of the ID just some new one, not used previous time! And everything continues smooth!

PHP missing xml extension – Fatal error: Uncaught Error: Class DOMDocument not found

We upgraded one of our servers and at first did not notice this error. Soon logs began to fill with this error:

 Fatal error: Uncaught Error: Class 'DOMDocument' not found in /var/www/htdocs/site1/wp-content/plugins/all-in-one-seo-pack/aioseop_class.php:4501 Stack trace: #0 /var/www/htdocs/site1/wp-content/plugins/all-in-one-seo-pack/aioseop_class.php(4056): All_in_One_SEO_Pack->get_prev_next_links(Object(WP_Post)) #1 /var/www/htdocs/site1/wp-includes/class-wp-hook.php(286): All_in_One_SEO_Pack->wp_head('') #2 /var/www/htdocs/site1/wp-includes/class-wp-hook.php(310): WP_Hook->apply_filters(NULL, Array) #3 /var/www/htdocs/site1/wp-includes/plugin.php(465): WP_Hook->do_action(Array) #4 /var/www/htdocs/site1/wp-includes/general-template.php(2668): do_action('wp_head') #5 /var/www/htdocs/site1/wp-content/themes/twentysixteen/header.php(21): wp_head() #6 /var/www/htdocs/site1/wp-includes/template.php(704): require_once('/var/www/h...') #7 /var/www/htdocs/site1/wp-includes/template.php(653): load_template('/var/www/h...', true) #8 /var/www/htdocs/site1/wp-incl in /var/www/htdocs/site1/wp-content/plugins/all-in-one-seo-pack/aioseop_class.php on line 4501

Apparently, we missed to install all PHP extensions as before and one of our WordPress plugins (All in One SEO Pack) began to throw this nasty error and blank pages, where the error is supposed to be shown!

The fix is easy enough just install PHP-XML

CentOS 7

  • default PHP:
    yum -y install php-xml
    
  • Using ius-release repo:
    yum -y install php72u-xml
    

Ubuntu 16/17/18

sudo apt install -y php-xml php-dom

Gentoo

Just add to your current USE the following “xml” in your /etc/portage/make.conf and emerge (be sure “xml” is in USE emerge output, not “-xml”!):

srv1 ~ # emerge -va --nodeps php

These are the packages that would be merged, in order:

[ebuild   R    ] dev-lang/php-7.2.10:7.2::gentoo  USE="acl apache2 bcmath berkdb bzip2 calendar cgi cli ctype curl exif fileinfo filter fpm ftp gd gdbm hash iconv ipv6 json mhash mysql mysqli nls opcache pcntl pdo phar posix readline session sharedmem simplexml snmp soap sockets sqlite ssl tokenizer truetype unicode xml xmlreader xmlrpc xmlwriter zip zlib -argon2 -cdb -cjk -coverage -debug -embed -enchant -firebird -flatfile -gmp -imap -inifile -intl -iodbc -kerberos -ldap -ldap-sasl -libedit -libressl -lmdb -mssql -oci8-instant-client -odbc -phpdbg -postgres -qdbm -recode (-selinux) -session-mm -sodium -spell -systemd -sysvipc -test -threads -tidy -tokyocabinet -wddx -webp -xpm -xslt -zip-encryption" 0 KiB

Bonus

Here is how our WordPress page looks like with the error:

 <!DOCTYPE html>
<html lang="en-US" class="no-js">
<head>
	<meta charset="UTF-8">
	<meta name="viewport" content="width=device-width, initial-scale=1">
	<link rel="profile" href="http://gmpg.org/xfn/11">
		<link rel="pingback" href="https://ahelpme.com/xmlrpc.php">
		<script>(function(html){html.className = html.className.replace(/\bno-js\b/,'js')})(document.documentElement);</script>
<title>How to compile xmr-stak (2.4.5) under CentOS 7 for CPU mining cryptocurrencies | Any IT here? Help Me! - Part 2</title>

<!-- All in One SEO Pack 2.12 by Michael Torbert of Semper Fi Web Design[387,517] -->
<br />
<b>Fatal error</b>:  Uncaught Error: Class 'DOMDocument' not found in /var/www/htdocs/site1/wp-content/plugins/all-in-one-seo-pack/aioseop_class.php:4501
Stack trace:
#0 /var/www/htdocs/site1/wp-content/plugins/all-in-one-seo-pack/aioseop_class.php(4056): All_in_One_SEO_Pack-&gt;get_prev_next_links(Object(WP_Post))
#1 /var/www/htdocs/site1/wp-includes/class-wp-hook.php(286): All_in_One_SEO_Pack-&gt;wp_head('')
#2 /var/www/htdocs/site1/wp-includes/class-wp-hook.php(310): WP_Hook-&gt;apply_filters(NULL, Array)
#3 /var/www/htdocs/site1/wp-includes/plugin.php(465): WP_Hook-&gt;do_action(Array)
#4 /var/www/htdocs/site1/wp-includes/general-template.php(2668): do_action('wp_head')
#5 /var/www/htdocs/site1/wp-content/themes/twentysixteen/header.php(21): wp_head()
#6 /var/www/htdocs/site1/wp-includes/template.php(704): require_once('/var/www/h...')
#7 /var/www/htdocs/site1/wp-includes/template.php(653): load_template('/var/www/h...', true)
#8 /var/www/htdocs/site1/wp-incl in <b>/var/www/htdocs/site1/wp-content/plugins/all-in-one-seo-pack/aioseop_class.php</b> on line <b>4501</b><br />

And here is what you can expect in your web server logs (nginx + php-fpm setup)

2019/02/23 03:36:14 [error] 27441#27441: *56837 FastCGI sent in stderr: "PHP message: PHP Fatal error:  Uncaught Error: Class 'DOMDocument' not found in /var/www/htdocs/site1/root/wp-content/plugins/all-in-one-seo-pack/aioseop_class.php:4501
Stack trace:
#0 /var/www/htdocs/site1/root/wp-content/plugins/all-in-one-seo-pack/aioseop_class.php(4056): All_in_One_SEO_Pack->get_prev_next_links(Object(WP_Post))
#1 /var/www/htdocs/site1/root/wp-includes/class-wp-hook.php(286): All_in_One_SEO_Pack->wp_head('')
#2 /var/www/htdocs/site1/root/wp-includes/class-wp-hook.php(310): WP_Hook->apply_filters(NULL, Array)
#3 /var/www/htdocs/site1/root/wp-includes/plugin.php(465): WP_Hook->do_action(Array)
#4 /var/www/htdocs/site1/root/wp-includes/general-template.php(2668): do_action('wp_head')
#5 /var/www/htdocs/site1/root/wp-content/themes/twentysixteen/header.php(21): wp_head()
#6 /var/www/htdocs/site1/root/wp-includes/template.php(704): require_once('/var/www/h...')
#7 /var/www/htdocs/site1/root/wp-includes/template.php(653): load_template('/var/www/h...', true)
#8 /" while reading response header from upstream, client: 66.249.79.133, server: ahelpme.com, request: "GET /linux/fedora/review-of-freshly-installed-fedora-29-kde-plasma-desktop-kde-gui/2/ HTTP/1.1", upstream: "fastcgi://192.168.0.12:9000", host: "ahelpme.com"

And keep in mind if your HTTP 405 codes get in your access logs:

10.10.10.2 - - [22/Feb/2019:05:20:34 +0000] "GET /xmlrpc.php HTTP/1.1" 405 53 "-" "Mozilla/5.0 (X11; Linux i686; rv:2.0.1) Gecko/20100101 Firefox/4.0.1" "-"
10.10.10.2 - - [22/Feb/2019:05:20:38 +0000] "GET /xmlrpc.php HTTP/1.1" 405 53 "-" "Mozilla/5.0 (X11; Linux i686; rv:2.0.1) Gecko/20100101 Firefox/4.0.1" "-"

PHP module compilation error – libtool: Version mismatch error

If you happen to get this error trying to build your own a PHP module (probably, because you need a newer version than the included one in the disto you use and because it is fairly easy):

libtool: Version mismatch error.  This is libtool 2.4.6, but the
libtool: definition of this LT_INIT comes from an older release.
libtool: You should recreate aclocal.m4 with macros from libtool 2.4.6
libtool: and run autoconf again.
make: *** [Makefile:181: geoip.lo] Error 63

Here is what you can do to recreate the configuration files for the build process:

phpize
aclocal
libtoolize --force
autoheader
autoconf
./configure
make

This sequence of commands (probably without the phpize, because it is PHP specific) could save you in similar situations with similar error in other source code not related to PHP.
Keep on reading!

Persistent connections for PHP sessions stored using Memecached

If you have a highly loaded PHP application server you probably came to conclusion to use memcached to store the PHP sessions, because files are too slow and generate many IOs. And when the session are enabled to be stored in memcached server the server could be overloaded with so many connections to the memcached instance(s) that the server could delay the creation of new connections to the memcached server and the server is in trouble even more than before with the file stored sessions.
The solution is to use persistent connections to the memcached server, so when a connection to the memcached is needed it will check to reuse an existing idle connection stored in a pool of connections so no new connection is created each time PHP needs something from the cache like the sessions. In fact PHP accesses sessions from the cache at least once for every request to the server and it could generate a pretty big number of opened connections.

On top of that there are two PHP modules, which offer objects or functions to use memcached server cache in your PHP code. The old module is called “memcache” (link) and the new one is “memcached” (link) – just to note these are the names of the PHP modules not to be mistaken by the server caching system “memcached”.

To configure the php-fpm or mod_php to store sessions in memcached caching server:

  1. use PHP “memcached”, this is the new and better supported module for using a memcached cache server. It supports the newer versions of PHP like 7+.
  2. use the following configuration added to your php.ini or the module ini file included in your php.ini (it really depends on your configuration, but if you are not sure put phpinfo() function in a PHP file and load through your web server – you’ll see the physical location of your php.ini file);
session.save_path = "PERSISTENT=1 localhost:11212"

Replace localhost with the server IP or leave it if the memcached server is on the localhost. This single configuration could decrease the usage of opened sockets from several tens of thousands (literally 30 000 – 100 000 and above) to couple of hundreds, which could have a great impact on system performance! (You’ll see it like a system usage time in your server graphs…)

*Worth mentioning the PHP configuration (php.ini file) must be edited to use memcached server for storing sessions:

session.save_handler = memcached

Install GO language in Ubuntu 16 LTS from the official site package

Ubuntu 16 LTS comes with too old GO language version, the version in the official repository is

golang-1.6.

Many programs released past couple of years depend on go lang 1.7+, so 1.6 in Ubuntu 16 LTS is really obsolete. At present the latest version is 1.9, so we’ll show you simple steps to install the go lang on your ubuntu 16 lts server

STEP 1) Get root if you want to install it globally to be used from any user!

sudo su
cd

STEP 2) Download the GO language binaries from the official site

https://golang.org/dl/

And copy the URL link of the program from the Linux section, now it has the following text:

Linux 2.6.23 or later, Intel 64-bit processor
go1.10.linux-amd64.tar.gz (114MB)

And download it with wget:

wget https://dl.google.com/go/go1.9.4.linux-amd64.tar.gz

STEP 4) Uncompress the file with “tar”

tar xzvf go1.9.4.linux-amd64.tar.gz

STEP 5) Move the files in “/usr/local/go”

you could put the installation anywhere, but most manual installed programs go to “/usr/local”

mv go /usr/local/

STEP 6) Set properly the environment to use your GO 1.9 installation

    • Create file with path and name

      /etc/profile.d/golang.sh

    • Make the file executable with
      chmod 755 /etc/profile.d/golang.sh
      
    • Insert the following line, which will add to the environment PATH your GO executable binaries
[[ -d "/usr/local/go" ]] && export PATH="$PATH:/usr/local/go/bin"

STEP 7) Reload the environment or just log off/in and check if you could execute the GO tool

root@srv.local:~# source /etc/profile
root@srv.local:~# go version
go version go1.9.4 linux/amd64

* Here are all the commands in one place to install the specific version of GO 1.9.4 AMD64, you can use them in a script:

cd
mkdir go-installation
cd ./go-installation
sudo wget https://dl.google.com/go/go1.9.4.linux-amd64.tar.gz
sudo tar xzvf go1.9.4.linux-amd64.tar.gz
sudo mv go /usr/local/
cat <<END | sudo tee /etc/profile.d/golang.sh 1>/dev/null
[[ -d "/usr/local/go" ]] && export PATH="\$PATH:/usr/local/go/bin"
END
sudo chmod 755 /etc/profile.d/golang.sh
source /etc/profile

And do not forget to

source /etc/profile

in your current shell session. It is mandatory for your current shell if you execute the above command from a script!

* This installation of GO Language 1.9 is from the official source the home site of GO Language. There are other easier ways to do it using, for example there are PPA repositories with Ubuntu packages ready to install, but they are not supported by the official GO team! Use them at your own risk as with all other PPA repositories.

PHP posix_kill missing in CentOS7

So you think you PHP code is running ok and your script for killing bad guys is perfect. Let’s assume you made a script to kill processes according to some criteria, put you script on your good all CentOS 7 server, but suddenly you encounter the error:

PHP Fatal error: Call to undefined function posix_kill() in ./kill-process.php on line 64
Killed process id: 21899
PHP Fatal error: Call to undefined function posix_kill() in ./kill-process.php on line 64
Killed process id: 21899
PHP Fatal error: Call to undefined function posix_kill() in ./kill-process.php on line 64

So you just missed a php plugin and there it is: “php-process”. Just install it with yum!

yum install -y php-process

And the error will disappear and the next time you want to kill some bad process you’ll sure do it!