Socket file that ISPConfig references in nginx config does not exist by default

I just installed ISPConfig 3.0.5.4, with nginx, on a brand new box.

Immediately after finishing the ISPConfig installation, I attempted to access the interface at https://example.com:8080, and received "502 Bad Gateway".

A quick look at the log file (/etc/nginx/error.log) reveals the reason:

Code:

2014/07/23 20:54:10 [error] 4580#0: *8 connect() to unix:/var/lib/php5-fpm/ispconfig.sock failed (111: Connection refused) while connecting to upstream, client: 1.2.3.4, server: _, request: "GET / HTTP/1.1", upstream: "fastcgi://unix:/var/lib/php5-fpm/ispconfig.sock:", host: "example.com:8080"

The socket file referenced in the ISPConfig vhost (/etc/nginx/sites-available/ispconfig.vhost [line 28]) references the following socket:

Code:

fastcgi_pass unix:/var/lib/php5-fpm/ispconfig.sock;

Yet, the socket file does not actually exist on the filesystem.

Is this a known issue? Am I supposed to create the socket file manually?

Just trying to determine whether or not I should open a bug report. Surely, others have run into this.

Thanks!

UPDATE: Huh! After about 10 minutes, the socket file appeared at the expected location, /var/lib/php5-fpm/ispconfig.sock. I'm sure there is some "good reason" for the delay.

But, I'm still seeing an error. The original error

Code:

connect() to unix:/var/lib/php5-fpm/ispconfig.sock failed (2: No such file or directory)

changed to the following, once the socket file mysteriously showed-up

Code:

connect() to unix:/var/lib/php5-fpm/ispconfig.sock failed (111: Connection refused)

Here are the socket details:

Code:

# stat /var/lib/php5-fpm/ispconfig.sock
  File: â/var/lib/php5-fpm/ispconfig.sockâ
  Size: 0               Blocks: 0          IO Block: 4096   socket
Device: ca01h/51713d    Inode: 1632266     Links: 1
Access: (0660/srw-rw----)  Uid: ( 5003/ispconfig)   Gid: ( 5004/ispconfig)
Access: 2014-07-23 21:00:53.696859916 +0000
Modify: 2014-07-23 21:00:53.696859916 +0000
Change: 2014-07-23 21:00:53.696859916 +0000
 Birth: -

Thanks for any ideas!

UPDATE 2: I restarted the box and now everything works. Maybe the php5-fpm service had not been started (or failed to restart), or something along those lines.

Sooo... never mind. Maybe this will help someone else that gets trigger-happy immediately after a fresh installation.

UPDATE 3: I had actually installed ISPConfig 3.0.5.3 (not 3.0.5.4 as stated above). I later realized the reason: ISPConfig 3.0.5.3 is listed as the latest version at http://www.ispconfig.org/files/file/2-ispconfig/ . Should this be changed to 3.0.5.4?

 

see also http://www.howtoforge.com/forums/showthread.php?goto=newpost&t=66473

 

Whcih distribution you are using?

Socket files are not created manually, they appear after proper successful service running status of php-fpm.

Obviously goingto updated ISPConfig version will help.

You must check your php-fpm running status to diagnose the socket error

 

Thanks for the input, guys.

I setup another server today and the same issue occurred (and I installed 3.0.5.4 this time).

Again, a full system reboot fixed it. I did try "service php5-fpm start" before that, and the service appears to start (although, I didn't check its log file).

Not sure what is getting disconnected, but that seems to be the solution: a reboot.

 

Reboot works but you can kill off any php-fpm processes and then restart the php5-fpm service.

This seems to only happen right after a fresh ispconfig install.

Think it is something about upstart bug with php-fpm when using reload.

 

Srijan, I forgot to answer your question re: which distro I am using: Ubuntu 14.04.1 LTS.

Guys, I'm convinced that there is a larger problem at play, after what happened last night.

I had made some very minor changes to php.ini during the day, and so decided to wait until late at night (off-peak hours) to reload (not restart) php5-fpm and make the changes effective.

To my (moderate) surprise, when I reloaded the php5-fpm service, the socket problem returned!

This time, every single vhost on the box (excepting only ISPConfig's) was down, with "502 Bad Gateway". Checking several vhosts' log files revealed the same exact issue: the sockets are present, the permissions and ownership look correct, but they refuse connections from nginx.

The entries in the nginx logs look like this (exact same message as 2/3 down in my OP):

Code:

connect() to unix:/var/lib/php5-fpm/web1.sock failed (111: Connection refused) while connecting to upstream

Restarting php5-fpm with "service php5-fpm restart" did not work, nor did completely stopping and starting php5-fpm. In panic-mode, I didn't think to go into top/htop and kill-off any still-running php5-fpm processes, but I don't think it would have made a difference.

The only thing that fixes this, in my limited experience with it, is a full system reboot. Obviously, there must be a simpler, less disruptive way to correct this problem when it happens, but a 2-minute long system reboot cycle was the most attractive option.

The file properties look like this (although, this was after I rebooted, at which time the problem was resolved, but this looks the same as what I posted in my OP):

Code:

# stat /var/lib/php5-fpm/web1.sock
  File: ‘/var/lib/php5-fpm/web1.sock’
  Size: 0               Blocks: 0          IO Block: 4096   socket
Device: ca03h/51715d    Inode: 205676      Links: 1
Access: (0660/srw-rw----)  Uid: ( 5004/    web1)   Gid: ( 5005/ client1)
Access: 2014-07-29 19:33:34.876304994 -0700
Modify: 2014-07-29 19:33:34.876304994 -0700
Change: 2014-07-29 19:33:34.876304994 -0700
 Birth: -

The state of this issue makes me extremely nervous. Seriously. I can't sleep at night knowing that if php5-fpm is reloaded or restarted for any reason, every vhost might be taken offline. I'm constantly waiting for my beeper to go off (happens whenever a site is down).

This is especially nerve-wracking given that certain end-user-initiated actions may force php5-fpm to be reloaded, which puts the issue out of my control, and means it could happen at just about any time.

I am working on a virtual replica of this server as we speak, so I can perform some testing without risk of a crisis on a heavily-trafficked production server.

Has anybody else observed this most troubling behavior? Surely, my setup is not unique. (It's based largely on Falko's "Perfect Server" series, with very few deviations therefrom.)

Thanks for any help with this!

 

In the meantime, I will be changing every single vhost to use TCP/IP instead of a socket.

And for the record, I am still bothered by the fact that ISPConfig's vhost did not suffer the same socket fate. For whatever reason, I could still reach the ISPConfig vhost when all others were responding with "502 Bad Gateway".

UPDATE:

After searching around the Internet some more, I'm not convinced that the problem is socket-specific. dcurrey, I think you were correct in that this is a php5-fpm bug relating to child process management.

Here's a bug report that may be describing the same issue:

https://bugs.launchpad.net/ubuntu/+source/php5/+bug/1242376

And a duplicate report of the same bug:

https://bugs.launchpad.net/ubuntu/+source/php5/+bug/1279819

Sounds exactly correct.

Again, sounds the same.

This is a major ****ing problem. The bug was reported to Ubuntu on 2013-10-20 and still isn't fixed. Unbelievable. Well, I think I've found the culprit. Now, I will sleep even less easy at night!

The fact that this is only a problem with reload (and not a full stop/start) makes every vhost on the server susceptible to being knocked offline when a single ISPConfig user makes a single change to a vhost. Gaaaaaaahhhh!!!

 

Killing them off does work. Just make sure they are all killed off before restart or your restart will fail.

Like I mention before its a problem with debian/ubuntu php implementation. (Bug #1242376) This think this has been fixed in the ppandrej branch but for some reason hasn't worked its way down.

Just use "service php5-fpm restart" and you should be fine. I have had running ispconfig server on both Ubuntu 13.10 and 14.04 (both have this problem) and no issues as long as I use "restart"

edit: Now that I think of it I was using the ondrej PPA when I was using 13.10

 

Thanks, dcurrey. It looks like I beat you to the punch with my update/edit. You were spot-on with the culprit.

When you say,

are you accounting for the fact that ISPConfig may/will attempt to issue a reload to php5-fpm whenever you change the properties of a given vhost? I could be wrong here, but doesn't ISPConfig rewrite the vhost config file and then attempt to reload php5-fpm whenever a vhost's configuration is changed? And if so, wouldn't that cause this same problem scenario?

We need to put together explicit instructions for fixing this on Ubuntu 13 and 14... this is a show-stopping issue. I'll volunteer if I can figure out how to fix it on my own setup first.

UPDATE:

Users report that using ondrej's php5 PPA ( https://launchpad.net/~ondrej/+archive/ubuntu/php5 ) fixes the problem. I'll try that and report back with explicit, step-by-step instructions.

 

I am thinking ispconfig is already taking into account this bug. I swear I remember seeing something about the ispconfig maintainer making changes to accommodate 14.04 with this bug. This was the reason for perfect server 14.04 delay. I could be going crazy however.

Anyway this should be easy enough to test.

 

Step-by-step instructions for fixing this (on Debian or Ubuntu; tested on Ubuntu 14.04.1 LTS):

1.)

Add the following sources to /etc/apt/sources.list and save the file:

Code:

# REQUIRED to fix a show-stopping bug in php5-fpm.
# See: https://bugs.launchpad.net/ubuntu/+source/php5/+bug/1242376

deb http://ppa.launchpad.net/ondrej/php5/ubuntu trusty main
deb-src http://ppa.launchpad.net/ondrej/php5/ubuntu trusty main

2.)

Execute the following commands (as root, or use "sudo" as necessary):

Code:

# apt-get install software-properties-common

# add-apt-repository ppa:ondrej/php5

3.)

Code:

# apt-get update

4.)

Perform a dry-run first, to ensure that nothing unexpected will occur:

Code:

# apt-get --dry-run upgrade

This should produce the following output:

Code:

Reading package lists... Done
Building dependency tree
Reading state information... Done
Calculating upgrade... Done
The following packages have been kept back:
  php-pear php5-cli php5-common php5-curl php5-dev php5-fpm php5-gd php5-imap
  php5-intl php5-mcrypt php5-mysql php5-pspell php5-readline php5-recode
  php5-snmp php5-sqlite php5-tidy php5-xmlrpc php5-xsl
The following packages will be upgraded:
  accountsservice libaccountsservice0 libjson-c2 libjson0 liblzo2-2
  php5-imagick php5-json php5-memcache php5-ps php5-xcache python3-gi
11 upgraded, 0 newly installed, 0 to remove and 19 not upgraded.

Notice that several PHP packages have been kept back. Those need to be upgraded, explicitly, to continue.

Code:

# apt-get install php-pear php5-cli php5-common php5-curl php5-dev php5-fpm php5-gd php5-imap php5-intl php5-mcrypt php5-mysql php5-pspell php5-readline php5-recode php5-snmp php5-sqlite php5-tidy php5-xmlrpc php5-xsl

This should produce the following output:

Code:

Reading package lists... Done
Building dependency tree
Reading state information... Done
The following extra packages will be installed:
  dh-php5 libonig2 libqdbm14
The following NEW packages will be installed:
  dh-php5 libonig2 libqdbm14
The following packages will be upgraded:
  php-pear php5-cli php5-common php5-curl php5-dev php5-fpm php5-gd php5-imap
  php5-intl php5-mcrypt php5-mysql php5-pspell php5-readline php5-recode
  php5-snmp php5-sqlite php5-tidy php5-xmlrpc php5-xsl
19 upgraded, 3 newly installed, 0 to remove and 11 not upgraded.
Need to get 5,795 kB of archives.
After this operation, 238 kB disk space will be freed.
Do you want to continue? [Y/n] Y

If notified that one or more of PHP's "ini" files have local modifications, choose the appropriate action for your particular installation. The "safest" option is to install the package maintainer's version, but doing so could wipe-out any customizations that you may have made to your ini files. The best course of action is to perform a "diff" for each file that is flagged and decide which action to take based on the observed changes.

Once the installation completes, we the final step is to perform "upgrade" one more time:

Code:

# apt-get upgrade
Reading package lists... Done
Building dependency tree
Reading state information... Done
Calculating upgrade... Done
The following packages will be upgraded:
  accountsservice libaccountsservice0 libjson-c2 libjson0 liblzo2-2
  php5-imagick php5-json php5-memcache php5-ps php5-xcache python3-gi
11 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Need to get 692 kB of archives.
After this operation, 9,216 B disk space will be freed.
Do you want to continue? [Y/n] Y

Once the upgrade is complete, it should be possible to issue

Code:

# service php5-fpm reload

without php5-fpm crashing and becoming unresponsive.

Good luck!

 

Never mind you already got it.