Rogue site control and php-fpm modification

Hi,

I'm trying to deal with some of the issues of rogue sites or, in other words, the possibility for a site to get hacked and used for other purposes, consuming cpu and other resources.

The problem happens when a site gets hacked and starts scanning the internet, or doing other damage. The hosting server gets slow and other customers start complaining.

A viable solution would be to use cloudlinux and it's apache module that will put all requests made from a site in it's own lve (i.e. context). This works well, but is $$ in licenses and not open source at all.

One way of dealing with this would be to leverage the systemd sockets and "multi-master" php-fpm setup. The pieces are there , but need to be assembled in a proper way (for example):

1. systemd can place each process in it's own cgroup limiting cpu, memory and via a proper TC setup the network bandwidth. To do this, each process must be started from systemd via a listening socket (i.e. on demand, which also uses less resources than dynamic php-fpm).

2. php-fpm can be run in multiple-master mode, where each site pool is served by a single master and a single socket. Each of these masters will have a single pool corresponding to the web site. The master can be started from it's own socket by systemd.
The above will require a some rewrite of the current templates. Some new configuration templates for systemd will be necessary. With the addition of some "systemctl daemon-reload" and enable/disable commands all could work automagically.

I'd like to ask @till or the other developers what they think of this idea and if there're issues with this approach.

ispcomm.

 

I'll do a mock up of the setup and see if it works and try to find problems then.
If it works, it won't require special apache/nginx modules or patches to fpm.

 

Till, I'm trying to set up a tidy and clean dev environment.
I have a question:
How do you keep track of what you change in /usr/local/ispconfig to copy back to your clean git checkout directory ?
My idea was to install ispconfig as usual then "git init" the /usr/local/ispconfig directory, and then extract patches from there to apply to the clean ispconfig git that I checkout somewhere else.
Is there a better way?
Thank you.

 

Thanks. How do you debug code running on your remote installs and how to you sync your local copy to the remote install ? I found this to be the most time consuming part when doing it manually and a running debugger speeds up code testing.

Unfortunately I tend to complicate things. My current setup is that I use a bunch of different workstations in different locations, and to make things harder, I use different notebooks running windows, osx and linux.
When inspiration comes, I like to be able to work on whatever I have within my hands, being it my desktop in the office or the specific notebook I took with me on a trip.
For ispconfig I have now setup a dockerized runtime with php xdebug and code with netbeans. I work on my private git in a local branch and netbeans copies changed files to the docker container at start. All this is synced to a private gitlab server and changes, when ready, (will be) pushed to my area on your ispconfig gitlab server for PR issued there.
I'm trying to replace netbeans with vs code but cannot get it to sync nicely to the remote server (will have to look at it asap).

 

I come from there too. Echoing debug trough the serial port of an embedded device. However it will take me months to read the code of ispconfig and understand it. A debugger makes is much easier to get going in the projects of someone else

Anyway, I'm up and running since few days now. This php separation is my top #2 priority now. #1 is the remoting api mod for predefined domain_id (the other thread). As soon as that is done, I'll jump on this issue.

 

So, here I am after a short break, bringing some good news.

I mocked up a php-fpm install using separated fpm master processes. They're started by systemd and they're all ondemand. They all get their cpu slice in a separated cgroup which allows us to tweak max execution cap (for example). When idle, the child worker processes expire and only the master fpm process stays in memory, holding the php socket.
This extra process can be cleaned up by a cron script run at predefined intervals, killing master processes without childs. I have a p.o.c. script in the workings.
Further, the master process can be started directly via systemd via it's socket, when a request comes by. This will also allow for a bigger density of sites on the same server.
There's also a benefit for log files. As each site now has it's own fpm process, we could put the fpm log file in the "logs" of the site. This will help with customer support and leave diagnosis on the 5xx errors to themselves.

So, now I'd like to implement this in ispconfig.

For compatibility reasons with existing setups, I'd see this as a new fpm "plugin", and not an extension to the current one, much like the hhvm was implemented. Then it would be easy to switch sites from one version or the other, even on the same server.

The implementation itself is straight forward.
I have created a new directory (on debian /etc/php5/fpm/sites.d) where I put the config for each site.
Then I need to put two more files in the /etc/systemd/system directory. One for the fpm service dedicated to that site and one for the corresponding socket. A few more system commands will enable the new configuration in systemd and the fpm is ready for operation.

Comments welcome, before I start implementing this. I'd be specially interested if you think that it would be better to add a "fpm-mode" parameter or similar to the php-fpm implementation instead of adding a completely new php type (there are a lot of them altready).

 

I agree that a hoster will probably set this globally. But let's assume that you stick with the current model (less processes globally, same settings for everyone) which is fine globally, you want nevertheless to be able to fine tune a single website, or maybe a group of websites based on the package. So I think that a+c is the best option. On the other hand I think that b+d are no options, because you do not want to let your customers choose the resources they want to use...

 

I see this as an automatic protection mode for the server, so it is better enabled for all sites. As such, I too agree, that it shall be out of the user control, and in the hands of the admin.
To handle the transition period, however, it might be better enabled per-site so that we can enable on current servers too, without major dramas.

From another perspective, I'm trying to imagine reasons not to enable this. One reason would be (old) systems without systemd.
There's one more process per site, when the site is serving. The current usage of this process (debian jessie standard ispconfig install) is 12MB unique memory and 16MB shared memory for a total RSS of 28MB. Each additional process eats 12MB of additional unique memory.
For 100 sites running at the same time, this is 120MB total additional memory.

I think this is somewhat manageable.

I've been looking how hhvm was implemented in ispconfig. It's a little scary and modifications are everywhere. I'm not sure if my knowledge of ispconfig is up to the task of implementing this. What would be the best way to proceed from here?

 

I have a question: how do I add a per-site setting that is only manageable by the admin and not the customer ?

 

Jesse,
I have not dug enough inside ispconfig programming to answer your last question precisely. If I'm correct, multiple php versions will work automagically as I'll be working below that level. Hopefully @till will shed some light when needed.
The path to configurations is a variable in the server php tab. It takes whatever is on the server by default. On a xenial distro, stuff sits in /etc/php/7.0/fpm, while on a jessie it's in /etc/php5. I haven't seen a stretch yet.
To deal with this variations, I'll probably add another parameter because the default php behavior is to include all pools in the pool.d in the default fpm process. This might require a modification to the install script too.
You're right about memory (ouch). Its 1.2GB. But consider that each php process from the worker pools has a minimum of 64MB by itself, and you probably have 2 to 5 workers per site. In perspective I'd be willing to pay a little extra memory for the benefits a separated fpm space gives.
At this point I have a working ispconfig install where the fpm startup scripts were changed by hand, just to make sure the final setup I had in mind can work. I'm working on this as time permits, but I'm trying to get it done asap to deploy it to my servers.

 

I made some progress so I'll update the thread.
I added the option for "isolated process" for the fpm pool on the web sites and modified the templates for the creation of the webxxx.conf pool file. server.php now creates the correct file depending on the option.
Current TODO:
-) Add an extra directory option in server config to store these files. It cannot be the same directory where the normal pool are sourced from (because they're included in full from the standard fpm configuration files).
-) Generate the systemd service and socket configurations (partially done)
-) update server.php to start/stop the isolated systemd pool instead of the big fpm pool.
-) create a script to terminate pools with zero workers and restart their socket auto-start

 

@till: here's the current state https://git.ispconfig.org/ispcomm/ispconfig3/tree/fpm-isolated-process
It is p.o.c. and isolated process has been implemented in the apache2 plugin. Most of the things work. Enabling/disabling the isolated pool will update systemd based services and fpm based services and restart what is needed.
There are 2 new variables in the server config. 1 for setting the directory where the isolated pool configs are stored and 1 for setting the default php handler for fpm.
The sites have one setting which is "isolated fpm process".
There's an update_085 sql to add the new site parameter to the database.

Let me know if you prefer that I make a PR so you can look at the differences. Your input is wellcome.

Side note: I found a bug in the current install (master): It fails on a new server in the ispconfig3.sql database restore because there's a duplicate "symantec.com" value in the CA index

EDIT: TODO
To implement the multiple php-fpm versions in the case of isolated process, I will need to add a record to the server_php containing the full path to the fpm handler (/usr/sbin/php-fpmX.Y). This is needed in the systemd configuration files.
However, the way this is saved now in the site config in the database is by concatenating all values in 'fastcgi_php_version' in a positional way separated by ':'. I can append the new parameter at the end and this will be compatible with existing installations. I'm not sure is we'll need a specific upgrade process to go trough all sites and update this column of the db. Any ideas?

EDIT2: So, I went ahead and implemented the multiple php versions switching. works but we'll need to handle the upgrades of existing installs.

I'll wait now for your comments before doing any more work.

 

I also have another question: I temporarily hardcoded the location of the systemd config files to /etc/systemd/system.

What is the best way to add this config setting to ispconfig ? I see there's a distribution-specific $conf array in install.php but I'm not sure how this is used later. Is it in a configuration file or is it in the database?