ISPConfig 3.0.1.4 causes Apache to have too many open files

Discussion in 'Installation/Configuration' started by gkovacs, Oct 3, 2009.

  1. gkovacs

    gkovacs Member

    We have encountered a very weird error after installing ISPConfig 3.0.1.4.
    Basically it causes Apache/mod_php to create thousands of open files (actually FIFO pipes), and they are staying open until the system runs out.

    REFERENCE
    On our production servers running ISPConfig 3.0.1.3, lsof |wc-l shows 2-5000 open files, and if traffic is declining this number decreases fast, no problem there.
    On newly installed servers with 3.0.1.3, there is no problem.

    ERROR
    Ou our newly installed server with 3.0.1.4, after clicking through the web interface a couple of times (and maybe some PHPMyAdmin pages as well, to increase load a bit), lsof shows a growth to 10-20000 open files in a minute, and they never close.
    After a couple of minutes, the web server slows to a halt and PHP pages do not load anymore.

    ENVIRONMENT
    We have tested this on several different (freshly installed) servers:

    - Ubuntu 8.04 with perfect server setup (OpenVZ)
    Apache/2.2.8 (Ubuntu) PHP/5.2.4-2ubuntu5.7

    - Debian 5 with perfect server setup (OpenVZ)
    Apache/2.2.9 (Debian) PHP/5.2.6-1

    The only difference is that on Debian, restarting apache actually closes the open files, while on Ubuntu they are kept open even then.
     
  2. damir

    damir New Member

    This could be related to OpenVZ, beacuse i can't reproduce it on non vps server and on esxi vps. I'm running multi server setup (ispconfig 3.0.1.4) on debian 5.0.3.
     
  3. till

    till Super Moderator Staff Member ISPConfig Developer

    I'am also not able to reproduce this on my servers. Also the ispconfig interface is basically a normal php script, it is not able to keep files open.

    Maybe you use fcgi and have a high number of fcgi children configured. Check the server settings and set fcgi children to 1. You will have to open the settings of your website(s), cahnge a value and and save it to update this setting for a site.
     
  4. gkovacs

    gkovacs Member

    First of all, I'm not using FCGI (it's not even installed).
    Second, there are no sites, only one that has PHPMyAdmin installed under it for the testing.
    Third, the problem does not manifest itself until ISPConfig 3.0.1.4 has been installed.

    Basically, the ONLY DIFFERENCE between the working and the problematic server is the version of ISPConfig installed.

    This is what I do for every VPS on the host (needed for Pure-FTPD):
    Code:
    cat >vpscap.sh
    VPSID=XXX
    for CAP in CHOWN DAC_READ_SEARCH SETGID SETUID NET_BIND_SERVICE NET_ADMIN SYS_CHROOT SYS_NICE CHOWN DAC_READ_SEARCH SETGID SETUID NET_BIND_SERVICE NET_ADMIN SYS_CHROOT SYS_NICE
    do
      vzctl set $VPSID --capability ${CAP}:on --save
    done
    
    chmod +x vpscap.sh
    ./vpscap.sh
    And here is what I do before installing ISPConfig:
    Code:
    apt-get update
    apt-get dist-upgrade
    apt-get install mc psmisc wget man ftp lsof
    
    apt-get install mysql-client openssl apache2 apache2.2-common apache2-doc apache2-mpm-prefork apache2-utils libexpat1 ssl-cert libapache2-mod-php5 php5 php5-common php5-gd php5-mysql php5-imap php5-cli
    a2enmod suexec rewrite ssl include actions
    /etc/init.d/apache2 restart
    
    modprobe capability
    echo 'capability' >> /etc/modules
    apt-get install pure-ftpd-common pure-ftpd-mysql quota quotatool
    echo 'yes' > /etc/pure-ftpd/conf/DontResolve
    echo '50000 10' > /etc/pure-ftpd/conf/LimitRecursion
    
    apt-get install vlogger webalizer cron
    Are there any Apache or vhost configuration differences between 3.0.1.3 and 3.0.1.4?
     
  5. till

    till Super Moderator Staff Member ISPConfig Developer

    No, nothing has changed in this area.

    Please run:

    ps aux

    and check if you have a high number of php processes.
     
  6. ezhandossov

    ezhandossov New Member

    Hi! I think I have same problem, but on real server - single server setup (ISPConfig 3.0.1.4) on Debian Lenny.

    Problem occurs after some time of working in control panel (if i don't use it everything works fine).

    Problem is php get error "Failed to open stream. Too many open files..." while file-nr is about 1000 times smaller than file-max.
     
  7. damir

    damir New Member

    Are you both running single server or multi server setups?

    I'm running multi server setup, with the interface on a signle server and lsof | wc -l stays between 2100 - 2500. I will keep my eye on it during the day to see if it grows.
     
  8. ezhandossov

    ezhandossov New Member

    I'am new to ISPconfig so I don't clearly undestand this question. If you mean that all SW servers (apache, mysql etc.) runing on the same HW server (PC) than YES I running single server setup. If you mean something different, please explain.

    P.S. sorry for my english
     
  9. till

    till Super Moderator Staff Member ISPConfig Developer

    Please post the output of:

    ps aux | grep php
     
  10. ezhandossov

    ezhandossov New Member

    There is nothing except GREP itself
     
  11. clixclix

    clixclix ISPConfig Developer ISPConfig Developer

    Hi,
    I'm having the same problem on a fresh Debian 5.0.3 64bit server, with ISPConfig 3.0.1.4 installed as "slave" server.

    Here is an example of the error I get, trying to add a client:
    Code:
    Warning: listform_actions::include(/usr/local/ispconfig/interface/lib/lang/en.lng) [listform-actions.include]: 
    failed to open stream: Too many open files in /usr/local/ispconfig/interface/lib/classes/listform_actions.inc.php on line 143
    Some more info:
    Code:
    /home/admin# lsof |wc -l
    6192
    Code:
    /home/admin# ps aux | grep php
    root      7593  0.0  0.0   5160   784 pts/0    S+   23:02   0:00 grep php
    I get the same "Too many open files in ..." error randomly across other sections.


    Also, any website creation done from the master panel to the slave server remains forever in the monitor jobqueue of the master.... maybe it's related.

    The master server works correctly on itself.
     
  12. till

    till Super Moderator Staff Member ISPConfig Developer

    Ok, but if thats a slave, then you can not run a interface there and you can not get this error message on the slave. Are you really sure that this error message was not shown on the master thats runs the interface?


    Thats most likely not related. Its just a mysql permission problem. Make sure that the master and slave are able to resolve the hostnames of each other e.g. by adding them in bost /tc/hosts files. otherwise the mysql authentication will fail and no data gets replicated.
     
  13. clixclix

    clixclix ISPConfig Developer ISPConfig Developer

    Probably the interface is there because I first installed ISPConfig3 in normal mode, then I deleted the ispconfig database and reinstalled all in expert mode in order to let it work as slave.
    Should I remove everything and reinstall everything from scratch?
    Should I just remove the admin interface application on slave? How can I do that?



    Ok, I fixed that: the slave created on the master a database user with access permission on the wrong IP address. The cause is likely to be that on the slave I have more than one IP address on the same interface.
     
  14. till

    till Super Moderator Staff Member ISPConfig Developer

    No, thats not nescessary as long as you do not add anything through the interface on the slave as this would cuase conflicts with the amster and replication will fail.
     
  15. gkovacs

    gkovacs Member

    Has this bug been found and fixed since 3.0.1.4?
     
  16. till

    till Super Moderator Staff Member ISPConfig Developer

    This is no behaviour that could be reproduced on any other server by one of the developers.
     
  17. ezhandossov

    ezhandossov New Member

    I don't know about all other, but I don't have this problem any more. It just disappear... with no actions from me...
     
  18. gkovacs

    gkovacs Member

    Well, I just created a new ISPConfig installation based on 3.0.1.6, and the problem is still there. After a couple of clicks on PHP pages, lsof climbs to the tens of thousands.

    If any of the developers are interested, I can give you SSH and HTTP access to this server. Message or email me and I'll send you the login details.
     
    Last edited: Nov 13, 2009
  19. SamTzu

    SamTzu Active Member

    I have the same problem with .1.6.
    On the server there is only myDNS, apache2 and mySQL installed.
    No slave, only master.
     
  20. till

    till Super Moderator Staff Member ISPConfig Developer

    The problem of the first poster was that he had manually changed mysql login details and forgot to change it in /etc/vlogger-dbi.conf too which caused the problems on his server. So it was not a bug in ISPConfig.

    @SamTzu: Please make sure that the database login details in /etc/vlogger-dbi.conf are correct.
     

Share This Page