Hello All, I wonder if someone might be able to help. It seems that the lockfile on the web server within our multi-server setup is not being removed. In the ISPConfig logs we see: 2010-11-17 17:11 xxx.xxx.com Debug There is already a lockfile set. Waiting another 10 seconds... 2010-11-17 17:11 xxx.xxx.com Debug There is already a lockfile set. Waiting another 10 seconds... 2010-11-17 17:10 xxx.xxx.com Debug No Updated records found, starting only the core. 2010-11-17 17:10 xxx.xxx.com Debug Set Lock: /usr/local/ispconfig/server/temp/.ispconfig_lock Each time we delete the lock file, the queue is able to run, but before long it's back in there stopping future tasks. Thanks David
Please delete the lock file and run Code: /usr/local/ispconfig/server/server.sh manually. Do you get any errors?
I've got the same problem. I was removed lock file and run server.php manually; this is cron.log of ispconfig: Code: Resolving www.ispconfig.org... 78.46.59.59 Connecting to www.ispconfig.org|78.46.59.59|:80... connected. HTTP request sent, awaiting response... 200 OK Length: 2976564 (2,8M) [application/x-gzip] Saving to: `ISPConfig-3.0.3.tar.gz' 0K .......... .......... .......... .......... .......... 1% 451K 6s 50K .......... .......... .......... .......... .......... 3% 1,28M 4s 100K .......... .......... .......... .......... .......... 5% 1,33M 3s 150K .......... .......... .......... .......... .......... 6% 843K 3s 200K .......... .......... .......... .......... .......... 8% 101M 3s 250K .......... .......... .......... .......... .......... 10% 4,57M 2s 300K .......... .......... .......... .......... .......... 12% 1,67M 2s 350K .......... .......... .......... .......... .......... 13% 6,55M 2s 400K .......... .......... .......... .......... .......... 15% 1,80M 2s ... 2900K ...... 100% 12,6M=0,6s 2010-11-19 23:36:15 (5,04 MB/s) - `ISPConfig-3.0.3.tar.gz' saved [2976564/2976564] PHP Deprecated: Comments starting with '#' are deprecated in /etc/php5/cli/conf.d/imagick.ini on line 1 in Unknown on line 0 PHP Deprecated: Comments starting with '#' are deprecated in /etc/php5/cli/conf.d/imap.ini on line 1 in Unknown on line 0 PHP Deprecated: Comments starting with '#' are deprecated in /etc/php5/cli/conf.d/mcrypt.ini on line 1 in Unknown on line 0 hostname: Name or service not known PHP Notice: Undefined offset: 0 in /tmp/ispconfig3_install/install/autoupdate.php on line 89 PHP Notice: Undefined variable: hosts in /tmp/ispconfig3_install/install/lib/installer_base.lib.php on line 323 hostname: Name or service not known The value of variable $myhostname is "", but should have been a fully qualified domain name; perhaps uname(3) did not provide such. You must explicitly assign a FQDN of this host to variable $myhostname in /etc/amavis/conf.d/05-node_id, or fix what uname(3) provides as a host's network name! PHP Deprecated: Comments starting with '#' are deprecated in /etc/php5/cli/conf.d/imagick.ini on line 1 in Unknown on line 0 PHP Deprecated: Comments starting with '#' are deprecated in /etc/php5/cli/conf.d/imagick.ini on line 1 in Unknown on line 0 PHP Deprecated: Comments starting with '#' are deprecated in /etc/php5/cli/conf.d/imap.ini on line 1 in Unknown on line 0 PHP Deprecated: Comments starting with '#' are deprecated in /etc/php5/cli/conf.d/mcrypt.ini on line 1 in Unknown on line 0 PHP Fatal error: Out of memory (allocated 1512046592) (tried to allocate 261900 bytes) in /usr/local/ispconfig/server/lib/app.inc.php on line 69 ... 19.11.2010-23:51 - WARNING - DB::connect()-> mysql_select_db MySQL server has gone away 19.11.2010-23:51 - WARNING - DB::connect()-> mysql_select_db MySQL server has gone away 19.11.2010-23:51 - WARNING - DB::connect()-> mysql_select_db MySQL server has gone away 19.11.2010-23:51 - WARNING - DB::connect()-> mysql_select_db MySQL server has gone away 19.11.2010-23:51 - WARNING - DB::connect()-> mysql_select_db MySQL server has gone away 19.11.2010-23:51 - WARNING - DB::connect()-> mysql_select_db MySQL server has gone away 19.11.2010-23:51 - WARNING - DB::connect()-> mysql_select_db MySQL server has gone away 19.11.2010-23:51 - WARNING - DB::connect()-> mysql_select_db MySQL server has gone away 19.11.2010-23:51 - WARNING - DB::connect()-> mysql_select_db MySQL server has gone away 19.11.2010-23:51 - WARNING - DB::connect()-> mysql_select_db MySQL server has gone away 19.11.2010-23:51 - WARNING - DB::connect()-> mysql_select_db MySQL server has gone away 19.11.2010-23:51 - WARNING - DB::connect()-> mysql_select_db MySQL server has gone away 19.11.2010-23:51 - WARNING - DB::connect()-> mysql_select_db MySQL server has gone away 19.11.2010-23:51 - WARNING - DB::connect()-> mysql_select_db MySQL server has gone away ... 19.11.2010-23:51 - WARNING - DB::connect()-> mysql_select_db MySQL server has gone away PHP Fatal error: Out of memory (allocated 1497104384) (tried to allocate 261900 bytes) in /usr/local/ispconfig/server/lib/app.inc.php on line 69 and this is the event log monitor in Ispconfig: Code: 2010-11-19 23:54 ns1.pcrednet.net Debug There is already an instance of server.php running. Exiting. 2010-11-19 23:53 ns1.pcrednet.net Debug There is already an instance of server.php running. Exiting. 2010-11-19 23:52 ns1.pcrednet.net Debug There is already an instance of server.php running. Exiting. 2010-11-19 23:51 ns1.pcrednet.net Debug There is already an instance of server.php running. Exiting. 2010-11-19 23:50 ns1.pcrednet.net Debug There is already an instance of server.php running. Exiting. 2010-11-19 23:49 ns1.pcrednet.net Debug Found 36 changes, starting update process. 2010-11-19 23:49 ns1.pcrednet.net Debug Set Lock: /usr/local/ispconfig/server/temp/.ispconfig_lock 2010-11-19 23:49 ns1.pcrednet.net Debug There is already an instance of server.php running. Exiting. 2010-11-19 23:48 ns1.pcrednet.net Debug There is already an instance of server.php running. Exiting. 2010-11-19 22:35 ns1.pcrednet.net Debug Found 34 changes, starting update process. 2010-11-19 22:35 ns1.pcrednet.net Debug Set Lock: /usr/local/ispconfig/server/temp/.ispconfig_lock 2010-11-19 22:34 ns1.pcrednet.net Debug There is already a lockfile set. Waiting another 10 seconds... 2010-11-19 22:34 ns1.pcrednet.net Debug There is already a lockfile set. Waiting another 10 seconds... 2010-11-19 22:34 ns1.pcrednet.net Debug There is already a lockfile set. Waiting another 10 seconds... 2010-11-19 22:34 ns1.pcrednet.net Debug There is already a lockfile set. Waiting another 10 seconds... Job queue task is waiting for 36hrs... Thx in advance.
Now; i delete lock file and run manually server.sh. Code: /usr/local/ispconfig/server# sh server.sh PHP Deprecated: Comments starting with '#' are deprecated in /etc/php5/cli/conf.d/imagick.ini on line 1 in Unknown on line 0 PHP Deprecated: Comments starting with '#' are deprecated in /etc/php5/cli/conf.d/imap.ini on line 1 in Unknown on line 0 PHP Deprecated: Comments starting with '#' are deprecated in /etc/php5/cli/conf.d/mcrypt.ini on line 1 in Unknown on line 0 PHP Warning: mysql_connect(): Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2) in /usr/local/ispconfig/server/lib/classes/db_mysql.inc.php on line 78 PHP Warning: mysql_errno() expects parameter 1 to be resource, boolean given in /usr/local/ispconfig/server/lib/classes/db_mysql.inc.php on line 63 PHP Warning: mysql_error() expects parameter 1 to be resource, boolean given in /usr/local/ispconfig/server/lib/classes/db_mysql.inc.php on line 64 Can't connect to mysql??¿?¿?¿¿?
netstat -tap output: Code: /usr/local/ispconfig/server# netstat -tap Active Internet connections (servers and established) Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name tcp 0 0 *:postgresql *:* LISTEN 363/postgres tcp 0 0 localhost.localdoma:953 *:* LISTEN 9828/named tcp 0 0 *:smtp *:* LISTEN 8746/master tcp 0 0 *:https *:* LISTEN 9791/apache2 tcp 0 0 *:2812 *:* LISTEN 13824/monit tcp 0 0 *:2082 *:* LISTEN 9791/apache2 tcp 0 0 localhost.localdo:10024 *:* LISTEN 8973/amavisd (maste tcp 0 0 localhost.localdo:10025 *:* LISTEN 8746/master tcp 0 0 *:mysql *:* LISTEN 9870/mysqld tcp 0 0 localhost.localdo:spamd *:* LISTEN 558/spamd.pid tcp 0 0 *:www *:* LISTEN 9791/apache2 tcp 0 0 *:tproxy *:* LISTEN 9791/apache2 tcp 0 0 localhost.locald:domain *:* LISTEN 9828/named tcp 0 0 *:ftp *:* LISTEN 9809/pure-ftpd (SER tcp 0 0 *:munin *:* LISTEN 1118/munin-node tcp 0 0 *:ssh *:* LISTEN 131/sshd tcp 0 0 localhost.localdo:10024 localhost.localdo:36009 TIME_WAIT - tcp 0 0 localhost.localdo:39708 localhost.localdoma:ssh TIME_WAIT - tcp 0 0 localhost.localdo:59088 localhost.localdoma:953 TIME_WAIT - tcp 0 0 localhost.lo:postgresql localhost.localdo:51791 ESTABLISHED 14664/postgres: buc tcp 0 0 localhost.localdoma:www localhost.localdo:46782 TIME_WAIT - tcp 0 0 localhost.localdo:49808 localhost.localdo:pop3s TIME_WAIT - tcp 0 0 localhost.lo:postgresql localhost.localdo:51771 ESTABLISHED 14634/postgres: buc tcp 0 0 localhost.lo:postgresql localhost.localdo:51793 ESTABLISHED 14665/postgres: adm tcp 0 0 localhost.localdo:51789 localhost.lo:postgresql ESTABLISHED 14661/Bucardo Contr tcp 0 0 localhost.localdo:51771 localhost.lo:postgresql ESTABLISHED 14633/Bucardo Maste tcp 0 0 localhost.localdom:smtp localhost.localdo:45235 TIME_WAIT - tcp 0 0 localhost.localdo:48647 localhost.lo:postgresql TIME_WAIT - tcp 0 0 localhost.localdo:mysql localhost.localdo:48809 TIME_WAIT - tcp 0 0 localhost.localdo:51793 localhost.lo:postgresql ESTABLISHED 14663/Bucardo Kid. tcp 0 0 localhost.lo:postgresql localhost.localdo:51789 ESTABLISHED 14662/postgres: buc tcp 0 0 localhost.localdo:51791 localhost.lo:postgresql ESTABLISHED 14663/Bucardo Kid. tcp 0 0 localhost.localdo:48648 localhost.lo:postgresql TIME_WAIT - tcp 0 0 localhost.localdo:51774 localhost.lo:postgresql ESTABLISHED 14633/Bucardo Maste tcp 0 0 localhost.lo:postgresql localhost.localdo:51774 ESTABLISHED 14649/postgres: adm tcp6 0 0 [::]:postgresql [::]:* LISTEN 363/postgres tcp6 0 0 ::1%137029864:953 [::]:* LISTEN 9828/named tcp6 0 0 [::]:imaps [::]:* LISTEN 9733/couriertcpd tcp6 0 0 [::]:pop3s [::]:* LISTEN 9761/couriertcpd tcp6 0 0 [::]:pop3 [::]:* LISTEN 9744/couriertcpd tcp6 0 0 [::]:imap2 [::]:* LISTEN 9716/couriertcpd tcp6 0 0 [::]:domain [::]:* LISTEN 9828/named tcp6 0 0 [::]:ftp [::]:* LISTEN 9809/pure-ftpd (SER tcp6 0 0 [::]:ssh [::]:* LISTEN 131/sshd
every time that run server.sh (or server.php), the system try to update; How do i change "the flag" to cancel this update?. Thx.
Hello Falko, Back to my original problem, I don't have any errors running server.sh after manually deleting the lock file: 22.11.2010-11:24 - DEBUG - Set Lock: /usr/local/ispconfig/server/temp/.ispconfig_lock 22.11.2010-11:24 - DEBUG - No Updated records found, starting only the core. 22.11.2010-11:24 - DEBUG - Remove Lock: /usr/local/ispconfig/server/temp/.ispconfig_lock finished. It seems that after a while a lock is getting stuck in there and not getting deleted. Is there any way we can debug the cause of this? Thanks David
Trying to find the cause, it seems to happen when there's nothing in the queue for some random reason. web-01:/usr/local/ispconfig/server/temp# /usr/local/ispconfig/server/server.sh 23.11.2010-15:09 - DEBUG - Set Lock: /usr/local/ispconfig/server/temp/.ispconfig_lock 23.11.2010-15:09 - DEBUG - No Updated records found, starting only the core. 23.11.2010-15:09 - DEBUG - Remove Lock: /usr/local/ispconfig/server/temp/.ispconfig_lock finished. web-01:/usr/local/ispconfig/server/temp# /usr/local/ispconfig/server/server.sh 23.11.2010-15:09 - DEBUG - Set Lock: /usr/local/ispconfig/server/temp/.ispconfig_lock 23.11.2010-15:09 - DEBUG - No Updated records found, starting only the core. 23.11.2010-15:09 - DEBUG - Remove Lock: /usr/local/ispconfig/server/temp/.ispconfig_lock finished. web-01:/usr/local/ispconfig/server/temp# /usr/local/ispconfig/server/server.sh 23.11.2010-15:09 - DEBUG - Set Lock: /usr/local/ispconfig/server/temp/.ispconfig_lock 23.11.2010-15:09 - DEBUG - No Updated records found, starting only the core. 23.11.2010-15:09 - DEBUG - Remove Lock: /usr/local/ispconfig/server/temp/.ispconfig_lock finished. web-01:/usr/local/ispconfig/server/temp# /usr/local/ispconfig/server/server.sh 23.11.2010-15:09 - DEBUG - Set Lock: /usr/local/ispconfig/server/temp/.ispconfig_lock 23.11.2010-15:09 - DEBUG - No Updated records found, starting only the core. 23.11.2010-15:09 - DEBUG - Remove Lock: /usr/local/ispconfig/server/temp/.ispconfig_lock finished. web-01:/usr/local/ispconfig/server/temp# /usr/local/ispconfig/server/server.sh 23.11.2010-15:09 - DEBUG - Set Lock: /usr/local/ispconfig/server/temp/.ispconfig_lock 23.11.2010-15:09 - DEBUG - No Updated records found, starting only the core. 23.11.2010-15:09 - DEBUG - Remove Lock: /usr/local/ispconfig/server/temp/.ispconfig_lock finished. web-01:/usr/local/ispconfig/server/temp# /usr/local/ispconfig/server/server.sh 23.11.2010-15:10 - DEBUG - Set Lock: /usr/local/ispconfig/server/temp/.ispconfig_lock 23.11.2010-15:10 - DEBUG - No Updated records found, starting only the core. web-01:/usr/local/ispconfig/server/temp# Why does the script not remove the lock despite nothing being in the queue and no errors being shown? Is there a way to debug this further, appreciate your advice.
check with phpmyadmin if there are any records in the table sys_remoteaction and delete them manually with phpmyadmin.
Thanks Till, I found the a record for 'ispc_update' with a status of 'pending' in this table so have deleted. Will report back if any further issues.
Unfortunately the problem still seems to be occuring. Although it does look like jobs are eventually being put through, they seem to hang in the queue for a long time whilst the error messages regarding the lockfile persist.
We have also noticed that this server is not being monitored correctly, i.e. there are no stats for the RAID State, Load, etc, etc. Any ideas?
After a lot of digging and debugging of the scripts, it seems that the @ftp_connect($host, $port) line in /usr/local/ispconfig/server/mods-core/monitor_core_module.inc.php is crashing out the script but not throwing a PHP error (because of the @), in the case that the PHP-FTP module is not enabled. Can I highly recommend that a requirements page is put up on the ISPConfig site, specifically with reference to the PHP modules required by the software?
The perfect setup guide is that reference. Everything that is installed there is required by ispconfig.
Thanks Till, That's a fair point, but some of us don't want to follow the guides in their entirity as we may have custom requirements (for example we are building our own PHP rather than using binaries). Although we've based our install on the Perfect Install Guides, when you look at the PHP section, there isn't explicity mention of the php ftp module so it's hard to know to install this in the first place: apt-get install apache2 apache2.2-common apache2-doc apache2-mpm-prefork apache2-utils libexpat1 ssl-cert libapache2-mod-php5 php5 php5-common php5-gd php5-mysql php5-imap phpmyadmin php5-cli php5-cgi libapache2-mod-fcgid apache2-suexec php-pear php-auth php5-mcrypt mcrypt php5-imagick imagemagick libapache2-mod-suphp libapache2-mod-ruby Maybe a community Wiki might be a good idea too so we (the community) can help build up the documentation for you guys?
We've tried that so many times already, but hardly anyone is willing or has the time to contribute...
Thanks Falko, Perhaps i'll set one up in the future, if I have time By the way, we now have Litespeed working perfectly alongside ISPConfig 3! A perfect combination
