Jobqueue randomly not processing

ISPconfig 3.0.3.2 on Ubuntu 10.10 64 on a vps.

Spent the weekend on a wild goose chase trying to track why changes made in the control panel weren't working.

Finally found today that some jobs happen after a minute, some take 5, 10, 20 minutes.

So, I manually ran

/usr/local/ispconfig/server/server.sh

Nothing.

Dug around inside and ran

/usr/bin/php -q /usr/local/ispconfig/server/server.php

and it updated the job queue. Made another change to test.
This time /usr/local/ispconfig/server/server.sh worked.

Next 10 times, both files, despite 2 jobs in the queue, nothing worked. 11th time, it worked.

This is all getting a bit confusing. I've seen mention of turning on debugging in /usr/local/ispconfig/server/lib/config.inc.php but I don't actually see what line that is.

Thank you.

 

Ispconfig 3.0.32, Ubuntu 10.10 64 bit vps, 512Mb RAM

I had to give up with cron entirely when something happened so that when I made a change, when cron kicked in the next minute, I could watch the update process climb up the "top" table until it was using all the memory, and the server ground to a halt.

Commenting out the line

Code:

* * * * * /usr/local/ispconfig/server/server.sh > /dev/null 2>> /var/log/ispconfig/cron.log

in crontab allowed my server to live, and I was happily using

sh /usr/local/ispconfig/server/server.sh

to get things moving. Yes, I might have to do it 10-15 times in a row, but at least I knew it would happen.

Now all I get is:

Code:

08.03.2011-15:35 - DEBUG - Set Lock: /usr/local/ispconfig/server/temp/.ispconfig_lock
08.03.2011-15:35 - DEBUG - Found 3 changes, starting update process.
postqueue: warning: Mail system is down -- accessing queue directly

(It's always said that last line, as I don't have any mail services running. Hasn't been a problem before).

In cron.log, the last few lines are:

Code:

PHP Notice:  Undefined offset: 1 in /usr/local/ispconfig/server/cron_daily.php on line 95
Warning: Truncating oversized referrer field
Warning: Truncating oversized referrer field
PHP Warning:  Invalid argument supplied for foreach() in /usr/local/ispconfig/server/cron_daily.php on line 216
PHP Warning:  Invalid argument supplied for foreach() in /usr/local/ispconfig/server/cron_daily.php on line 330

However, this isn't much help as it doesn't have a timestamp.

All that's in the queue are a couple of site changes and a new mysql user and database.

Code:

2011-03-08 11:31	mydomain.com	Insert	web_database	
2011-03-05 10:15	mydomain.com	Update	web_domain	
2011-03-05 10:12	mydomain.com	Update	web_domain

How can I see what the commands are and what might be blocking it?

 

Just one:

Code:

action_id	server_id	tstamp	action_type	action_param	action_state	response
4	        1	1295540461	os_update	 	ok	 

 

Yes! That cleared it, thank you, and I could manually run those jobs.

But then I tried activating cron again, made a simple change (turned ssl off), and watched the jobqueue. It never clears. In the processes I see:

Code:

root      5620  5618  0 17:55 ?        00:00:00 /bin/sh -c /usr/local/ispconfig/server/server.sh > /dev/null 2>> /var/log/ispconfig/cron.log
root      5621  5620  0 17:55 ?        00:00:00 /bin/sh /usr/local/ispconfig/server/server.sh
root      5622  5621  0 17:55 ?        00:00:00 /usr/bin/php -q /usr/local/ispconfig/server/server.php

and in the cron.log I see:

Code:

2011-03-08 18:00	mydomain.com	Debug	There is already an instance of server.php running. Exiting.	
2011-03-08 17:59	mydomain.com	Debug	There is already an instance of server.php running. Exiting.	
2011-03-08 17:58	mydomain.com	Debug	There is already an instance of server.php running. Exiting.	
2011-03-08 17:57	mydomain.com	Debug	There is already an instance of server.php running. Exiting.	
2011-03-08 17:56	mydomain.com	Debug	There is already an instance of server.php running. Exiting.	
2011-03-08 17:55	mydomain.com	Debug	Found 1 changes, starting update process.	
2011-03-08 17:55	mydomain.com	Debug	Set Lock: /usr/local/ispconfig/server/temp/.ispconfig_lock	
2011-03-08 17:54	mydomain.com	Debug	Remove Lock: /usr/local/ispconfig/server/temp/.ispconfig_lock	
2011-03-08 17:54	mydomain.com	Debug	No Updated records found, starting only the core.	

so once again, the only way to get the jon to run would be kill the server.php/sh processes, comment out the line in cron, delete the lock file, and manually run /bin/sh /usr/local/ispconfig/server/server.sh

Not ideal. Is there anything else I can do to fix this?

 

I've been asked this before, and here's the problem: I have no access to the "host" physical machine. Therefore, as an "instance" of a vps, /proc/user_beancounters doesn't actually exist.

Still, I am always only using 50% RAM.

 

OK, but I have turned off EVERYTHING to do with mail, spam, smtp, imap and all the bits that come with it. MySQL is tweaked to limit memory usage, so I can't understand how this server.sh uses over 200Mb of my free RAM plus another 512Mb of swap, just to, for example, configure a vhost to be non-ssl and restart apache. And when I do it manually and it works OK, the whole process barely takes 3-4 seconds and the memory bar hardly flutters at all.

I can run a complex large sound encode using lame on an 80Mb file, but ispconfig needs more than this just to run a shell script? Seems odd.