Hello, I have been running my website on a dedicated server for around 3 years now with Fasthosts. Everything had been running fine until recently and I believe that the server is now being attacked. I have checked my Apache error_log, it has a huge list of errors which are mostly repeated, but when searching them in Google to find out what they mean and how to fix them all I have found is people saying they are minor issues which makes me believe they are not the reason for the server going down. I do know that it seems to go offline mainly between 3 - 12PM GMT meaning if I restart the server on a morning when I wake up, it can be online for several hours before going offline again, but once it does go offline I can restart it several times and it will just keep going back offline each time. So could someone please help me find out what is going wrong and how to rectify it? I would also like to be able to get it so that if the server itself or any of the services on it could start themselves back up if they where ever to go offline in the future aswell if possible. I have tried installing something called SIM but that doesn't seem to be working. Thanks.
check the logs to see what actually happens look at /var/log/messages as well as other logs not just the apache log
Hi thanks for your reply, I have checked through /var/log/messages and quite far down the log it has the list of errors below which caught my attention, I didn't really understand any of the messages which I seen within the log but these seem as though something is going wrong: Code: Dec 13 21:14:59 localhost avahi-daemon[2623]: Network interface enumeration completed. Dec 13 21:14:59 localhost avahi-daemon[2623]: Registering new address record for fe80::230:5ff:fee5:2a90 on eth0. Dec 13 21:14:59 localhost avahi-daemon[2623]: Registering new address record for 88.208.230.130 on eth0. Dec 13 21:15:00 localhost avahi-daemon[2623]: Registering new address record for 88.208.230.131 on eth0. Dec 13 21:15:00 localhost avahi-daemon[2623]: Registering HINFO record with values 'I686'/'LINUX'. Dec 13 21:15:00 localhost avahi-daemon[2623]: Withdrawing address record for 88.208.230.130 on eth0. Dec 13 21:15:00 localhost avahi-daemon[2623]: Withdrawing address record for 88.208.230.131 on eth0. Dec 13 21:15:00 localhost avahi-daemon[2623]: Host name conflict, retrying with <localhost-2> Dec 13 21:15:00 localhost avahi-daemon[2623]: Registering new address record for fe80::230:5ff:fee5:2a90 on eth0. Dec 13 21:15:00 localhost avahi-daemon[2623]: Registering new address record for 88.208.230.130 on eth0. Dec 13 21:15:00 localhost avahi-daemon[2623]: Registering new address record for 88.208.230.131 on eth0. Dec 13 21:15:00 localhost avahi-daemon[2623]: Registering HINFO record with values 'I686'/'LINUX'. Dec 13 21:15:00 localhost avahi-daemon[2623]: Withdrawing address record for 88.208.230.130 on eth0. Dec 13 21:15:00 localhost avahi-daemon[2623]: Withdrawing address record for 88.208.230.131 on eth0. Dec 13 21:15:01 localhost avahi-daemon[2623]: Host name conflict, retrying with <localhost-3> Dec 13 21:15:01 localhost avahi-daemon[2623]: Registering new address record for fe80::230:5ff:fee5:2a90 on eth0. Dec 13 21:15:01 localhost avahi-daemon[2623]: Registering new address record for 88.208.230.130 on eth0. Dec 13 21:15:01 localhost avahi-daemon[2623]: Registering new address record for 88.208.230.131 on eth0. When checking the contents of /var/log/mysqld.log I seem to get these messages repeating over and over again: Code: 091215 02:23:02 mysqld started 091215 2:23:03 [Warning] option 'max_connections': unsigned value 20000 adjusted to 16384 091215 2:23:03 InnoDB: Started; log sequence number 0 377946 091215 2:23:03 [Note] /usr/libexec/mysqld: ready for connections. Version: '5.0.86' socket: '/var/lib/mysql/mysql.sock' port: 3306 Source distribution 091215 16:31:35 mysqld started 091215 16:31:36 [Warning] option 'max_connections': unsigned value 20000 adjusted to 16384 InnoDB: The log sequence number in ibdata files does not match InnoDB: the log sequence number in the ib_logfiles! 091215 16:31:36 InnoDB: Database was not shut down normally! InnoDB: Starting crash recovery. InnoDB: Reading tablespace information from the .ibd files... InnoDB: Restoring possible half-written data pages from the doublewrite InnoDB: buffer... 091215 16:31:37 InnoDB: Started; log sequence number 0 380788 091215 16:31:37 [Note] /usr/libexec/mysqld: ready for connections. Version: '5.0.86' socket: '/var/lib/mysql/mysql.sock' port: 3306 Source distribution 091215 16:39:02 mysqld started 091215 16:39:02 [Warning] option 'max_connections': unsigned value 20000 adjusted to 16384 InnoDB: The log sequence number in ibdata files does not match InnoDB: the log sequence number in the ib_logfiles! 091215 16:39:02 InnoDB: Database was not shut down normally! InnoDB: Starting crash recovery. Obviously something is happening with that as it states that the database was not shut down properly and that is had crashed, is that enough to take the entire server offline or just a minor issue? And here at the main error messages from /var/log/httpd/error_log: This one seems to repeat in big blocks, not sure what exactly it means but my public files are stored within /user/htdocs not /var/www/html/ do I need to change something to remove that error? Code: [Sun Dec 13 21:04:30 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/ I also seem to get this error repeated quite a lot aswell: Code: [Tue Dec 15 00:57:25 2009] [notice] child pid 3488 exit signal Segmentation fault (11) zend_mm_heap corrupted This one appears once that I have noticed: Code: [Tue Dec 15 02:23:10 2009] [notice] Graceful restart requested, doing restart Then this seems to be a typical block of code which gets repeated over and over hundreds of times per day: Code: [Tue Dec 15 16:31:51 2009] [notice] mod_python: using mutex_directory /tmp [Tue Dec 15 16:31:52 2009] [notice] Apache/2.2.3 (FH) configured -- resuming normal operations [Tue Dec 15 16:32:43 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/ [Tue Dec 15 16:32:44 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/ [Tue Dec 15 16:32:47 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/ [Tue Dec 15 16:39:10 2009] [notice] suEXEC mechanism enabled (wrapper: /usr/sbin/suexec) [Tue Dec 15 16:39:11 2009] [notice] Digest: generating secret for digest authentication ... [Tue Dec 15 16:39:11 2009] [notice] Digest: done [Tue Dec 15 16:39:12 2009] [notice] mod_python: Creating 4 session mutexes based on 3000 max processes and 0 max threads. Are those the right logs to check and have I supplied enough useful information? Not sure what other logs their are to check. Thanks for your time and help
turn of the avahi-daemon you should not be running that on a server, use static configuration for your network interfaces.
Thanks again for the reply, I'm a complete newbie when it comes to servers so could you please tell me what I would need to do to configure my network interfaces? My installation at the moment is basically an "out of the box" package from Fasthosts.co.uk and their system automatically installed and setup my CentOS. Thanks
Unfortunately that is well beyond the scope of what a forum post can provide, i would advise that you read up on the documentation. http://www.centos.org/docs/5/html/5.2/Deployment_Guide/pt-network-related-config.html
Sorry i did not actually see this Code: [Tue Dec 15 00:57:25 2009] [notice] child pid 3488 exit signal Segmentation fault (11) zend_mm_heap corrupted Something is crushing your php/apache stack u need to investigate what it is.
Thanks for all your help, after quickly checking my hosts file and a few other files mentioned on the first page of that configuration documentation I went ahead and disabled avahi-daemon anyway and then also stopped the service aswell and everything still seems to be working fine. Hopefully that will also stop the server from going offline, but if not I will post an update within this topic. Thanks again
Woops, did not see this post. I have just quickly done a search for that error and came across this, would you recommend trying their idea as a solution? http://ubuntuforums.org/archive/index.php/t-18490.html
are u by any chance using the apc php module ? as there seems to be a bug similer to what you are experiencing. http://pecl.php.net/bugs/bug.php?id=13511
Try that and see if you actually have the python module installed. Code: yum remove mod_python service httpd restart
Yes it is installed, after typing yum remove mod_python it displays some data with the name, status and size. Should I go ahead with the uninstall? I'm not sure, how can I check?
On second thoughts i do not really think you problem is the same though because your crush is happening inside the php module Code: zend_mm_heap corrupted if you are not running any python code in apache then uninstall it. You can check the installed php modules using Code: php -m
I have uninstalled python as I was not using it. I don't seem to have the APC module installed, but just as a bit more information here is a list of the modules which where returned: bz2 calendar ctype curl date dbase exif filter ftp gd gettext gmp hash iconv ionCube Loader json ldap libxml mysql mysqli openssl pcntl pcre PDO pdo_mysql pdo_sqlite posix readline Reflection session shmop SimpleXML sockets SPL standard sysvmsg sysvsem sysvshm tokenizer wddx xml zip zlib
I'll try removing that aswell then, and just reinstall it if something I need stops working. I think I installed it so that Cast Control would run on my website which I no longer use.
Hmm I can't seem to find anything online explaining how to uninstall ionCube Loader, just found a few people mentioning that having both ionCube & Zend installed takes their servers offline.
I have managed to stop the service from starting by commenting out the start up lines for it in php.ini. The server has just died on me again there.
