Hi all I have been using isp3 for about 2+ years now using ubuntu 14.04 x64. (and isp2 before that) We aren't an internet or hosting company, we use it to manage internal websites and email systems. Last week I decided to push for an overhaul of all the servers and connect isp3 into a cluster so we can manage all the servers from 1 server. I installed Ubuntu 15.10 x64 and latest ISP3 build p9. Problem #1 - cpu usage spikes I noticed that every couple of minutes the server would spike in usage. This isn't something that we observed in prior installations. but of course, this new setup has a new ubuntu build AND a new ISP3 build so I am not sure if this might have something to do with it. (it also has mysql 5.6 instead) While the spike isn't much, 10-25% usage for 2-3 minutes, this might be a concern if it happens during peak usage hours. Problem #2 - higher than average server loads Another problem. The old mysql server was an 8gig ram server 4 core just like the new one. they are identical in hardware spec. only difference is 14.04 vs 15.10 and p8 vs p9. both server running mysql 5.6 with SAME my.cnf however, after migrating to new server i have observed higher server loads than the old server. by a wide margin. seems to be 20-40% more cpu usage. high enough at one point to cause unrecoverable lag/bottleneck. Once again, wondering if p9 has some cron job that triggers this kind of behavior or is 15.10 not very compatible?
1) Can you say at which exact time the spike happens each hour, but they normally dont run for minutes and dont cause high load. There are several monitoring jobs that run at different times, so the exact time matters to find the job. Do you use any network filesystems like nfs in your new setup? You might also want to check with top command which exact process is consuming the cpu cycles. 2) The difference between ispconfg p8 and p9 is a check of one variable in the web form, nothing in cronjobs changed, so the different ispconfig version is not related to your issue and there is no issue in Ubuntu 15.10 regarding ispconfig. ISPConfig is also not able to produce a constant higher server load as it is just a cronjob that runs once a minute for a few seconds. So your issue must be related to another difference in your setup, you should check with top which exact processes are causing the load.
Hi till Responding to part 1. I took a closer look at the HD spikes, they are all right around hour, every hour. each spike lasts 2-3 minutes. The spike on CPU load is more frequent, seems to be every 10 minutes or so lasting for 2-3 minutes. Please note that the screen capture is from the isp3 cluster controller and there are no active website or db traffic. this server is running 2gigs ram, 15.10x64 and 9p. its sole purpose for now is to manage other isp3 setups. there are a total of 4 servers including this one being managed. I used this server as a benchmark because its not being used so any spikes shouldnt be caused by websites or db being ran. I just started atop and will be collecting data every minute hoping to capture which process is causing the spike. The server is a virtualized server from digital ocean. i am guessing the entire OS is a virtualized but contiguous file on a SSD.
Good that you mention that these are virtual machines as thats the explanation for your issues. You should run a harddisk speed test and compare your new servers with the old servers and a cpu benchmark. When you want to break down speed issues to a software installation, then you can only compare the speed of 2 servers when they are physical identical systems. You cant do that with virtual machines on systems where you do not control the hardware to ensure that the system is not running any other virtual machine at the same time that shares the ressources with your VM. On virtual servers even a very small cpu usage that you wont even notice on a normal server can cause a huge spike when there are other vm's on the same hardware that use cpu ressources as you don't know which other vms are running at which load on the same HW node at this time. And the disk usage that you show above is the save process of a file with 2-3 megabytes, thats something that happens on servers all the time and real servers have harddisk bandwidths of 50 - 200MB/sec, so nothing that you would ever notice in the statistics of a root server. The bandwodth is the transfer of a few hunder kilobytes between the nodes to sync them, so nothing notable as well. Not more then a few page requests on a normal wordpress blog or similar website.
Problem identified to be Munin. taking snap shots every 5 minutes. I have not found a way to resolve the issue yet but for future reference atop works like a charm in capturing data. FYI I posted the specs of identical server setup, 1 running 15.10 and another running 14.04 both at 2gig ram It appears that 15.10 performance seems to good to be true. Any thoughts? Going to forward the results to digital ocean for them to take a look as well.
The software installation might be the same but the servers are not. The setups would be identical if you run them on the same physical hardware without virtualisation. As far as I can see, they are virtualized (Digitalocean droplets) and therefor they are not identical as you can not ensure that the exact same other virtual machines of other customers with the exact same load are running on the same hardware at a given time. Ecah other customer that shares the hardware on this node with you takes a part of the system ressources, so the speed can vary alot and you could have the same results with a exact copy of your 14.04 server. If you want to compare 2 setups software wise, then you must use the same dedicated hardware, otherwise you can not know if a difference is caused by hard or software like in your case with digitalocean droplets.
