High Disk IO

Since the latest update of ISPConfig (not sure if that's connected), I've been finding that at about 6:40 each morning, the server starts using 100% CPU and ~100% Disk IO - this is reported through New Relic, which runs an agent on the server.

Now I know a little about servers, but not a great deal...

Attached are a couple of screenshots of top, htop, iostat and iotop. These screenshots are after a reboot - notice how it doesn't show apache or mysql, as the server didn't start the processes for some reason (maybe too busy?). Suppose this is a good thing, as it rules out an issue with apache and mysql etc.

One thing that does stand out, is 'quotacheck -augm' using 99.99% IO when viewed in iotop. I'm not sure if this is normal or not though.

My first thought was maybe the HDD is going bad, but this is a VPS so I would hope that my provider would catch that if it was an issue, however I won't rule it out.

I would really appreciate if someone was able to offer some advise on pinpointing exactly what is going on, and how to sort it out.

Many thanks in advance!

 

I had this same issue on my Amazon hosted multi server setup. I disabled "Mailbox quota statistics" for each server and my IO dropped big time. Hope this helps

 

I found that it was updatedb.mlocate that was causing the high disk IO usage. I've disabled it, but it is slightly annoying, as I use locate quite often.

 

Ubuntu uses ionice to only use idle IO , I would check to see if that is an option for you. Locate is a very useful command


if [ -x /usr/bin/ionice ] &&
/usr/bin/ionice -c3 true 2>/dev/null; then
IONICE="/usr/bin/ionice -c3"
fi

$IONICE /usr/bin/updatedb.mlocate

 

Yeah, that's in the /etc/cron.daily/mlocate file;

Code:

#! /bin/bash

set -e

[ -x /usr/bin/updatedb.mlocate ] || exit 0

if which on_ac_power >/dev/null 2>&1; then
    ON_BATTERY=0
    on_ac_power >/dev/null 2>&1 || ON_BATTERY=$?
    if [ "$ON_BATTERY" -eq 1 ]; then
        exit 0
    fi
fi

##

LOCKFILE="/var/lib/mlocate/daily.lock"

trap "rm -f $LOCKFILE" EXIT

if [ -e "$LOCKFILE" ]; then
    echo >&2 "Warning: $LOCKFILE present, not running updatedb."
    exit 1
else
    touch "$LOCKFILE"
fi

##

# See ionice(1)
if [ -x /usr/bin/ionice ] &&
    /usr/bin/ionice -c3 true 2>/dev/null; then
    IONICE="/usr/bin/ionice -c3"
fi

$IONICE /usr/bin/updatedb.mlocate

 

Is ionice installed ?

 

I believe so;

Code:

root@vps:~# ionice
none: prio 0