Maria OOM-KILL

Brand new ISPConfig server has started to fail because Maria/MySQL wont restart after oom-kill.
I can't pinpoint what actually happens but the LXC container does not run out of memory so...
What kills Maria? I'm thinking it's Apache.
Why can't ISPconfig restart MariaDB after oom?

 

Your statements are conflicting. You have to determine whether it was killed because your server is out of memory or not.

So far that I am concern, it is not ISPConfig job to check and restart your services if they were stopped for whatever reasons. Due to that I personally set monit to monitor such thing and restart any services if they are stopped, so I would suggest the same.

 

You would think that
Normally ISPconfig restarts apache and mysql just fine if they crash. For some reason oom-kill prevents service restart.

Code:

systemctl status mysqld
* mariadb.service - MariaDB 10.5.15 database server
     Loaded: loaded (/lib/systemd/system/mariadb.service; enabled; vendor preset: enabled)
    Drop-In: /etc/systemd/system/mysql.service.d
             `-limits.conf
     Active: failed (Result: oom-kill) since Tue 2022-10-25 16:35:37 UTC; 1h 39min ago

 

Well, not really to my experience.

Anyway, your server memory is of your main concern now and why it has ran out of memory killing your mariadb.

You still don't believe in having swap for your ISPConfig in LXC? I remember you calculated it as a waste.

 

Can you provide the output of

Code:

dmesg -T | egrep -i 'killed process'

so we can make sure that the process got oom killed?
You can set the overcommit memory and ratio within sysctl to prevent application crashes from happening. However as @ahrasis already mentioned, using swap isn't a bad idea.

 

Code:

dmesg -T | egrep -i 'killed process'
dmesg: read kernel buffer failed: Operation not permitted

 

I raised the VM RAM to 4Gb from 2Gb (even though it rarely topped 1Gb in use.)
Now it seems stable.
It's odd that MariaDB would require so much "extra" memory that we can't see.

 

And then there is the whole OOMps I did it again. And failed to get up afterwards.

 

please run the command with super user rights, and provide the Output

Code:

sudo dmesg -T | egrep -i 'killed process'

 

Well the service should do this itself if it crashes. I've checked the systemd unit file for mariadb service on a Debian 11 install with ISPConfig

Code:

Restart=on-abort
RestartSec=5s

This means the mariadb server would restart on crash. Note: on-failure is not set here, if that would be the case the server would restart even with config errprs in my.cnf