Maria OOM-KILL

Discussion in 'General' started by SamTzu, Oct 25, 2022.

  1. SamTzu

    SamTzu Active Member

    Brand new ISPConfig server has started to fail because Maria/MySQL wont restart after oom-kill.
    I can't pinpoint what actually happens but the LXC container does not run out of memory so...
    What kills Maria? I'm thinking it's Apache.
    Why can't ISPconfig restart MariaDB after oom?
     
  2. ahrasis

    ahrasis Well-Known Member

    Your statements are conflicting. You have to determine whether it was killed because your server is out of memory or not.

    So far that I am concern, it is not ISPConfig job to check and restart your services if they were stopped for whatever reasons. Due to that I personally set monit to monitor such thing and restart any services if they are stopped, so I would suggest the same.
     
  3. SamTzu

    SamTzu Active Member

    You would think that :)
    Normally ISPconfig restarts apache and mysql just fine if they crash. For some reason oom-kill prevents service restart.

    Code:
    systemctl status mysqld
    * mariadb.service - MariaDB 10.5.15 database server
         Loaded: loaded (/lib/systemd/system/mariadb.service; enabled; vendor preset: enabled)
        Drop-In: /etc/systemd/system/mysql.service.d
                 `-limits.conf
         Active: failed (Result: oom-kill) since Tue 2022-10-25 16:35:37 UTC; 1h 39min ago
    
     
  4. ahrasis

    ahrasis Well-Known Member

    Well, not really to my experience.

    Anyway, your server memory is of your main concern now and why it has ran out of memory killing your mariadb.

    You still don't believe in having swap for your ISPConfig in LXC? I remember you calculated it as a waste. ;)
     
  5. pyte

    pyte Active Member HowtoForge Supporter

    Can you provide the output of
    Code:
    dmesg -T | egrep -i 'killed process'
    so we can make sure that the process got oom killed?
    You can set the overcommit memory and ratio within sysctl to prevent application crashes from happening. However as @ahrasis already mentioned, using swap isn't a bad idea.
     
  6. SamTzu

    SamTzu Active Member

    Code:
    dmesg -T | egrep -i 'killed process'
    dmesg: read kernel buffer failed: Operation not permitted
     
  7. SamTzu

    SamTzu Active Member

    I raised the VM RAM to 4Gb from 2Gb (even though it rarely topped 1Gb in use.)
    Now it seems stable.
    It's odd that MariaDB would require so much "extra" memory that we can't see.
     
  8. SamTzu

    SamTzu Active Member

    And then there is the whole OOMps I did it again. And failed to get up afterwards.
     
  9. pyte

    pyte Active Member HowtoForge Supporter

    please run the command with super user rights, and provide the Output
    Code:
    sudo dmesg -T | egrep -i 'killed process'
     
  10. Th0m

    Th0m ISPConfig Developer Staff Member ISPConfig Developer

    It restarts it if it crashes due to a config change. It does not monitor those processes and restart them afaik.

    Best practice is to have a daemon monitoring your services, so if they crash, they are restarted. I recently wrote a script for this that will be incorporated in the autoinstaller as well: https://forum.howtoforge.com/thread...r-debian-10-11-and-ubuntu-20-04-server.89482/

    What is the output of
    Code:
    free -m
    ?
     
    ahrasis likes this.
  11. pyte

    pyte Active Member HowtoForge Supporter

    Well the service should do this itself if it crashes. I've checked the systemd unit file for mariadb service on a Debian 11 install with ISPConfig

    Code:
    Restart=on-abort
    RestartSec=5s
    This means the mariadb server would restart on crash. Note: on-failure is not set here, if that would be the case the server would restart even with config errprs in my.cnf
     
    Last edited: Oct 27, 2022
    ahrasis likes this.

Share This Page