MySQL Down

Discussion in 'Installation/Configuration' started by jbonlinea, May 29, 2019.

  1. jbonlinea

    jbonlinea Member

    Hi guys,

    I'l using the stretch perfect server and so fat so good.

    Since a couple of days, mysql is falling !
    I've checked in the error.log which gives me some clues of what can I do but I would rather understand what am I doing rather than just doing it.

    So if one of you can have a look at the log below, and explain a little bit why innodb is switching to legacy mode, why now and not before, and what should be done and why this and not something else that would be awesome and taughtfull

    Thank's in advance

    Code:
    2019-05-28  9:51:58 140429806931328 [Note] InnoDB: innodb_empty_free_list_algorithm has been changed to legacy because of small buffer pool size. In order to use backoff, increase buffer pool at least up to 20MB.
    
    2019-05-28  9:51:59 140429806931328 [Note] InnoDB: Using mutexes to ref count buffer pool pages
    2019-05-28  9:51:59 140429806931328 [Note] InnoDB: The InnoDB memory heap is disabled
    2019-05-28  9:51:59 140429806931328 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
    2019-05-28  9:51:59 140429806931328 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier
    2019-05-28  9:51:59 140429806931328 [Note] InnoDB: Compressed tables use zlib 1.2.8
    2019-05-28  9:51:59 140429806931328 [Note] InnoDB: Using Linux native AIO
    2019-05-28  9:51:59 140429806931328 [Note] InnoDB: Using SSE crc32 instructions
    2019-05-28  9:51:59 140429806931328 [Note] InnoDB: Initializing buffer pool, size = 128.0M
    2019-05-28  9:51:59 140429806931328 [Note] InnoDB: Completed initialization of buffer pool
    2019-05-28  9:51:59 140429806931328 [Note] InnoDB: Highest supported file format is Barracuda.
    2019-05-28  9:51:59 140429806931328 [Note] InnoDB: The log sequence number 1541147180 in ibdata file do not match the log sequence number 1993914455 in the ib_logfiles!
    2019-05-28  9:52:00 140429806931328 [Note] InnoDB: Restoring possible half-written data pages from the doublewrite buffer...
    2019-05-28  9:52:01 140429806931328 [Note] InnoDB: 128 rollback segment(s) are active.
    2019-05-28  9:52:01 140429806931328 [Note] InnoDB: Waiting for purge to start
    2019-05-28  9:52:01 140429806931328 [Note] InnoDB:  Percona XtraDB (http://www.percona.com) 5.6.42-84.2 started; log sequence number 1993914455
    2019-05-28  9:52:01 140429186000640 [Note] InnoDB: Dumping buffer pool(s) not yet started
    2019-05-28  9:52:01 140429806931328 [Note] Plugin 'FEEDBACK' is disabled.
    2019-05-28  9:52:01 140429806931328 [Note] Recovering after a crash using tc.log
    2019-05-28  9:52:01 140429806931328 [Note] Starting crash recovery...
    2019-05-28  9:52:01 140429806931328 [Note] Crash recovery finished.
    2019-05-28  9:52:01 140429806931328 [Note] Server socket created on IP: '::'.
    2019-05-28  9:52:01 140429806931328 [Note] /usr/sbin/mysqld: ready for connections.
    Version: '10.1.38-MariaDB-0+deb9u1'  socket: '/var/run/mysqld/mysqld.sock'  port: 3306  Debian 9.8
    2019-05-28  9:52:02 140429805909760 [Warning] IP address '88.214.26.17' could not be resolved: Temporary failure in name resolution
    2019-05-28  9:53:02 140429762467584 [ERROR] mysqld: Table './dbispconfig/sys_cron' is marked as crashed and should be repaired
    2019-05-28  9:53:02 140429762467584 [Warning] Checking table:   './dbispconfig/sys_cron'
    2019-05-28  9:53:02 140429762467584 [ERROR] mysqld: Table './dbispconfig/monitor_data' is marked as crashed and should be repaired
    2019-05-28  9:53:02 140429762467584 [Warning] Checking table:   './dbispconfig/monitor_data'
    2019-05-28  9:56:25 140429762467584 [ERROR] mysqld: Table './dbispconfig/ftp_traffic' is marked as crashed and should be repaired
    2019-05-28  9:56:25 140429762467584 [Warning] Checking table:   './dbispconfig/ftp_traffic'
    2019-05-28  9:56:26 140429762467584 [ERROR] mysqld: Table './dbispconfig/web_traffic' is marked as crashed and should be repaired
    2019-05-28  9:56:26 140429762467584 [Warning] Checking table:   './dbispconfig/web_traffic'
    2019-05-28 10:06:12 140429805909760 [Warning] IP address '89.248.174.4' could not be resolved: Name or service not known
    2019-05-28 10:19:32 140429762262784 [Warning] IP address '88.214.26.17' could not be resolved: Temporary failure in name resolution
    2019-05-28 10:47:39 140429741504256 [Warning] IP address '88.214.26.17' could not be resolved: Temporary failure in name resolution
    2019-05-28 11:15:09 140429741504256 [Warning] IP address '88.214.26.17' could not be resolved: Temporary failure in name resolution
    2019-05-28 11:20:11 140429761443584 [Warning] Hostname 'security.criminalip.com' does not resolve to '80.82.77.227'.
    2019-05-28 11:20:11 140429761443584 [Note] Hostname 'security.criminalip.com' has the following IP addresses:
    2019-05-28 11:20:11 140429761443584 [Note]  - 15.164.5.209
    2019-05-28 11:43:17 140429806319360 [Warning] IP address '88.214.26.17' could not be resolved: Temporary failure in name resolution
    2019-05-28 12:11:15 140429741504256 [Warning] IP address '88.214.26.17' could not be resolved: Temporary failure in name resolution
    2019-05-28 12:39:22 140429761853184 [Warning] IP address '88.214.26.17' could not be resolved: Temporary failure in name resolution
    2019-05-28 13:05:59 140429761443584 [Warning] Host name 'hostby.fcloud.biz' could not be resolved: Name or service not known
    2019-05-28 16:58:08 140429806319360 [Warning] Host name 'host210-228-110-95.serverdedicati.aruba.it' could not be resolved: Name or service not known
    2019-05-28 17:30:43 140536744246656 [Note] InnoDB: innodb_empty_free_list_algorithm has been changed to legacy because of small buffer pool size. In order to use backoff, increase buffer pool at least up to 20MB.
    
    2019-05-28 17:30:43 140536744246656 [Note] InnoDB: Using mutexes to ref count buffer pool pages
    2019-05-28 17:30:43 140536744246656 [Note] InnoDB: The InnoDB memory heap is disabled
    2019-05-28 17:30:43 140536744246656 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
    2019-05-28 17:30:43 140536744246656 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier
    2019-05-28 17:30:43 140536744246656 [Note] InnoDB: Compressed tables use zlib 1.2.8
    2019-05-28 17:30:43 140536744246656 [Note] InnoDB: Using Linux native AIO
    2019-05-28 17:30:43 140536744246656 [Note] InnoDB: Using SSE crc32 instructions
    2019-05-28 17:30:43 140536744246656 [Note] InnoDB: Initializing buffer pool, size = 128.0M
    InnoDB: mmap(140574720 bytes) failed; errno 12
    2019-05-28 17:30:43 140536744246656 [ERROR] InnoDB: Cannot allocate memory for the buffer pool
    2019-05-28 17:30:43 140536744246656 [ERROR] Plugin 'InnoDB' init function returned error.
    2019-05-28 17:30:43 140536744246656 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
    2019-05-28 17:30:43 140536744246656 [Note] Plugin 'FEEDBACK' is disabled.
    2019-05-28 17:30:43 140536744246656 [ERROR] Unknown/unsupported storage engine: InnoDB
    2019-05-28 17:30:43 140536744246656 [ERROR] Aborting
    
     
  2. Taleman

    Taleman Well-Known Member HowtoForge Supporter

    Did you repair the tables that show up as crashed? Use for example PHPMyAdmin to do the repair.
     
  3. jbonlinea

    jbonlinea Member

    hi and thank's for your reply
    not before posting, but it's done now !

    I knew i could repair table, but always wonder why it crashed in the first place ? and if repairing a table wasn't juste delaying the issue ?
     
  4. Taleman

    Taleman Well-Known Member HowtoForge Supporter

    You can try reading the database log to maybe find why there were crashes originally. Also examine other logs like syslog at time of the original crash. Now it crashed at startup because there already were messed up tables.
     
  5. jbonlinea

    jbonlinea Member

    ok
    is it diferent from what I can find in /var/log/mysql ?
     
  6. jbonlinea

    jbonlinea Member

    hi again guys

    I've monitored /var/log/mysql/error.log
    My mysql server went down again and I could find theses lines in the log.
    Does it simply means that my server do not have enough memory ?
    Why would it run out of memory now while the trafic hasn't changed ?

    Thank's in advance
    Code:
    2019-06-04  9:35:04 139946440940928 [ERROR] mysqld: Out of memory (Needed 128663552 bytes)                                                                                                                        │
    2019-06-04  9:35:04 139946440940928 [ERROR] mysqld: Out of memory (Needed 96485376 bytes)                                                                                                                         │
    2019-06-04  9:35:04 139946440940928 [Note] InnoDB: innodb_empty_free_list_algorithm has been changed to legacy because of small buffer pool size. In order to use backoff, increase buffer pool at least up to 20M│
                                                                                                                                                                                                                      │
    2019-06-04  9:35:04 139946440940928 [Note] InnoDB: Using mutexes to ref count buffer pool pages                                                                                                                   │
    2019-06-04  9:35:04 139946440940928 [Note] InnoDB: The InnoDB memory heap is disabled                                                                                                                             │
    2019-06-04  9:35:04 139946440940928 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins                                                                                                                   │
    2019-06-04  9:35:04 139946440940928 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier                                                                                                 │
    2019-06-04  9:35:04 139946440940928 [Note] InnoDB: Compressed tables use zlib 1.2.8                                                                                                                               │
    2019-06-04  9:35:04 139946440940928 [Note] InnoDB: Using Linux native AIO                                                                                                                                         │
    2019-06-04  9:35:04 139946440940928 [Note] InnoDB: Using SSE crc32 instructions                                                                                                                                   │
    2019-06-04  9:35:04 139946440940928 [Note] InnoDB: Initializing buffer pool, size = 128.0M                                                                                                                        │
    InnoDB: mmap(140574720 bytes) failed; errno 12                                                                                                                                                                    │
    2019-06-04  9:35:04 139946440940928 [ERROR] InnoDB: Cannot allocate memory for the buffer pool                                                                                                                    │
    2019-06-04  9:35:04 139946440940928 [ERROR] Plugin 'InnoDB' init function returned error.                                                                                                                         │
    2019-06-04  9:35:04 139946440940928 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.                                                                                                              │
    2019-06-04  9:35:04 139946440940928 [Note] Plugin 'FEEDBACK' is disabled.                                                                                                                                         │
    2019-06-04  9:35:04 139946440940928 [ERROR] Unknown/unsupported storage engine: InnoDB                                                                                                                            │
    2019-06-04  9:35:04 139946440940928 [ERROR] Aborting    
     
  7. Taleman

    Taleman Well-Known Member HowtoForge Supporter

    That looks like out of memory. Monitor free RAM and swap, and make bigger swap if they are exhausted. Or add more memory
    Code:
    free -h
     
  8. jbonlinea

    jbonlinea Member

    ok thank's it basically return that

    total used free shared buff/cache available
    Mem: 1.9G 1.4G 105M 224M 449M 191M
    Swap: 0B 0B 0B

    stupid question, but how could I free ram ?
    the sever run one nextcloud with colabora online which is pretty memory hungry. Could it be that theses apps got to use more ram through time ? could I make them re-use only what the truely need ?

    It's a vps runing the stretch perfect server.
    There is no swap, but could I add some ? I mean without partitioning ?
    Would this be an appropriate way to do it on the perfect server ? https://www.howtoforge.com/ubuntu-swap-file
     
  9. Taleman

    Taleman Well-Known Member HowtoForge Supporter

    That does not yet look like memory is full, but with no swap it can lead to out of memory when some big process tries to get memroy.
    Find prosesses that use lots of memory and do not run those. Or adjust settings for those processes to force them to use less ram.
    That 2 GB memory with no swap is not for memory hungry applications.
    Probably not. Some applications tend to increase their memory consumption the longer they have been running. In this case you can force restart of those applications for example daily to get them to start their memory consumption again from zero.
    That Ubuntu swap file system should work in Debian Stretch also. But you can resize the partitions on disk and make room for swap partition, provided the disk partitions are not already full. Make 4 GB swap, either swapfile or disk partition.
    https://wiki.debian.org/Swap
     
  10. jbonlinea

    jbonlinea Member

    Ok,
    Very insightfull, big thnak's for your advise/support
    Regards
     

Share This Page