Can't connect to database

Discussion in 'Server Operation' started by jimmie, May 20, 2024.

  1. jimmie

    jimmie Member

    Help, I think my file system is full, I created to many website backups and I need to delete all but last backup, how can th8s be accomplished through terminal. I can't connect to any site or ispconfig jnsflooringandsupplies.com
     
  2. ztk.me

    ztk.me Well-Known Member HowtoForge Supporter

    Looks like your database shutdown operating already, do not reboot.
    Depending what's more worthy delete old log files in
    /var/log
    rm *.gz
    rm *.1
    and sure enough you can delete backups manually aswell.
    just need to make ISPConfig aware after the fact, maybe though you can fake the files

    touch filename.tar.gz
    would create an empty file, I'd try "touching" those files you deleted.
    If you are lucky, database operation starts automatically again, or at least can be shutdown in a save manner.
     
  3. jimmie

    jimmie Member

    I already rebooted, wish I didn't. Before reboot I was able to see other sites, including ispconfig, I tried to do a restore when store manager did a computer restart,, now can't see any sites. Last thing computer did was do a backup which is set for 30 backups. Last night before this morning itworked
     
  4. ztk.me

    ztk.me Well-Known Member HowtoForge Supporter

    hope you do have some external storage where you could put some data on.
    If you are lucky, depending on your configuration, after space is available, you might have lost a certain amount of data
    but might be able to recover after some space has been freed.

    As the strategy on how to resolve incidients like that is nothing you can pull off a handbook but have to plan for yourself,
    can't really tell you how to proceed.
    If you enabled binlog on your database, it might be tempting to free up some space using that but
    this can/will lead to issues. If you have though, and the database is irecoverable, one could use them to try to
    extract data.
    Unfortunally if you rely on mysqldump it won't be a pretty result for sure :/

    Might be goot to seek for professional help depending on how worthy the data is.
     
  5. jimmie

    jimmie Member

    I got this
    phpMyAdmin - Error
    Error during session start; please check your PHP and/or webserver log file and configure your PHP installation properly. Also ensure that cookies are enabled in your browser.

    session_write_close(): write failed: No space left on device (28)

    session_write_close(): Failed to write session data (files). Please verify that the current setting of session.save_path is correct (/var/lib/php/sessions)
     
  6. jimmie

    jimmie Member

    I can access my server using putty or terminal
     
  7. ztk.me

    ztk.me Well-Known Member HowtoForge Supporter

    ssh does work even if you do not have disk space left.
    php sessions are, at least by default, saved on disk.

    If you have partitions or you are not sure what is full ( well /var for sure )
    du -h << plain wrong / typo ; check @Taleman 's post below
    that's why I suggested moving data to external drive if possible ( always better than deleting )
    /var/log
    may not free up much but maybe just enough to get to the next steps
    if not locate backups you can sacrifice
     
    Last edited: May 20, 2024
  8. jimmie

    jimmie Member

    du -h
    4.0K ./.cache
    4.0K ./.ssh
    60K .
     
  9. Taleman

    Taleman Well-Known Member HowtoForge Supporter

    These commands should show what is full:
    Code:
    df -hT
    df -hi
    If for example /var is full or maybe root / is full, find out what uses most disk space:
    Code:
    cd /var
    du -shc * | sort -h
    Then free space, by deleting oldest log files for example, or some other stuff not needed or that can be restored from backups. You can copy the data to somewhere else for safe keeping, if it is too valuable to be deleted.
    You should start monitoring disk space regularly. Maybe install logwatch and set it to e-mail you the report daily, report shows disk usage among other things.
     
    ztk.me likes this.
  10. jimmie

    jimmie Member

    when trying to login to phpmyadmin i get this
    mysqli_real_connect(): (HY000/2002): No such file or directory
    and
    Connection for controluser as defined in your configuration failed.
     
  11. jimmie

    jimmie Member

    i deleted all website backups
    and this is what it looks like now
    df -hT
    Code:
    Filesystem     Type      Size  Used Avail Use% Mounted on
    udev           devtmpfs  7.8G     0  7.8G   0% /dev
    tmpfs          tmpfs     1.6G  1.5M  1.6G   1% /run
    /dev/nvme0n1p2 ext4      915G   76G  794G   9% /
    tmpfs          tmpfs     7.8G     0  7.8G   0% /dev/shm
    tmpfs          tmpfs     5.0M  4.0K  5.0M   1% /run/lock
    tmpfs          tmpfs     7.8G     0  7.8G   0% /sys/fs/cgroup
    /dev/nvme0n1p1 vfat      1.1G  6.1M  1.1G   1% /boot/efi
    /dev/loop0     squashfs   64M   64M     0 100% /snap/core20/1828
    /dev/loop2     squashfs   50M   50M     0 100% /snap/snapd/18357
    /dev/loop1     squashfs   92M   92M     0 100% /snap/lxd/24061
    tmpfs          tmpfs     1.6G     0  1.6G   0% /run/user/1000
    df -hi
    Code:
    Filesystem     Inodes IUsed IFree IUse% Mounted on
    udev             2.0M   554  2.0M    1% /dev
    tmpfs            2.0M   972  2.0M    1% /run
    /dev/nvme0n1p2    59M  500K   58M    1% /
    tmpfs            2.0M     1  2.0M    1% /dev/shm
    tmpfs            2.0M     9  2.0M    1% /run/lock
    tmpfs            2.0M    18  2.0M    1% /sys/fs/cgroup
    /dev/nvme0n1p1      0     0     0     - /boot/efi
    /dev/loop0        12K   12K     0  100% /snap/core20/1828
    /dev/loop2        496   496     0  100% /snap/snapd/18357
    /dev/loop1        815   815     0  100% /snap/lxd/24061
    tmpfs            2.0M    21  2.0M    1% /run/user/1000
     
  12. jimmie

    jimmie Member

    Code:
    root@computer-hosting:/var# du -shc * | sort -h
    0       lock
    0       run
    4.0K    local
    4.0K    opt
    48K     snap
    76K     tmp
    632K    mail
    1.2M    crash
    3.1M    spool
    3.3M    backups
    50M     backup
    142M    cache
    1.9G    vmail
    2.4G    lib
    5.2G    log
    57G     www
    67G     total
    
     
  13. ztk.me

    ztk.me Well-Known Member HowtoForge Supporter

    disk usage is fine. snap packages are unusual but 100% is fine for those mounts, do not worry.
    you need to restart your database server again, likely
    $ systemctl restart mysql
    should also work for mariadb up to a point.

    Before you do that however, as you have enough space, make a copy of your database data.
    Default would be in /var/lib/mysql ( verify )
    you could do

    $ cd /var/lib/
    $ cp -pR mysql mysql_broken

    and now try to start the database. Depending on errors or not decide what to do next.
    $ journalctl -u mariadb -f
    $
    tail -f /var/log/syslog | grep mysql
    or /var/log/mysql/error.log

    there are various configurations out there.

    If you can't find the error log / output,

    you could aso try starting mariadb / mysql in foreground using

    $ mysqld_safe



    if you think something went wrong, need something to fix / do any force recovery,

    it is good you have the "broken backup"
     
  14. jimmie

    jimmie Member

    Logs begin at Thu 2024-05-16 04:20:58 UTC. --
    May 20 23:43:56 computer-hosting.com systemd[1]: mariadb.service: Failed with result 'exit-code'.
    May 20 23:43:56 computer-hosting.com systemd[1]: Failed to start MariaDB 10.3.39 database server.
    May 20 23:44:25 computer-hosting.com systemd[1]: Starting MariaDB 10.3.39 database server...
    May 20 23:44:25 computer-hosting.com sh[7356]: [108B blob data]
    May 20 23:44:25 computer-hosting.com sh[7356]: Fatal error in defaults handling. Program aborted
    May 20 23:44:25 computer-hosting.com mysqld[7359]: [98B blob data]
    May 20 23:44:25 computer-hosting.com mysqld[7359]: Fatal error in defaults handling. Program aborted
    May 20 23:44:25 computer-hosting.com systemd[1]: mariadb.service: Main process exited, code=exited, status=1/FAILURE
    May 20 23:44:25 computer-hosting.com systemd[1]: mariadb.service: Failed with result 'exit-code'.
    May 20 23:44:25 computer-hosting.com systemd[1]: Failed to start MariaDB 10.3.39 database server.
     
  15. jimmie

    jimmie Member

    May 21 00:19:07 computer-hosting dovecot: auth-worker: Error: Can't read dir of '/etc/mysql/conf.d/' (OS errno 2 - No such file or directory)
    May 21 00:19:07 computer-hosting dovecot: auth-worker: Error: [ERROR] Stopped processing the 'includedir' directive in file /etc/mysql/my.cnf at line 22.
    May 21 00:19:07 computer-hosting dovecot: auth-worker(9938): Error: mysql(localhost): Connect failed to database (dbispconfig): Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2) - waiting for 25 seconds before retry
     
  16. jimmie

    jimmie Member

    root@computer-hosting:~# mysqld_safe
    /usr/bin/my_print_defaults: Can't read dir of '/etc/mysql/conf.d/' (Errcode: 2 "No such file or directory")
    Fatal error in defaults handling. Program aborted
    /usr/bin/my_print_defaults: Can't read dir of '/etc/mysql/conf.d/' (Errcode: 2 "No such file or directory")
    Fatal error in defaults handling. Program aborted
    240521 00:21:54 mysqld_safe Logging to '/var/lib/mysql/computer-hosting.com.err'.
    240521 00:21:54 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
     
  17. ztk.me

    ztk.me Well-Known Member HowtoForge Supporter

    please examine the mentioned log /var/lib/mysql/computer-hosting.com.err
    but other than that, you have a ( temporarily ) running mysql server and at first glance it is looking good.
    Not sure why the "Error Can't read..." message is from dovecot though. Have a look at
    ls -l /etc/mysql
    there may be some mariadb files or folders, the my.cnf is there, so that's good. Nothing to worry, some folders may be referenced as include dir but are not required. Especially transission folders from mysql to mariadb renaming.

    after you stopped the mysqld_safe process ( go to another shell and use $ mysql -e 'SHUTDOWN'
    or follow this procedure https://www.percona.com/blog/prepare-mysql-for-a-safe-shutdown/ depending on your setup.

    You should be able to connect to your database now, and probably be able to
    systemctl start mariadb
     
  18. jimmie

    jimmie Member

    root@computer-hosting:/var/lib# mysql -e 'SHUTDOWN'
    mysql: Can't read dir of '/etc/mysql/conf.d/' (Errcode: 2 "No such file or directory")
    Fatal error in defaults handling. Program aborted
     
  19. jimmie

    jimmie Member

    root@computer-hosting:/var/lib# systemctl start mariadb
    Job for mariadb.service failed because the control process exited with error code.
    See "systemctl status mariadb.service" and "journalctl -xe" for details.
     
  20. jimmie

    jimmie Member

    40521 00:21:54 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
    ^G/usr/sbin/mysqld: Can't read dir of '/etc/mysql/conf.d/' (Errcode: 2 "No such file or directory")
    Fatal error in defaults handling. Program aborted
    240521 00:21:54 mysqld_safe mysqld from pid file /var/lib/mysql/computer-hosting.com.pid ended
     

Share This Page