[solved] Mailserver - "insufficient system storage" || Increasing Diskspace?

Discussion in 'ISPConfig 3 Priority Support' started by niemand-glaumy, Mar 19, 2019.

  1. Servus! Sorry, but I noobishly require help again:
    Ubuntu 18.04 on a Hyper-V machine with 15GB HDD (little low, turned it up, see below)
    Maillogs state "Insufficient System Storage".

    "df":
    Code:
    Filesystem                        1K-blocks    Used Available Use% Mounted on
    udev                                 435136       0    435136   0% /dev
    tmpfs                                 93152    1008     92144   2% /run
    /dev/mapper/ubuntu--vg-ubuntu--lv   4062912 3826956      9860 100% /
    tmpfs                                465760       0    465760   0% /dev/shm
    tmpfs                                  5120       0      5120   0% /run/lock
    tmpfs                                465760       0    465760   0% /sys/fs/cgroup
    /dev/loop0                            93184   93184         0 100% /snap/core/6350
    /dev/sda2                            999320   76940    853568   9% /boot
    tmpfs                                 93152       0     93152   0% /run/user/1000
    I don't quite understand the /dev/mapper/ubuntu--vg[...] part and why it is full. :/

    "fdisk -l":
    Code:
    Disk /dev/loop0: 91 MiB, 95408128 bytes, 186344 sectors
    Units: sectors of 1 * 512 = 512 bytes
    Sector size (logical/physical): 512 bytes / 512 bytes
    I/O size (minimum/optimal): 512 bytes / 512 bytes
    
    
    
    
    GPT PMBR size mismatch (31457279 != 104857599) will be corrected by w(rite).
    Disk /dev/sda: 50 GiB, 53687091200 bytes, 104857600 sectors
    Units: sectors of 1 * 512 = 512 bytes
    Sector size (logical/physical): 512 bytes / 4096 bytes
    I/O size (minimum/optimal): 4096 bytes / 4096 bytes
    Disklabel type: gpt
    Disk identifier: D444AFBC-006C-49F3-8D57-7F6C7B621906
    
    Device       Start      End  Sectors Size Type
    /dev/sda1     2048     4095     2048   1M BIOS boot
    /dev/sda2     4096  2101247  2097152   1G Linux filesystem
    /dev/sda3  2101248 31455231 29353984  14G Linux filesystem
    Now I turned up the virtual HDDspace to 50GB in Hyper-V. My question now is: which sda should I increase? Via fdisk or via lvextend?
     
  2. till

    till Super Moderator Staff Member ISPConfig Developer

    First, the most important part: make a backup of the whole VM before you continue :) These disk resize things are always a bit critical.

    I guess you will have to resize the sda3 partition. Do a backup of the mbr/gpt, delete sda3 with sfdisk or gdisk, create it again with same start sector but larger in size and finally use resize2fs to resize the filesystem.
     
    niemand-glaumy likes this.
  3. Taleman

    Taleman Well-Known Member HowtoForge Supporter

    niemand-glaumy likes this.
  4. Thanks to the both of you. I will look/read into it and report back when I got to it. :)
    Backing up the vHDD was the first thing I did before changing the size in Hyper-V. I seem to learn. :D
     
  5. Problem solved.
    That solution lead to another problem (classic hydra): Dovecot can't start because mysql is not working/starting.
    Code:
    root@web:/# service mysql restart
    Job for mariadb.service failed because the control process exited with error code.
    See "systemctl status mariadb.service" and "journalctl -xe" for details.
    root@web:/# service mysql status
    ● mariadb.service - MariaDB 10.1.38 database server
       Loaded: loaded (/lib/systemd/system/mariadb.service; enabled; vendor preset: enabled)
       Active: failed (Result: exit-code) since Thu 2019-03-21 16:05:41 UTC; 15s ago
         Docs: man:mysqld(8)
               https://mariadb.com/kb/en/library/systemd/
      Process: 3220 ExecStart=/usr/sbin/mysqld $MYSQLD_OPTS $_WSREP_NEW_CLUSTER $_WSREP_START_POSITION (code=exited, status=1/FAILURE)
      Process: 3147 ExecStartPre=/bin/sh -c [ ! -e /usr/bin/galera_recovery ] && VAR= ||   VAR=`/usr/bin/galera_recovery`; [ $? -eq 0 ]   && systemctl set-environment _WSREP_START_POSITION=$VAR
      Process: 3140 ExecStartPre=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exited, status=0/SUCCESS)
      Process: 3128 ExecStartPre=/usr/bin/install -m 755 -o mysql -g root -d /var/run/mysqld (code=exited, status=0/SUCCESS)
     Main PID: 3220 (code=exited, status=1/FAILURE)
       Status: "MariaDB server is down"
    
    Mar 21 16:05:38 web systemd[1]: Starting MariaDB 10.1.38 database server...
    Mar 21 16:05:38 web mysqld[3220]: 2019-03-21 16:05:38 139957119171712 [Note] /usr/sbin/mysqld (mysqld 10.1.38-MariaDB-0ubuntu0.18.04.1) starting as process 3220 ...
    Mar 21 16:05:41 web systemd[1]: mariadb.service: Main process exited, code=exited, status=1/FAILURE
    Mar 21 16:05:41 web systemd[1]: mariadb.service: Failed with result 'exit-code'.
    Mar 21 16:05:41 web systemd[1]: Failed to start MariaDB 10.1.38 database server.
    lines 1-17/17 (END)
    root@web:/# service mysql reload
     * Reloading MariaDB database server mysqld                                                                                                                                                  /usr/bin/mysqladmin: connect to server at 'localhost' failed
    error: 'Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2 "No such file or directory")'
    Check that mysqld is running and that the socket: '/var/run/mysqld/mysqld.sock' exists!
    mysqld.sock does of course not exist. Would redoing Step6 of the perfect server Tutorial fix this?
     
  6. till

    till Super Moderator Staff Member ISPConfig Developer

    restart mysql and then check /var/log/syslog for the complete error message that caused mysql to fail.
     
    niemand-glaumy likes this.
  7. Thanks, that was the file I was looking for.
    Code:
    Mar 21 17:38:06 web systemd[1]: Starting MariaDB 10.1.38 database server...
    Mar 21 17:38:06 web mysqld[17332]: 2019-03-21 17:38:06 140569265572992 [Note] /usr/sbin/mysqld (mysqld 10.1.38-MariaDB-0ubuntu0.18.04.1) starting as process 17332 ...
    Mar 21 17:38:09 web systemd[1]: mariadb.service: Main process exited, code=exited, status=1/FAILURE
    Mar 21 17:38:09 web systemd[1]: mariadb.service: Failed with result 'exit-code'.
    Mar 21 17:38:09 web systemd[1]: Failed to start MariaDB 10.1.38 database server.
    I feel like that wasn't very helpful.
     
  8. till

    till Super Moderator Staff Member ISPConfig Developer

    Yes, that's not very helpful indeed as it nowhere mentions why it fails. Did you check that the hard disk has enough free space on all partitions now and are the partitions writable?
     
  9. Except for loop0 (100%) everything is below 10% usage now. In "ls -l /dev/" every sda has the same permissions/owner/group.
     
  10. till

    till Super Moderator Staff Member ISPConfig Developer

    It is not easy to say what's wrong without a meaningful error. You might e.g. check the databases:

    mysqlcheck -u root -p --all-databases --optimize --auto-repair

    maybe one of the database files is broken and needs a repair.
     
  11. That's an error mssg I found and posted earlier. I don't know if solving it would solve my issue.
    EDIT:
    Code:
    root@web:/#  mysqlcheck -u root -p --all-databases --optimize --auto-repair
    Enter password:
    mysqlcheck: Got error: 2002: Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2 "No such file or directory") when trying to connect
    
    EDIT2: found a "/var/log/mysql/error.log":
    Code:
    2019-03-22  0:49:00 140418422365312 [Note] InnoDB: innodb_empty_free_list_algorithm has been changed to legacy because of small buffer pool size. In order to use backoff, increase buffer pool at least up to 20MB.
    
    2019-03-22  0:49:00 140418422365312 [Note] InnoDB: Using mutexes to ref count buffer pool pages
    2019-03-22  0:49:00 140418422365312 [Note] InnoDB: The InnoDB memory heap is disabled
    2019-03-22  0:49:00 140418422365312 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
    2019-03-22  0:49:00 140418422365312 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier
    2019-03-22  0:49:00 140418422365312 [Note] InnoDB: Compressed tables use zlib 1.2.11
    2019-03-22  0:49:00 140418422365312 [Note] InnoDB: Using Linux native AIO
    2019-03-22  0:49:00 140418422365312 [Note] InnoDB: Using SSE crc32 instructions
    2019-03-22  0:49:00 140418422365312 [Note] InnoDB: Initializing buffer pool, size = 128.0M
    2019-03-22  0:49:00 140418422365312 [Note] InnoDB: Completed initialization of buffer pool
    2019-03-22  0:49:00 140418422365312 [Note] InnoDB: Highest supported file format is Barracuda.
    2019-03-22  0:49:00 140418422365312 [Note] InnoDB: 128 rollback segment(s) are active.
    2019-03-22  0:49:00 140418422365312 [Note] InnoDB: Waiting for purge to start
    2019-03-22  0:49:01 140418422365312 [Note] InnoDB:  Percona XtraDB (http://www.percona.com) 5.6.42-84.2 started; log sequence number 3159781
    2019-03-22  0:49:01 140418422365312 [Note] Plugin 'FEEDBACK' is disabled.
    2019-03-22  0:49:01 140418422365312 [Note] Recovering after a crash using tc.log
    2019-03-22  0:49:01 140418422365312 [ERROR] Can't init tc log
    2019-03-22  0:49:01 140418422365312 [ERROR] Aborting
    
    SOLVED: https://fransdejonge.com/2018/02/mariadb-fix-cant-init-tc-log-error/
     
    Last edited: Mar 22, 2019

Share This Page