Today coworker woke me up with critical problem : mails simply stopped working. No matter what I tried, I always got the same message : no space left on device. This is c/p from log file May 19 07:29:55 pro1 postgrey[1661]: FATAL: ERROR: can't create DB environment: No space left on device (dbdir: /var/lib/postgrey uid/gid: 117,117) May 19 07:29:56 pro1 postfix/tlsmgr[2287]: error: open database /var/lib/postfix/smtpd_scache.db: No space left on device May 19 07:29:56 pro1 postfix/tlsmgr[2287]: error: open database /var/lib/postfix/smtp_scache.db: No space left on device May 19 07:29:57 pro1 amavis[2296]: (!!)TROUBLE in pre_loop_hook: BDB can't create db env. at /var/lib/amavis/db: No space left on device, No space left on device. at (eval 92) line 312. Of course that was not a case, there is plenty of space on all volumes. Except one : /dev/mapper/vg--ssd-lv--root is 100% full and I have no idea what to do with it. No configuration was not changed for quite some time now. When I moved operations on another (mirror) server, everything works smoothly. But my concerns are that if I move everything back to the first server the problem will repeat. I already googled, but could not find any usable solution. Any idea would be appreciated.
remove some files from /dev/mapper/vg--ssd-lv--root (maybe mounted to / so you can start with /var/log/ ? )
Filesystem Size Used Avail Use% Mounted on udev 7.9G 0 7.9G 0% /dev tmpfs 1.6G 9.1M 1.6G 1% /run /dev/mapper/vg--ssd-lv--root 19G 18G 0 100% / tmpfs 7.9G 39M 7.8G 1% /dev/shm tmpfs 5.0M 0 5.0M 0% /run/lock tmpfs 7.9G 0 7.9G 0% /sys/fs/cgroup /dev/vdc1 20G 1.7G 17G 10% /var/log /dev/vda1 938M 105M 786M 12% /boot tmpfs 1.6G 0 1.6G 0% /run/user/1000 log is separate mount, and ls of root reveals nothing. I am lost... but there's something strange. Although should be 1 GB free it shows 100% usage. Something seem to be locked.
Still have to find another 2-3 GB, but at least I managed to free something: /var/lib/fail2ban/fail3ban.sqlite was over 2 GB... Thanks for help.
Code: Filesystem Size Used Avail Use% Mounted on /dev/mapper/vg--ssd-lv--root 19G 18G 0 100% / The root partition is on the small side for an e-mail server. If you need to keep partition that small, monitor the disk usage.