web folders of all websites disappeared after server reboot

Hi,
I restarted server today after a few months, just to give it a refresher. I was on 3.3.0p1.

after server restart, control panel didnt load, I checked and apache2 status is errored. It first gave me .crt certificate file missing error for all websites, I commented out that error in .vhost file, now I am getting web folder missing for all websites. I restored my last night backup, but after restore VM still reboot as it does after restore. and again same error, all web folders inside /var/www/clients/client*/web*/web are missing, what in the world went wrong. Any help highly appreciated at the earliest, my server has some 15 websites, it's a production server, it's down for about 5 hours now.

I upgrade to 3.3.0p2, still same errors.
Thanks in advance.

 

Do you know if /var was a separate partition? If yes, check that it still exists and you may just need to fsck that partition.
If it wasn't a separate partition, your only option will be to look into older backups, if the partition or /var folder exists in the other remaining backups.

 

most likely separate partition/drive/share was mounted to /var/www
possibly originally created and mounted manually, but not added to fstab, so isn't automatically remounted after a reboot.

 

I had similar things happen with multi disks in Ubuntu on Hyper-V when the extra disk/partition was mounted from /dev/sdb1 for example.
On a reboot Ubuntu had switched disks. Sdb had become sda and vice versa.
Not an issue for Ubuntu itself as boot and root partitions are mounted from /dev/disk/by-id/xxx or /dev/disk/by-uuid/xxx

To solve the probleem and keep it from happening in the future you need to make sure that the extra disk(s)/partition(s) are not mounted from /dev/sdx (or whatever your disk is called) but from /dev/disk/by-id/xxx or /dev/disk/by-uuid/xxx
The id never changes so it doesn't have any effect anymore when disks get swapped around.

 

i find it easier to label any additional disks/partitions, and then use the label in /etc/fstab to mount them...

eg:
output of blkid:

Code:

/dev/sdb: LABEL="backup" UUID="1560f9e7-4f80-4acd-8faf-4bcf37020808" BLOCK_SIZE="4096" TYPE="ext4"
/dev/sdc: LABEL="webdisk" UUID="ce28d6f5-2a0f-46cd-882a-06c7d4cda1c4" BLOCK_SIZE="4096" TYPE="ext4"
/dev/sda: LABEL="mysql" UUID="78b7e4bd-5708-4b33-8c80-2ee55c65ac43" BLOCK_SIZE="4096" TYPE="ext4"

where /dev/sda /dev/sdb and /dev/sdc are hetzner volumes (block devices)

then to mount them in /etc/fstab:

Code:

LABEL=webdisk /var/www ext4 defaults,discard,errors=remount-ro,usrjquota=quota.user,grpjquota=quota.group,jqfmt=vfsv0 0 0
LABEL=backup /var/backup ext4 defaults,nofail,discard,errors=remount-ro,usrjquota=quota.user,grpjquota=quota.group,jqfmt=vfsv0 0 0
LABEL=mysql /var/lib/mysql ext4 defaults,nofail,discard,errors=remount-ro 0 0

each label is still fixed to a volume by the uuid so they don't change around like /dev/sda or /dev/nvme0n1p1 etc might at boot time, and they're much easier to identify quickly than by trying to read the actual uuid, also makes it easier to tell exactly what each drive/partition is used for.

 

Your label alternative probably uses the uuid shown by blkid in the background, so effectively /dev/disk/by-uuid/xxx is mounted.
Different approche, same result and solution.
I have to keep that labeling in mind It's interesting to use it for mounting.

 

Thanks for all for your input.

My finding so far is that, I had tried to move /var/www and /var/vmail few months back to new disk partition, but it had failed and /var/www remained on the root partition, and server was running with no issues. it was formatting issue on new partition, so, I had then formatted and then later changed mind and moved only /var/vmail. and with guide provided, things worked well and server has been working great with no issues with /var/www on root partition and /var/vmail on new disk due to ever increasing size of email folders.

I remember rebooting server a few times after, but honestly not sure about rebooting, all been working great for over 2 months after trying that.
When the web folders disappeared, I had reboot server as I felt it's been a while and giving it a bit of refresh. Before reboot, I had checked, everything was working fine. but not sure what went wrong but /var/www subfolders were all there before reboot and all of a sudden after reboot, the world came crashing on me.
I checked my Veeam backups upto 2 months old and and only /var/www folder exists, but no inside folders, so, I think most weirdly those folders on the server were present only in memory and not the disk and not on the file index. and server has been working all this time from the memory. There must be some kind of major glitch that made this possible, but I am not sure where is that glitch.

For my server, I ended up setting up new server and restored backups of individual websites. It was 25-50 times more work than restoring VM. but my VM backup was also corrupted, so, I had no choice. I have kept that instance for further troubleshooting. I have spent some 10 hours digging, but found no trace of any of those folders inside /var/www on the server or any of the full and incremental backups.

I am glad my backup plan mostly worked, except 1 website where I could only find 2024 backup. That's my fault that it's cloud integation had failed and I forgot to fix it, I have ISPconfig backup but couldn't use it.

Another lesson I feel is that ISPconfig's own backup is kinda useless for server fail. I couldn't restore any of those backups by simply uploading to new server. I think it's usable only while server is working and I need to restore to previous version. but not for full restore on new server. I will probably not use it going forward. Let me know if this changes.