Websites have stopped responding, how to troubleshoot?

Hello everyone!
Today, I tried to add a new website to the ISPC control panel and was met with very slow page load times. When I tried to visit one of the sites hosted on the machine, it resulted in a non-responsive website error. Testing with an independent service confirmed that the site was not responsive.

After a server reboot, I was no longer able to even access the ISPC control panel. I can still shell in as root or user and my SSHFS connections are still working. I just can't access the websites on the machine, including ISPC control panel.

Where should I begin troubleshooting this issue?

Thanks!
Jason

 

Hello there and thanks for the help!

 

I don't know about your websites space but I'd normally look update (if any) and test my nginx web server and if there is nothing wrong, I'll check my php-fpm for all versions. For php-fpm I normally ensure all the needed and latest Ondrej Sury versions are installed / updated which now include php-8.0. I ensure my nginx web server is also from Ondrej Sury PPA as suggested in his php PPA.

 

It does look bad, indeed

Here's my syslog. I'm a layman and am having problems finding meaningful red lights in the document but hope that you might be able to pull a root issue from it:

https://schw.im/syslog.txt

This is quite a pickle in which I find myself. I'm sorry I'm not providing more helpful info but when prompted, I'll do my best to add more info.

Thanks!

 

I've found thousands of clamav errors in syslog, is there a chance that's causing an issue or is it just a symptom of something upstream?

I played around with trying to restart apache2 and found some hopefully useful information:

The "read only filesystem" sounds ominous. Could this be the root of my issue?

 

Looks like Apache is starting and running OK.
You had tons of other services also failing, most importantly mariadb. Test all of those and find out why they are not starting.

 

root@adolf:/var/log# touch /test.txt
touch: cannot touch '/test.txt': Read-only file system

It 's a read only filesystem. I wanted to look at /etc/fstab and found this:

That dogwoodthicket path is the last site I added to ISPC and doing that somehow wiped my fstab, there are no other lines in that file.

How do I go about restoring the mount lines in fstab?

 

Unfortunately, I have no backup for the file. Nothing in /etc and I've not upgraded ISPC on this VPS. I'm pretty sure I'm going at it from scratch.

 

Looking at this:


Could I just use this to mount /dev/xvda2 to / ?

Code:

/dev/xvda2     /               ext4    errors=remount-ro 0       1

 

I asked and will see what they say but for now, I figured I'd just steal my fstab mounting line from my home debian install and I'm back in a writeable OS. Unfortunately, most of the services are still failing to start. As an example, here's mariadb(from syslog):

The "no such process" seems worrisome. I looked in /usr/bin and there is indeed the "install" process. Does the error point toward another issue I need to pursue like the read-only error did?

 

thanks Jesse, I've got the file system mounting writable, am now trying to sort all of the failed starting processes. See my post above yours for the first one I'm trying to tacckle(mariadb). I've searched for "Failed to determine group credentials: No such process" and "Failed at step GROUP spawning /usr/bin/install: No such process" but all I find in Google are results relating to systemd and other errors not quite matching.

 

Try to install monit to monitor your server services in the future as it can send emails alerting you even if you forget to check manually. It also can viewed inside ISPConfig but you will need to add "use address 0.0.0.0" and enable ssl in monitrc file.

 

I'm still struggling to get this machine up and running again. None of the processes ISPC require are operating and after days of Googling, the only thing I can come up with is that the groups required by systemd aren't being found:

As an example, here’s mariadb:

Dec 8 09:03:39 adolf systemd[473]: mariadb.service: Failed to determine group credentials: No such process
Dec 8 09:03:39 adolf systemd[473]: mariadb.service: Failed at step GROUP spawning /usr/bin/install: No such process
Dec 8 09:03:39 adolf systemd[1]: mariadb.service: Control process exited, code=exited, status=216/GROUP
Dec 8 09:03:39 adolf systemd[1]: mariadb.service: Failed with result 'exit-code'.
Dec 8 09:03:39 adolf systemd[1]: Failed to start MariaDB 10.3.22 database server.
Failed to determine group credentials: No such process
Failed at step GROUP spawning /usr/bin/install: No such process

These groups clearly existed for nearly a year as the server has worked without issue but I don't know how to determine which groups are needed, how to see which currently exist and how to add the required groups. Is this something that can be managed or is this irrecoverable?