Broke the server while securing the admin panel

schwim · Jan 3, 2021

Hi there!

I had just finished the perfect Debian 10 setup, which worked perfectly but while following this tutorial, I broke the http access to the server and for the life of me, I can't figure out what I did wrong. The only warning I got was the at the very last step, where apache2 refused to restart. Up until that point, all responses to commands were as expected. I've got this to go by and would be very appreciative if someone could lead me toward a resolution:

root@adolf:~# systemctl restart apache2
Job for apache2.service failed because the control process exited with error code.
See "systemctl status apache2.service" and "journalctl -xe" for details.
root@adolf:~# systemctl status apache2.service
● apache2.service - The Apache HTTP Server
Loaded: loaded (/lib/systemd/system/apache2.service; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Sun 2021-01-03 12:15:53 CST; 3s ago
Docs: https://httpd.apache.org/docs/2.4/
Process: 1460 ExecStart=/usr/sbin/apachectl start (code=exited, status=1/FAILURE)

Jan 03 12:15:53 adolf systemd[1]: Starting The Apache HTTP Server...
Jan 03 12:15:53 adolf apachectl[1460]: AH00548: NameVirtualHost has no effect and will be removed in the next release /etc/apache2/sites-enabled/000-ispconfig.vhost:7
Jan 03 12:15:53 adolf apachectl[1460]: AH00526: Syntax error on line 65 of /etc/apache2/sites-enabled/000-ispconfig.vhost:
Jan 03 12:15:53 adolf apachectl[1460]: SSLCertificateFile: file '/usr/local/ispconfig/interface/ssl/ispserver.crt' does not exist or is empty
Jan 03 12:15:53 adolf apachectl[1460]: Action 'start' failed.
Jan 03 12:15:53 adolf apachectl[1460]: The Apache error log may have more information.
Jan 03 12:15:53 adolf systemd[1]: apache2.service: Control process exited, code=exited, status=1/FAILURE
Jan 03 12:15:53 adolf systemd[1]: apache2.service: Failed with result 'exit-code'.
Jan 03 12:15:53 adolf systemd[1]: Failed to start The Apache HTTP Server.
root@adolf:~# ^C
root@adolf:~# journalctl -xe
-- The process' exit code is 'exited' and its exit status is 1.
Jan 03 12:15:37 adolf systemd[1]: apache2.service: Failed with result 'exit-code'.
-- Subject: Unit failed
-- Defined-By: systemd
-- Support: https://www.debian.org/support
--
-- The unit apache2.service has entered the 'failed' state with result 'exit-code'.
Jan 03 12:15:37 adolf systemd[1]: Failed to start The Apache HTTP Server.
-- Subject: A start job for unit apache2.service has failed
-- Defined-By: systemd
-- Support: https://www.debian.org/support
--
-- A start job for unit apache2.service has finished with a failure.
--
-- The job identifier is 585 and the job result is failed.
Jan 03 12:15:46 adolf sshd[1452]: error: Could not load host key: /etc/ssh/ssh_host_ecdsa_key
Jan 03 12:15:46 adolf sshd[1452]: error: Could not load host key: /etc/ssh/ssh_host_ed25519_key
Jan 03 12:15:47 adolf sshd[1452]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=87.27.78.7 user=root
Jan 03 12:15:49 adolf sshd[1452]: Failed password for root from 87.27.78.7 port 33408 ssh2
Jan 03 12:15:51 adolf sshd[1452]: Received disconnect from 87.27.78.7 port 33408:11: Bye Bye [preauth]
Jan 03 12:15:51 adolf sshd[1452]: Disconnected from authenticating user root 87.27.78.7 port 33408 [preauth]
Jan 03 12:15:53 adolf systemd[1]: Starting The Apache HTTP Server...
-- Subject: A start job for unit apache2.service has begun execution
-- Defined-By: systemd
-- Support: https://www.debian.org/support
--
-- A start job for unit apache2.service has begun execution.
--
-- The job identifier is 647.
Jan 03 12:15:53 adolf apachectl[1460]: AH00548: NameVirtualHost has no effect and will be removed in the next release /etc/apache2/sites-enabled/000-ispconfig.vhost:7
Jan 03 12:15:53 adolf apachectl[1460]: AH00526: Syntax error on line 65 of /etc/apache2/sites-enabled/000-ispconfig.vhost:
Jan 03 12:15:53 adolf apachectl[1460]: SSLCertificateFile: file '/usr/local/ispconfig/interface/ssl/ispserver.crt' does not exist or is empty
Jan 03 12:15:53 adolf apachectl[1460]: Action 'start' failed.
Jan 03 12:15:53 adolf apachectl[1460]: The Apache error log may have more information.
Jan 03 12:15:53 adolf systemd[1]: apache2.service: Control process exited, code=exited, status=1/FAILURE
-- Subject: Unit process exited
-- Defined-By: systemd
-- Support: https://www.debian.org/support
--
-- An ExecStart= process belonging to unit apache2.service has exited.
--
-- The process' exit code is 'exited' and its exit status is 1.
Jan 03 12:15:53 adolf systemd[1]: apache2.service: Failed with result 'exit-code'.
-- Subject: Unit failed
-- Defined-By: systemd
-- Support: https://www.debian.org/support
--
-- The unit apache2.service has entered the 'failed' state with result 'exit-code'.
Jan 03 12:15:53 adolf systemd[1]: Failed to start The Apache HTTP Server.
-- Subject: A start job for unit apache2.service has failed
-- Defined-By: systemd
-- Support: https://www.debian.org/support
--
-- A start job for unit apache2.service has finished with a failure.
--
-- The job identifier is 647 and the job result is failed.
Jan 03 12:15:56 adolf named[447]: client @0x7f4e640e40f0 187.49.71.174#15983 (.): query (cache) './ANY/IN' denied
Jan 03 12:15:56 adolf named[447]: client @0x7f4e640e40f0 187.49.71.174#15983 (.): query (cache) './ANY/IN' denied
Jan 03 12:15:56 adolf named[447]: client @0x7f4e640e40f0 187.49.71.174#15983 (.): query (cache) './ANY/IN' denied
Jan 03 12:15:56 adolf named[447]: client @0x7f4e640e40f0 187.49.71.174#15983 (.): query (cache) './ANY/IN' denied
Jan 03 12:15:56 adolf named[447]: client @0x7f4e640e40f0 187.49.71.174#15983 (.): query (cache) './ANY/IN' denied
Click to expand...

schwim · Jan 3, 2021

I've gone back over every step to ensure I didn't clip any commands and I've found something odd, I've used both mobaxterm and the terminal to edit in incron file as per this instruction in the how-to

incrontab -e
Add this line in it incrontab:
/etc/letsencrypt/archive/$(hostname -f)/ IN_MODIFY ./etc/init.d/le_ispc_pem.sh
Click to expand...

So I paste the following into -e:
Code:
/etc/letsencrypt/archive/$(hostname -f)/ IN_MODIFY ./etc/init.d/le_ispc_pem.sh
But when I run -e again to look at it, it shows this:
Code:
/etc/letsencrypt/archive/$(hostname     IN_ALL_EVENTS   IN_MODIFY ./etc/init.d/le_ispc_pem.sh
I have no idea if this is the issue but I can't get it to honor what I'm entering into the file. Every time I open it back up, it's the modified entry above. I tried replacing '$(hostname -f)' with the actual server.server.com but it still doesn't work. I even tried replacing '$(hostname -f)' with the server.server.com value but I still can't start/restart apache2.

Can someone tell me where I'm going wrong and what I need to do to bring the service back up?

Th0m · Jan 3, 2021

You have to replace $(hostname -f) with the hostname used for your server. I guess this used to work but I had the same issue when following that tutorial a few months ago.

@till, it should be changed in the guide.

schwim · Jan 3, 2021

Hi there th0m and thanks for the help!

As indicated above I did try that in the incron file but the apache server still fails to start, I am guessing because of the previous breaking of the instructions. What do I need to do to get apache working again?

Th0m · Jan 3, 2021

Rerun this but replace $(hostname -f) with your hostname. Then start apache
Code:
ln -s /etc/letsencrypt/live/$(hostname -f)/fullchain.pem ispserver.crt
ln -s /etc/letsencrypt/live/$(hostname -f)/privkey.pem ispserver.key
Code:
systemctl start apache2

schwim · Jan 3, 2021

Th0m said: ↑
Rerun this but replace $(hostname -f) with your hostname. Then start apache
Code:
ln -s /etc/letsencrypt/live/$(hostname -f)/fullchain.pem ispserver.crt
ln -s /etc/letsencrypt/live/$(hostname -f)/privkey.pem ispserver.key
Code:
systemctl start apache2
Click to expand...
I ran the two commands, replacing with my actual hostname -f, starting apache2 still fails:

root@adolf:/etc/ssl/private# service apache2 start
Job for apache2.service failed because the control process exited with error code.
See "systemctl status apache2.service" and "journalctl -xe" for details.
root@adolf:/etc/ssl/private# systemctl status apache2.service
● apache2.service - The Apache HTTP Server
Loaded: loaded (/lib/systemd/system/apache2.service; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Sun 2021-01-03 14:26:29 CST; 24s ago
Docs: https://httpd.apache.org/docs/2.4/
Process: 8573 ExecStart=/usr/sbin/apachectl start (code=exited, status=1/FAILURE)

Jan 03 14:26:29 adolf systemd[1]: Starting The Apache HTTP Server...
Jan 03 14:26:29 adolf apachectl[8573]: AH00548: NameVirtualHost has no effect and will be removed in the next release /etc/apache2/sites-enabled/000-ispconfig
Jan 03 14:26:29 adolf apachectl[8573]: AH00526: Syntax error on line 65 of /etc/apache2/sites-enabled/000-ispconfig.vhost:
Jan 03 14:26:29 adolf apachectl[8573]: SSLCertificateFile: file '/usr/local/ispconfig/interface/ssl/ispserver.crt' does not exist or is empty
Jan 03 14:26:29 adolf apachectl[8573]: Action 'start' failed.
Jan 03 14:26:29 adolf apachectl[8573]: The Apache error log may have more information.
Jan 03 14:26:29 adolf systemd[1]: apache2.service: Control process exited, code=exited, status=1/FAILURE
Jan 03 14:26:29 adolf systemd[1]: apache2.service: Failed with result 'exit-code'.
Jan 03 14:26:29 adolf systemd[1]: Failed to start The Apache HTTP Server.
lines 1-15/15 (END)
root@adolf:/etc/ssl/private# journalctl -xe
-- The job identifier is 1595 and the job result is failed.
Jan 03 14:26:34 adolf sshd[8579]: error: Could not load host key: /etc/ssh/ssh_host_ecdsa_key
Jan 03 14:26:34 adolf sshd[8579]: error: Could not load host key: /etc/ssh/ssh_host_ed25519_key
Jan 03 14:26:34 adolf named[447]: client @0x7f4e640f2880 187.49.71.174#21663 (.): query (cache) './ANY/IN' denied
Jan 03 14:26:34 adolf named[447]: client @0x7f4e640f2880 187.49.71.174#21663 (.): query (cache) './ANY/IN' denied
Jan 03 14:26:34 adolf named[447]: client @0x7f4e640f2880 187.49.71.174#21663 (.): query (cache) './ANY/IN' denied
Jan 03 14:26:34 adolf named[447]: client @0x7f4e640f2880 187.49.71.174#21663 (.): query (cache) './ANY/IN' denied
Jan 03 14:26:34 adolf named[447]: client @0x7f4e640e40f0 187.49.71.174#21663 (.): query (cache) './ANY/IN' denied
Jan 03 14:26:37 adolf sshd[8579]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=51.77.140.36 user=root
Jan 03 14:26:39 adolf sshd[8579]: Failed password for root from 51.77.140.36 port 36436 ssh2
Jan 03 14:26:40 adolf sshd[8579]: Received disconnect from 51.77.140.36 port 36436:11: Bye Bye [preauth]
Jan 03 14:26:40 adolf sshd[8579]: Disconnected from authenticating user root 51.77.140.36 port 36436 [preauth]
Jan 03 14:26:42 adolf sshd[8585]: error: Could not load host key: /etc/ssh/ssh_host_ecdsa_key
Jan 03 14:26:42 adolf sshd[8585]: error: Could not load host key: /etc/ssh/ssh_host_ed25519_key
Jan 03 14:26:43 adolf sshd[8587]: error: Could not load host key: /etc/ssh/ssh_host_ecdsa_key
Jan 03 14:26:43 adolf sshd[8587]: error: Could not load host key: /etc/ssh/ssh_host_ed25519_key
Jan 03 14:26:44 adolf sshd[8585]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=119.63.84.130 user=root
Jan 03 14:26:44 adolf sshd[8587]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=220.88.1.208 user=root
Jan 03 14:26:46 adolf sshd[8585]: Failed password for root from 119.63.84.130 port 12208 ssh2
Jan 03 14:26:46 adolf sshd[8587]: Failed password for root from 220.88.1.208 port 49525 ssh2
Jan 03 14:26:46 adolf sshd[8585]: Received disconnect from 119.63.84.130 port 12208:11: Bye Bye [preauth]
Jan 03 14:26:46 adolf sshd[8585]: Disconnected from authenticating user root 119.63.84.130 port 12208 [preauth]
Jan 03 14:26:46 adolf sshd[8587]: Received disconnect from 220.88.1.208 port 49525:11: Bye Bye [preauth]
Jan 03 14:26:46 adolf sshd[8587]: Disconnected from authenticating user root 220.88.1.208 port 49525 [preauth]
Jan 03 14:26:49 adolf sshd[8589]: error: Could not load host key: /etc/ssh/ssh_host_ecdsa_key
Jan 03 14:26:49 adolf sshd[8589]: error: Could not load host key: /etc/ssh/ssh_host_ed25519_key
Jan 03 14:26:51 adolf sshd[8589]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=139.155.76.42 user=root
Jan 03 14:26:53 adolf sshd[8589]: Failed password for root from 139.155.76.42 port 58908 ssh2
Jan 03 14:26:55 adolf sshd[8589]: Received disconnect from 139.155.76.42 port 58908:11: Bye Bye [preauth]
Jan 03 14:26:55 adolf sshd[8589]: Disconnected from authenticating user root 139.155.76.42 port 58908 [preauth]
Jan 03 14:27:01 adolf CRON[8593]: pam_unix(cron:session): session opened for user root by (uid=0)
Jan 03 14:27:01 adolf CRON[8594]: pam_unix(cron:session): session opened for user root by (uid=0)
Jan 03 14:27:01 adolf CRON[8595]: (root) CMD (/usr/local/ispconfig/server/server.sh 2>&1 | while read line; do echo `/bin/date` "$line" >> /var/log/ispconfig/
Jan 03 14:27:01 adolf CRON[8596]: (root) CMD (/usr/local/ispconfig/server/cron.sh 2>&1 | while read line; do echo `/bin/date` "$line" >> /var/log/ispconfig/cr
Jan 03 14:27:01 adolf CRON[8593]: pam_unix(cron:session): session closed for user root
Jan 03 14:27:01 adolf CRON[8594]: pam_unix(cron:session): session closed for user root
Jan 03 14:27:08 adolf sshd[8607]: error: Could not load host key: /etc/ssh/ssh_host_ecdsa_key
Jan 03 14:27:08 adolf sshd[8607]: error: Could not load host key: /etc/ssh/ssh_host_ed25519_key
Jan 03 14:27:09 adolf sshd[8607]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=211.233.58.19 user=root
Jan 03 14:27:10 adolf sshd[8609]: error: Could not load host key: /etc/ssh/ssh_host_ecdsa_key
Jan 03 14:27:10 adolf sshd[8609]: error: Could not load host key: /etc/ssh/ssh_host_ed25519_key
Jan 03 14:27:11 adolf sshd[8607]: Failed password for root from 211.233.58.19 port 60352 ssh2

root@adolf:/etc/ssl/private
Click to expand...

Th0m · Jan 3, 2021

What is the output of
Code:
ls -la /usr/local/ispconfig/interface/ssl
?

schwim · Jan 3, 2021

root@adolf:/etc/ssl/private# ls -la /usr/local/ispconfig/interface/ssl
total 36
drwxr-x--- 2 root root 4096 Jan 3 12:57 .
drwxr-x--- 9 ispconfig ispconfig 4096 Jan 3 11:49 ..
-rwxr-x--- 1 root root 45 Jan 3 11:49 empty.dir
lrwxrwxrwx 1 root root 59 Jan 3 12:57 ispserver.crt -> /etc/letsencrypt/live/adolf.schwimserver3.com/fullchain.pem
-rwxr-x--- 1 root root 2204 Jan 3 11:49 ispserver.crt-210103115715.bak
lrwxrwxrwx 1 root root 59 Jan 3 11:57 ispserver.crt-210103125641.bak -> /etc/letsencrypt/live/adolf.schwimserver3.com/fullchain.pem
-rwxr-x--- 1 root root 1781 Jan 3 11:49 ispserver.csr
lrwxrwxrwx 1 root root 57 Jan 3 12:57 ispserver.key -> /etc/letsencrypt/live/adolf.schwimserver3.com/privkey.pem
-rwxr-x--- 1 root root 3243 Jan 3 11:49 ispserver.key-210103115724.bak
lrwxrwxrwx 1 root root 57 Jan 3 11:57 ispserver.key-210103125653.bak -> /etc/letsencrypt/live/adolf.schwimserver3.com/privkey.pem
-rwxr-x--- 1 root root 3311 Jan 3 11:48 ispserver.key.secure
-rw------- 1 root root 0 Jan 3 12:57 ispserver.pem
-rwxr-x--- 1 root root 5447 Jan 3 11:49 ispserver.pem-210103115733.bak
-rw------- 1 root root 0 Jan 3 11:58 ispserver.pem-210103125701.bak
root@adolf:/etc/ssl/private#
Click to expand...

Th0m · Jan 3, 2021

And the output of
Code:
ls -la /etc/letsencrypt/live/adolf.schwimserver3.com
?

schwim · Jan 3, 2021

No such file or directory. How would I rebiuld that?

Th0m · Jan 3, 2021

Did you create the site adolf.schwimserver3.com and enable Let's Encrypt?

schwim · Jan 3, 2021

Yes I did, both schwimserver3.com and adolf.schwimserver3.com resolved properly under https before I followed the tutorial to secure the admin panel.

Th0m · Jan 3, 2021

Are they both the same site, where one is a alias domain?
What is the content of /etc/letsencrypt/live ?

schwim · Jan 3, 2021

Th0m said: ↑

Are they both the same site, where one is a alias domain?
What is the content of /etc/letsencrypt/live ?
Click to expand...

I did not create an alias, they were two separate sites created via the admin panel. Both resolved without showing any https errors.

/etc/letsencrypt/live is an empty file.

Th0m · Jan 3, 2021

Ah, I see. You are using acme.sh instead of certbot, so the certificates are in a different location. The certs will be in /root/.acme.sh/ instead of /etc/letsencrypt/live. So replace that in your commands and it will work.

I prefer the manual method myself aswell, but it is possible to let the installer create a cert for the panel and other services. You can still do this by running
Code:
ispconfig_update.sh --force
and selecting yes for creating a new cert.

schwim · Jan 3, 2021

Th0m said: ↑
Ah, I see. You are using acme.sh instead of certbot, so the certificates are in a different location. The certs will be in /root/.acme.sh/ instead of /etc/letsencrypt/live. So replace that in your commands and it will work.

I prefer the manual method myself aswell, but it is possible to let the installer create a cert for the panel and other services. You can still do this by running
Code:
ispconfig_update.sh --force
and selecting yes for creating a new cert.
Click to expand...
Ok, so before I do any more damage, you're saying to create a new cert, then rerun the entire how-to, replacing hostname -f with the actual fqdn ,replacing /etc/letsencrypt/live with /root/.acme.sh?

Th0m · Jan 3, 2021

You can do 2 things:
- Run the update script and let it create a cert for you
OR
- Go through the howto again, replace /etc/letsencrypt/live with /root/.acme.sh and $(hostname -f) with adolf.schwimserver3.com, and when done, start apache

schwim · Jan 3, 2021

Ok, I ran the update script, chose to backup then reconfigure services, then recreate the ssl cert. Apache is working and schwimserver3.com, adolf.schwimserver.com and adolf.schwimserver3.com:8080 are all resolving securely without issue or errors.

With the problems I had, will this create issues down the road when the cert update script is supposed to run or am I good to go now do you think?

Th0m · Jan 3, 2021

No, but it would be good to remove the incron rule you created.

schwim · Jan 3, 2021

ok, I removed

etc/letsencrypt/archive/adolf.schwimserver3.com/ IN_MODIFY ./etc/init.d/le_ispc_pem.sh
Click to expand...

from incron, leaving it empty. Will I need to manually update 8080's certs when they expire now?

Log in or Sign up

Broke the server while securing the admin panel

schwim Member HowtoForge Supporter

schwim Member HowtoForge Supporter

Th0m ISPConfig Developer Staff Member ISPConfig Developer

schwim Member HowtoForge Supporter

Th0m ISPConfig Developer Staff Member ISPConfig Developer

schwim Member HowtoForge Supporter

Th0m ISPConfig Developer Staff Member ISPConfig Developer

schwim Member HowtoForge Supporter

Th0m ISPConfig Developer Staff Member ISPConfig Developer

schwim Member HowtoForge Supporter

Th0m ISPConfig Developer Staff Member ISPConfig Developer

schwim Member HowtoForge Supporter

Th0m ISPConfig Developer Staff Member ISPConfig Developer

schwim Member HowtoForge Supporter

Th0m ISPConfig Developer Staff Member ISPConfig Developer

schwim Member HowtoForge Supporter

Th0m ISPConfig Developer Staff Member ISPConfig Developer

schwim Member HowtoForge Supporter

Th0m ISPConfig Developer Staff Member ISPConfig Developer

schwim Member HowtoForge Supporter

Share This Page

Log in or Sign up

Broke the server while securing the admin panel

schwim Member HowtoForge Supporter

schwim Member HowtoForge Supporter

Th0m ISPConfig Developer Staff Member ISPConfig Developer

schwim Member HowtoForge Supporter

Th0m ISPConfig Developer Staff Member ISPConfig Developer

schwim Member HowtoForge Supporter

Th0m ISPConfig Developer Staff Member ISPConfig Developer

schwim Member HowtoForge Supporter

Th0m ISPConfig Developer Staff Member ISPConfig Developer

schwim Member HowtoForge Supporter

Th0m ISPConfig Developer Staff Member ISPConfig Developer

schwim Member HowtoForge Supporter

Th0m ISPConfig Developer Staff Member ISPConfig Developer

schwim Member HowtoForge Supporter

Th0m ISPConfig Developer Staff Member ISPConfig Developer

schwim Member HowtoForge Supporter

Th0m ISPConfig Developer Staff Member ISPConfig Developer

schwim Member HowtoForge Supporter

Th0m ISPConfig Developer Staff Member ISPConfig Developer

schwim Member HowtoForge Supporter

Share This Page

Useful Searches