SSH Service Inactive (dead) after installing ISP Config on Ubuntu 22.04

Discussion in 'Installation/Configuration' started by Neuropa, Jan 5, 2023.

  1. Neuropa

    Neuropa New Member

    Hosting provider : Aruba Business
    Machine : Hypervisor Openstack - 2 CPU - 4 GB RAM - SSD
    OS distro : Ubuntu Server 22.04 LTS 64bit
    ISP Config version: 3.2.9

    Hi, I'm experiencing a very strange behavior with ssh service.
    I can log in my machine via SSH and I can proceed successfully throughout the installation of ISP Config 3.2.9 following the HowtoForge tutorial "Perfect Server Automated ISPConfig 3 Installation on Debian 10, Debian 11, Ubuntu 20.04 and Ubuntu 22.04" with this installation arguments:
    --use-php=8.1 --use-ftp-ports=40110-40210 --no-mail --no-dns --no-local-dns --no-roundcube --monit [email protected] --unattended-upgrades=autoclean

    After installing ISP Config and rebooting the VPS I can Log in to the ISPConfig UI to setup the Firewall and configure everything else.
    But... when I try to access the machine via SSH I receive a "Connection refused" error. Firewall is OK.

    When I log in the console via hypervisor I can see the output of systemctl status ssh:
    ssh. service - OpenBSD Secure Shel1 server
    Loaded: loaded (/lib/systemd/system/ssh.service; enabled; vendor preset: enabled)
    Active: inactive (dead) since Thu 2023-01-05 18:06:23 CET; 50min ago
    Docs: man:sshd (8)
    man:sshd_config (5)
    Process: 728 ExecStartPre=/usr/sbin/sshd -t (code-exited, status=0/SUCCESS)
    Process: 752 ExecStart=/usr/sbin/sshd -D $SSHD_OPTS (code=exited, status=0/SUCCESS)
    Main PID: 752 (codesexited, status=0/SUCCESS)
    CPU: 12ms
    Jan 05 18:06:22 webserver-22 systemd[1]: Starting OpenBSD Secure Shel1 server...
    Jan 05 18:06:22 webserver-22 sshd [752]: Server listening on 0.0.0.0 port 9190.
    Jan 05 18:06:22 webserver-22 systemd [1]: Started OpenBSD Secure Shell server.
    Jan 05 18:06:22 webserver-22 sshd [752]: Server listening on :: port 9190.
    Jan 05 18:06:28 webserver-22 systemd[1]: Stopping OpenBSD Secure Shell server...
    Jan 05 18:06:23 webserver-22 systemd[1]: ssh.service: Deactivated successfully.
    Jan 05 18:06:23 webserver-22 sshd [752]: Received signal 15; terminating.
    Jan 05 18:06:23 webserver-22 systemd[1]: Stopped OpenBSD Secure Shel1 server.

    If I manually enable ssh service, something (maybe ISP Config itself?) stops it immediately.

    Can you please help me?
    Thanks in advance!
    Nico
     
  2. till

    till Super Moderator Staff Member ISPConfig Developer

    ISPConfig does not manage the ssh server config, so it neither starts nor stops this service. Your problem must be related to something else or is s side effect like system running out of memory or something similar.
     
  3. ahrasis

    ahrasis Well-Known Member HowtoForge Supporter

    Check if you swap file is also properly set in case of system runs out of memory as suggested by @till above. I noticed some took this swap thingy very lightly and ended up having avoidable problem(s) on their server.
     
  4. Neuropa

    Neuropa New Member

    Thanks everyone.
    @ahrasis the swap file is ok: Total 524284 Used 0 Free 524284
    @till I'm sure the problem is not with ISP Config, but it is definitely related to the autoinstaller script.
    I did a fresh install, starting from scratch again.
    These are the simple steps to reproduce my problem:
    - Ubuntu Server 22.04 freshly installed, ssh service is active and working.
    - I run the ISP Config autoinstaller script; as soon as the script finishes, without restarting the server, ssh is NOT active anymore.

    I'm trying to figure out what the script installs and configures. Any help is greatly appreciated. Thanks again!
     
  5. till

    till Super Moderator Staff Member ISPConfig Developer

    Run:

    cat /etc/ssh/sshd_config.d/custom.conf

    and post the output. Then delete the file /etc/ssh/sshd_config.d/custom.conf and restart the server, does it work now?
     
  6. Neuropa

    Neuropa New Member

    Hi everyone!
    @till /etc/ssh/sshd_config.d/custom.conf is a one row file with:
    # Created by the ISPConfig autoinstaller on 2023-01-07

    Thanks to a thorough investigation I discovered that the issue was monit!
    Running the ISP Config auto installer script without the --monit and --monit-alert-email arguments the problem no longer occurs.

    I still don't know why, but it just seems that installing monit breaks the ssh service on a freshly installed Ubuntu Server 22.04

    Anyone able to test this behavior with me?
     
  7. nhybgtvfr

    nhybgtvfr Well-Known Member HowtoForge Supporter

    you may just need to install rkhunter if it's not installed and then run rkhunter --propupd and restart/reload monit, it may be that monit notices that the ssh files don't match the checksum it expects and so stops the service because it may be unsafe/compromised.
     
    ahrasis likes this.
  8. ahrasis

    ahrasis Well-Known Member HowtoForge Supporter

    I don't see these monit params in the tutorial nor that I have tested it via cli before but I do believe this should be investigated though @nhybgtvfr suggestion may also be good.
     
  9. till

    till Super Moderator Staff Member ISPConfig Developer

    Just run a test install with enabling monit and there are no issues with the ssh connection.

    Code:
    wget -O - https://get.ispconfig.org | sh -s -- --use-ftp-ports=40110-40210 --unattended-upgrades --monit --monit-alert-email=root@localhost
     
  10. Neuropa

    Neuropa New Member

    @ahrasis the last version of the ISP Config autoinstaller has these monit related arguments:
    --monit. Install Monit and set it up to monitor installed services. Supported services: Apache2, NGINX, MariaDB, pure-ftpd-mysql, php-fpm, ssh, named, Postfix, Dovecot, rspamd.
    --monit-alert-email Set up alerts for Monit to be send to given e-mail address. e.g. [email protected]

    @till I also tried on a local vm and everything works fine, but I don't know why on that Aruba VPS (freshly installed) the problem occurs every time I try to install ISP Config with the monit arguments.
    I just found messages from the monit service in my spam box. I list them below:

    Does not exist Service redis-server
    Date: Sun, 08 Jan 2023 11:33:45
    Action: restart
    Description: process is not running

    Execution failed Service redis-server
    Date: Sun, 08 Jan 2023 11:34:15
    Action: alert
    Description: failed to start (exit status 5) -- '/usr/bin/systemctl start redis-server': Failed to start redis-server.service: Unit redis-server.service not found.

    Timeout Service redis-server
    Date: Sun, 08 Jan 2023 11:40:20
    Action: unmonitor
    Description: service restarted 5 times within 5 cycles(s) - unmonitor

    Exists Service pure-ftpd-mysql
    Date: Sun, 08 Jan 2023 11:41:22
    Action: alert
    Description: process is running with pid 1169

    Does not exist Service sshd_dsa_key
    Date: Sun, 08 Jan 2023 11:33:44
    Action: restart
    Description: file doesn't exist

    Thanks and regards.
     
  11. nhybgtvfr

    nhybgtvfr Well-Known Member HowtoForge Supporter

    Does not exist Service sshd_dsa_key
    Date: Sun, 08 Jan 2023 11:33:44
    Action: restart
    Description: file doesn't exist

    well. that's probably why ssh isn't working..

    you can probably get around that, if you can connect in any way, by removing
    Code:
       depend on sshd_dsa_key
    
    and possibly
    Code:
     check file sshd_dsa_key with path /etc/ssh/ssh_host_dsa_key
       group sshd
       include /etc/monit/templates/rootstrict
    
    from /etc/monit/conf-enabled/sshd and restart monit.

    although it might then progress to complaining about some other ssh file...
    no idea why that key file doesn't exist though.... also doesn't look like redis got installed/configured properly either.

    wonder if there's possibly somethiing odd / non-standard in the Aruba ubuntu images...
     
  12. sergio982

    sergio982 New Member

    The problem is that monit is killing sshd server. sshd stop litenning on port 22. If you use different port and not 22 you have change configuration on /etc/monit/conf-enabled/sshd. If it does not work yo have to remove file /etc/monit/conf-enabled/sshd
    Code:
    sudo rm /etc/monit/conf-enabled/sshd
    sudo systemctl restart monit
     
    Last edited: Jan 15, 2023
  13. nhybgtvfr

    nhybgtvfr Well-Known Member HowtoForge Supporter

    you do not have to remove /etc/monit/conf-enabled/sshd.
    you'll want that service monitored.
    yes, if you're using a different port you'll want to change the configuration of the checks.
    also. if any of the dependant files get changed or are missing, you'll want to check that those files were changed legitimately or why they don't exist, update the expected configuration/checksums for those files, and reload/restart monit so it uses the correct checksums.
    ie check the rkhunter logs to see what was changed, when, then check if a legitimate update was applied at that time. if so, run rkhunter --propupd to update the checksums, and reload monit.. or repair/reinstall the service files with ones from a trusted repository.
     
  14. ustoopia

    ustoopia Member

    I had the exact same issue. I used the automated installer including the --monit line, and noticed that ssh would stop functioning each time I started it. The solution that worked for me was this one mentioned earlier in this thread by nhybgtvfr. Thanks!
     
    ahrasis likes this.

Share This Page