Let's Encrypt unable to renew any certificates

Discussion in 'Installation/Configuration' started by LouTux, Dec 2, 2021.

  1. LouTux

    LouTux Member

    I have been trying to renew some certificates using Let's Encrypt, Check for dns issues, routing and updated the entire server including ISPConfig.
    I don't understand the issue pointed in the log. Prior to last week, I never had any issue renewing certificates.
    The server is using a public IP. So it is not even running being any NAT or other firewall besides UFW. Which again, never had issues with the server prior to last week.

    Any help would be greatly appreciated
     

    Attached Files:

    Last edited: Dec 2, 2021
  2. LouTux

    LouTux Member

    Yeah so I am starting to regret going for ispconfig, as I have been plague with soooooooooooo many issues lately, it's crazy

    Now since I performed the updates at around 13h50 today, the server now rejects all emails
    Recipient address rejected: Internal error occurred. Refer to server log for more information.; from=<[email protected]> to=<[email protected]> proto=ESMTP helo=<webmail.kimpex.com>

    This is a critical service and I can get it to work more then a week without issues
     
  3. Jesse Norell

    Jesse Norell ISPConfig Developer Staff Member ISPConfig Developer

    Code:
    2021-12-02 15:00:02,706:DEBUG:acme.client:Sending GET request to https://acme-v02.api.letsencrypt.org/directory.
    2021-12-02 15:00:02,711:DEBUG:requests.packages.urllib3.connectionpool:Starting new HTTPS connection (1): acme-v02.api.letsencrypt.org
    2021-12-02 15:00:02,868:DEBUG:certbot.log:Exiting abnormally:
    Traceback (most recent call last):
      File "/usr/lib/python3/dist-packages/urllib3/contrib/pyopenssl.py", line 417, in wrap_socket
        cnx.do_handshake()
      File "/usr/lib/python3/dist-packages/OpenSSL/SSL.py", line 1426, in do_handshake
        self._raise_ssl_error(self._ssl, result)
      File "/usr/lib/python3/dist-packages/OpenSSL/SSL.py", line 1174, in _raise_ssl_error
        _raise_current_error()
      File "/usr/lib/python3/dist-packages/OpenSSL/_util.py", line 48, in exception_from_error_queue
        raise exception_type(errors)
    OpenSSL.SSL.Error: [('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')]
    
    That sounds like your letsencrypt client was unable to verify the certificate of acme-v02.api.letsencrypt.org. In a quick check, I get this back from that servername:
    Code:
    Certificate chain
     0 s:CN = acme-v02.api.letsencrypt.org
       i:C = US, O = Let's Encrypt, CN = R3
     1 s:C = US, O = Let's Encrypt, CN = R3
       i:C = US, O = Internet Security Research Group, CN = ISRG Root X1
    
    Perhaps your server doesn't trust ISRG Root X1? Or your system is connecting to something else and getting a different cert chain; you can check yourself with:
    Code:
    openssl s_client -connect acme-v02.api.letsencrypt.org:443 -servername acme-v02.api.letsencrypt.org -showcerts
     
  4. Jesse Norell

    Jesse Norell ISPConfig Developer Staff Member ISPConfig Developer

    After updating your OS, did you run through the perfect server guide for the new OS version and ensure all packages are installed? After doing that, run the installer again (ispconfig_update.sh --force) and reconfigure services.
     
  5. LouTux

    LouTux Member

    my mail.err log is flooded with
    Dec 2 16:43:49 webman-01 dovecot: auth-worker(21126): Error: sql([email protected]): User query failed: Unknown column 'disablequota-status' in 'where clause'
    Dec 2 16:43:49 webman-01 dovecot: quota-status([email protected]): Error: user [email protected]: Auth USER lookup failed
     
  6. Jesse Norell

    Jesse Norell ISPConfig Developer Staff Member ISPConfig Developer

    Were there any errors when you updated ISPConfig? What version did you update from? That is a schema problem, perhaps your ISPConfig update didn't work correctly; you'll want to either roll back your update and run it again, or you could try running all the sql updates from your old version to the current version (ie. these are the files from the installer in the install/sql/incremental/ directory), or dump your current dbispconfig schema and compare to ispconfig3.sql in the installer to see what differs and make those changes.

    If you have a good backup prior to the update you can use this could be pretty quick to fix, but recovery from a bad update starts getting more technical than cut&pasting commands or running a script to setup a system; if you're feeling urgency and don't have the needed experience for this, keep in mind there are professional services you can hire to help out.
     
  7. LouTux

    LouTux Member

    Thanks for your reply.
    I had started setting up a new server before and had trouble with my ISP. Because they were filtering SMTP traffic. I have fix those issue last monday. I had also tested the server and everything worked during testing. So I'm thinking I will move all the mail users to that server. Once that's migrated, I will be able to focus more on fixing the various issues.
    From my past experience with ISPConfig, I found that using Ubuntu rather then Debian was more stable overall and was much much easier to update as their packages a usually more updated. My home webserver has been running on Ubuntu since what seems like forever. At work, I decided to go with Debian, since it had the reputation to be more stable in an enterprise environment. Boy was I wrong on that. I should've stayed with Ubuntu just because of the experience I had with it.
    Now on to the answers for your questions :
    Thats what I though too. But I did not verify with openssl. I have to admit, I never worked much directly with openssl. So my first though was ok maybe I should update the server to get all the latest certificates.
    Once I had updated, I did run ispconfig_update.sh --force and reconfigure services.
    I went from ISPConfig 3.15.2 to 3.2.7p1.
    During the update process, I marked mariadb-server on hold because it was causing issue when I tried to update it. I also encountered an error while updating pure-ftpd-mysql. But those problems seems to be related to an existing issue with the version that Debian has in it’s repository so I installed a fresh build using :
    Code:
    cd /tmp
    wget https://download.pureftpd.org/pub/pure-ftpd/releases/pure-ftpd-1.0.47.tar.gz
    tar xzvf pure-ftpd-1.0.47.tar.gz
    cd pure-ftpd-1.0.47
    ./configure --with-tls --with-virtualchroot --with-puredb --with-quotas --with-throttling --with-mysql
    make install-strip
    
    After this, I was able to start the pure-ftpd-mysql service and resume updating the system.

    Sorry for the late and long reply.
     
  8. LouTux

    LouTux Member

    Alright so, all accounts are being migrated, I was already to connect to one of the accounts and seems to be working fine on that server.
    Therefore, I will focus on the issues of the older server tomorrow.
     
  9. Taleman

    Taleman Well-Known Member HowtoForge Supporter

    But did you
     
  10. LouTux

    LouTux Member

    Yes

    Regardless, though I was able to migrate everyone to a different server. No I'll go ahead and reinstall the faulty server with Ubuntu instead of Debian.
     
  11. lnxgs

    lnxgs Member

    I also experienced problems with old Debian distribution like 8 and 9 too because the letsencrypt too was too old. I suggest you to keep your server distributions up to date and planning the installation of a new server with a view to migration in 1 or at most 2 years. So for example I use a "floating" public IP address that I can assign to another server, so I prepare the new server with all the sites and move the IP in a couple of minutes. No need of DNS changes for customers.
     
  12. LouTux

    LouTux Member

    Good thinking with the floating IP, but in my case, it's sitting behind a different Internet Service Provider. So I could not simply swap the IP. I had to either create a new DNS address or update the existing one.

    Initially when I built the debian machine, I had planned for futur updates. But the fact that some of my site required more up to date PHP versions and MariaDB Versions that are shipped with Debian, made it more complicated. I added the PHP repository and MariaDB repository directly, but 50% of the time doing an upgrade would fail on either MariaDB or PHP because some other requirement was not up to date. I am using Ubuntu Server 20.04 now, which seems to be more up to date on there repository. So far updating servers running Ubuntu did not cause any problem. It has been as simple as
    Code:
    sudo apt-get update && sudo apt-get upgrade -y
    .

    So based on my experience and my needs, I would not recommend Debian for ISPConfig.
     

Share This Page