Pre-upgrade 3.1 > 3.2 & Ubuntu 18.04 > 20.04 Questions

Discussion in 'Installation/Configuration' started by snowweb, Jun 1, 2021.

  1. snowweb

    snowweb Member

    Sorry. Just realized we could check that! No. There is nothing in those directories (except empty.dir)
     
  2. snowweb

    snowweb Member

    Since the upgrade (which we'll need to roll-back shortly), the mail clients are failing to accept the new certificate, saying it's untrusted.
     
  3. till

    till Super Moderator Staff Member ISPConfig Developer

    Please post the result of:

    ls -la /var/www/clients/client2/web2/ssl/

    From which exact ISPConfig version did you upgrade?

    Do you have an ispserver.bundle file in /usr/local/ispconfig/interface/ssl/ that contains the SSL chain certificate?
     
  4. snowweb

    snowweb Member

    ls -la /var/www/clients/client2/web2/ssl/
    Code:
    root@s1:~# ls -la /var/www/clients/client2/web2/ssl/
    total 92
    drwxr-xr-x  2 root root    4096 Sep 13  2020 .
    drwxr-xr-x 11 web2 client2 4096 Aug 20  2020 ..
    lrwxrwxrwx  1 root root      47 Sep 13  2020 s1.snowweb.info-le.bundle -> /etc/letsencrypt/live/s1.snowweb.info/chain.pem
    -r--------  1 root root    1647 Aug 23  2020 s1.snowweb.info-le.bundle.old.20200823173906
    -r--------  1 root root    1647 Aug 25  2020 s1.snowweb.info-le.bundle.old.20200825124410
    -r--------  1 root root    1647 Aug 26  2020 s1.snowweb.info-le.bundle.old.20200826194315
    -r--------  1 root root    1647 Aug 26  2020 s1.snowweb.info-le.bundle.old.20200826194610
    -r--------  1 root root    1647 Aug 27  2020 s1.snowweb.info-le.bundle.old.20200827150634
    -r--------  1 root root    1647 Sep 13  2020 s1.snowweb.info-le.bundle.old.20200913204505
    lrwxrwxrwx  1 root root      51 Sep 13  2020 s1.snowweb.info-le.crt -> /etc/letsencrypt/live/s1.snowweb.info/fullchain.pem
    -r--------  1 root root    3932 Aug 23  2020 s1.snowweb.info-le.crt.old.20200823173906
    -r--------  1 root root    3932 Aug 25  2020 s1.snowweb.info-le.crt.old.20200825124410
    -r--------  1 root root    3932 Aug 26  2020 s1.snowweb.info-le.crt.old.20200826194315
    -r--------  1 root root    3903 Aug 26  2020 s1.snowweb.info-le.crt.old.20200826194610
    -r--------  1 root root    3903 Aug 27  2020 s1.snowweb.info-le.crt.old.20200827150634
    -r--------  1 root root    3903 Sep 13  2020 s1.snowweb.info-le.crt.old.20200913204505
    lrwxrwxrwx  1 root root      49 Sep 13  2020 s1.snowweb.info-le.key -> /etc/letsencrypt/live/s1.snowweb.info/privkey.pem
    -r--------  1 root root    3272 Aug 23  2020 s1.snowweb.info-le.key.old.20200823173906
    -r--------  1 root root    3272 Aug 25  2020 s1.snowweb.info-le.key.old.20200825124410
    -r--------  1 root root    3272 Aug 26  2020 s1.snowweb.info-le.key.old.20200826194315
    -r--------  1 root root    3272 Aug 26  2020 s1.snowweb.info-le.key.old.20200826194610
    -r--------  1 root root    3272 Aug 27  2020 s1.snowweb.info-le.key.old.20200827150634
    -r--------  1 root root    3272 Sep 13  2020 s1.snowweb.info-le.key.old.20200913204505
    lrwxrwxrwx  1 root root      51 Sep 13  2020 www.s1.snowweb.info-le.bundle -> /etc/letsencrypt/live/www.s1.snowweb.info/chain.pem
    -r--------  1 root root    1647 Sep 13  2020 www.s1.snowweb.info-le.bundle.old.20200913204404
    lrwxrwxrwx  1 root root      55 Sep 13  2020 www.s1.snowweb.info-le.crt -> /etc/letsencrypt/live/www.s1.snowweb.info/fullchain.pem
    -r--------  1 root root    3916 Sep 13  2020 www.s1.snowweb.info-le.crt.old.20200913204404
    lrwxrwxrwx  1 root root      53 Sep 13  2020 www.s1.snowweb.info-le.key -> /etc/letsencrypt/live/www.s1.snowweb.info/privkey.pem
    -r--------  1 root root    3272 Sep 13  2020 www.s1.snowweb.info-le.key.old.20200913204404
    
    ls -la /usr/local/ispconfig/interface/ssl/
    Code:
    root@s1:~# ls -la /usr/local/ispconfig/interface/ssl/
    total 36
    drwxr-x---  2 root      root      4096 Jun  4 00:20 .
    drwxr-x--- 10 ispconfig ispconfig 4096 Jun  4 00:14 ..
    -rwxr-x---  1 root      root        45 Jun  4 00:21 empty.dir
    -rwxr-x---  1 root      root      2078 Jun  4 00:20 ispserver.crt
    -rwxr-x---  1 root      root      1720 Jun  4 00:20 ispserver.csr
    -rwxr-x---  1 root      root      3247 Jun  4 00:20 ispserver.key
    -rwxr-x---  1 root      root      3311 Jun  4 00:19 ispserver.key.secure
    -rwxr-x---  1 root      root      5325 Jun  4 00:20 ispserver.pem
    
    Not sure why is says "total 36"? I only count 8 entries.

    Thanks Till
     
  5. Jesse Norell

    Jesse Norell ISPConfig Developer Staff Member ISPConfig Developer

    What do you see from 'ls -l /etc/letsencrypt/live/s1.snowweb.info/' and 'ls -tlr /etc/letsencrypt/archive/s1.snowweb.info/' ?
     
  6. snowweb

    snowweb Member

    Code:
    root@s1:~# ls -l /etc/letsencrypt/live/s1.snowweb.info/
    ls: cannot access '/etc/letsencrypt/live/s1.snowweb.info/': No such file or directory
    
    Code:
    root@s1:~# ls -tlr /etc/letsencrypt/archive/s1.snowweb.info/
    ls: cannot access '/etc/letsencrypt/archive/s1.snowweb.info/': No such file or directory
    
     
  7. snowweb

    snowweb Member

    Just noticed this though...
    Code:
    root@s1:~# ls -l /etc/letsencrypt/live/
    total 68
    drwxr-xr-x 2 root root 4096 Apr 29 03:00 audio4design.com
    drwxr-xr-x 2 root root 4096 May  6 03:00 cepher-bible-dist.info
    drwxr-xr-x 2 root root 4096 May  1 03:00 rjen.net
    drwxr-xr-x 2 root root 4096 Apr 19 02:25 snowkids.fun
    drwxr-xr-x 2 root root 4096 Apr 21 03:00 snowkids.fun-0001
    drwxr-xr-x 2 root root 4096 Apr 25 03:00 snowweb.net
    drwxr-xr-x 2 root root 4096 Apr 25 03:00 snowweb.net-0001
    drwxr-xr-x 2 root root 4096 Apr 29 03:00 webmail.audio4design.com
    drwxr-xr-x 2 root root 4096 Apr 30 03:00 webmail.dentaltravel.com.ph
    drwxr-xr-x 2 root root 4096 Apr 30 03:00 webmail.oralconfidence.com
    drwxr-xr-x 2 root root 4096 May  1 03:01 webmail.rjen.net
    drwxr-xr-x 2 root root 4096 Apr 21 03:00 webmail.snowkids.fun
    drwxr-xr-x 2 root root 4096 Aug 27  2020 webmail.snowweb.net
    drwxr-xr-x 2 root root 4096 Apr 24 02:32 webmail.snowweb.net-0001
    drwxr-xr-x 2 root root 4096 May 30 02:54 webmail.zconnect.ph
    drwxr-xr-x 2 root root 4096 May  9 03:00 www.s1.snowweb.info
    drwxr-xr-x 2 root root 4096 May 30 02:54 zconnect.ph
    
    Note the www.s1.snowweb.info near the bottom.
     
  8. snowweb

    snowweb Member

    Code:
    root@s1:~# ls -l /etc/letsencrypt/live/www.s1.snowweb.info/
    total 4
    lrwxrwxrwx 1 root root  43 May  9 03:00 cert.pem -> ../../archive/www.s1.snowweb.info/cert5.pem
    lrwxrwxrwx 1 root root  44 May  9 03:00 chain.pem -> ../../archive/www.s1.snowweb.info/chain5.pem
    lrwxrwxrwx 1 root root  48 May  9 03:00 fullchain.pem -> ../../archive/www.s1.snowweb.info/fullchain5.pem
    lrwxrwxrwx 1 root root  46 May  9 03:00 privkey.pem -> ../../archive/www.s1.snowweb.info/privkey5.pem
    -rw-r--r-- 1 root root 682 Sep 10  2020 README
    
     
  9. snowweb

    snowweb Member

    Code:
    root@s1:~# ls -tlr /etc/letsencrypt/archive/www.s1.snowweb.info/
    total 84
    -rw-r--r-- 1 root root 2269 Sep 10  2020 cert1.pem
    -rw-r--r-- 1 root root 3272 Sep 10  2020 privkey1.pem
    -rw-r--r-- 1 root root 3916 Sep 10  2020 fullchain1.pem
    -rw-r--r-- 1 root root 1647 Sep 10  2020 chain1.pem
    -rw-r--r-- 1 root root 3272 Nov 10  2020 privkey2.pem
    -rw-r--r-- 1 root root 3920 Nov 10  2020 fullchain2.pem
    -rw-r--r-- 1 root root 1647 Nov 10  2020 chain2.pem
    -rw-r--r-- 1 root root 2273 Nov 10  2020 cert2.pem
    -rw-r--r-- 1 root root 3272 Jan  9 03:00 privkey3.pem
    -rw-r--r-- 1 root root 3785 Jan  9 03:00 fullchain3.pem
    -rw-r--r-- 1 root root 1586 Jan  9 03:00 chain3.pem
    -rw-r--r-- 1 root root 2199 Jan  9 03:00 cert3.pem
    -rw-r--r-- 1 root root 3272 Mar 10 03:00 privkey4.pem
    -rw-r--r-- 1 root root 3790 Mar 10 03:00 fullchain4.pem
    -rw-r--r-- 1 root root 1586 Mar 10 03:00 chain4.pem
    -rw-r--r-- 1 root root 2204 Mar 10 03:00 cert4.pem
    -rw-r--r-- 1 root root 3272 May  9 03:00 privkey5.pem
    -rw-r--r-- 1 root root 5949 May  9 03:00 fullchain5.pem
    -rw-r--r-- 1 root root 3750 May  9 03:00 chain5.pem
    -rw-r--r-- 1 root root 2199 May  9 03:00 cert5.pem
    
    Sorry, realized you probably want that!
     
  10. Jesse Norell

    Jesse Norell ISPConfig Developer Staff Member ISPConfig Developer

    So your vhost file specifies a certificate file:
    Code:
       SSLCertificateFile /var/www/clients/client2/web2/ssl/s1.snowweb.info-le.crt
    Which is a symlink:
    Code:
    s1.snowweb.info-le.crt -> /etc/letsencrypt/live/s1.snowweb.info/fullchain.pem
    To a file in a directory which does not exist:
    Code:
    ls: cannot access '/etc/letsencrypt/live/s1.snowweb.info/': No such file or directory
    Presumably this directory does exist before you started your updates? Removing that would not be due to upgrading ISPConfig; I don't expext it comes from upgrading Ubuntu (eg. certbot package) either. Your earlier steps mentioned this, which is probably the cause:
     
    snowweb likes this.
  11. snowweb

    snowweb Member

    I think you've hit the bulls-eye there Jesse. Well spotted.

    So I was attempting to undo the steps of the tutorial Securing ISPConfig 3.1 With a Free Let's Encrypt SSL Certificate when I deleted the directory but I guess it's not right in this instance.

    The question is, do I just completely omit the command I used
    or do I just need to be more specific regarding what it should delete (I'm not sure what that is either!)?

    Thanks Till and Jesse. We really appreciate your patience and support.

    We're planning to give this another try tonight (8 hours time) but we're thinking that since support for 18.04 is built-in for 3.2, we will just update ISPConfig tonight and will let the dust settle for a few days before going for the OS upgrade.
     
  12. till

    till Super Moderator Staff Member ISPConfig Developer

    I would just leave out that command, I guess you can even leave the ahrasis setup in place as it is, just don't choose to create a new SSL cert during ISPConfig update.
     
  13. snowweb

    snowweb Member

    I'm really determined to try to get a manageable server this time round (one where things don't just suddenly stop working) and I think I need ISPConfig to manage the SSL renewal situation for best chance for that to happen. I also want to end up with a pretty standard setup which is easier to get support for if needed and easier to upgrade in the future (full server migrations are such a hassle), so do forgive me, but I'm going to try to press to get this working. Thanks for understanding.

    I'll try just not deleting that directory and hopefully the setup will still create the certificates and link them up correctly. I already un-did the rest of ahrasis setup.
     
  14. till

    till Super Moderator Staff Member ISPConfig Developer

    This will most likely fail, as the ispconfig certs are symlinks in /usr/local/ispconfig/interface/ssl/ and these symlinks point to a non existent location then. What you might do is this:

    Replace the symlinks in /usr/local/ispconfig/interface/ssl/ with files that contain the certs from /etc/letsencrypt/live/s1.snowweb.info/ folder before you delete the le folder (use certbot delete command to safely remove the cert), so basically you mimic that these certs are no LE certs by making a copy of them instead of symlinking to LE. Then you can let the ispconfig installer create new LE based certs during update, which will replace the copies you made earlier with symlinks to a new LE cert.
     
    snowweb likes this.
  15. snowweb

    snowweb Member

    Got it! Thanks for the steps Till. We'll give that a try later. :)
     
  16. ahrasis

    ahrasis Well-Known Member HowtoForge Supporter

    I am not sure what you did which causing the failures but if you run ISPConfig update and choose to create SSL for your server during the update process, even if you have deleted the LE certs before that update, that should in theory create new certs for the server and I do see that in your list.

    The only problem is if you run that delete command again after the update deliberately or by mistake.

    Do note also that the creation of the certs per that server FQDN has limit and you might hit that limit, so do check your logs if your LE certs creation failed, in case you are trying to create the new ones.
     
  17. snowweb

    snowweb Member

    Before I start, this is what I have in the following directories (this is different to what I sent yesterday because tonight it's before I start):

    Code:
    root@s1:~# ls -la /etc/letsencrypt/live/s1.snowweb.info/
    total 12
    drwxr-xr-x  2 root root 4096 Apr 17 02:56 .
    drwx------ 21 root root 4096 Oct  1  2020 ..
    lrwxrwxrwx  1 root root   39 Apr 17 02:56 cert.pem -> ../../archive/s1.snowweb.info/cert5.pem
    lrwxrwxrwx  1 root root   40 Apr 17 02:56 chain.pem -> ../../archive/s1.snowweb.info/chain5.pem
    lrwxrwxrwx  1 root root   44 Apr 17 02:56 fullchain.pem -> ../../archive/s1.snowweb.info/fullchain5.pem
    lrwxrwxrwx  1 root root   42 Apr 17 02:56 privkey.pem -> ../../archive/s1.snowweb.info/privkey5.pem
    -rw-r--r--  1 root root  682 Aug 19  2020 README
    Code:
    root@s1:~# ls -la /etc/letsencrypt/archive/s1.snowweb.info/
    total 88
    drwxr-xr-x  2 root root 4096 Apr 17 02:56 .
    drwx------ 21 root root 4096 Oct  1  2020 ..
    -rw-r--r--  1 root root 2285 Aug 19  2020 cert1.pem
    -rw-r--r--  1 root root 2289 Oct 18  2020 cert2.pem
    -rw-r--r--  1 root root 2220 Dec 17 17:39 cert3.pem
    -rw-r--r--  1 root root 2216 Feb 15 19:19 cert4.pem
    -rw-r--r--  1 root root 2216 Apr 17 02:56 cert5.pem
    -rw-r--r--  1 root root 1647 Aug 19  2020 chain1.pem
    -rw-r--r--  1 root root 1647 Oct 18  2020 chain2.pem
    -rw-r--r--  1 root root 1586 Dec 17 17:39 chain3.pem
    -rw-r--r--  1 root root 1586 Feb 15 19:19 chain4.pem
    -rw-r--r--  1 root root 1586 Apr 17 02:56 chain5.pem
    -rw-r--r--  1 root root 3932 Aug 19  2020 fullchain1.pem
    -rw-r--r--  1 root root 3936 Oct 18  2020 fullchain2.pem
    -rw-r--r--  1 root root 3806 Dec 17 17:39 fullchain3.pem
    -rw-r--r--  1 root root 3802 Feb 15 19:19 fullchain4.pem
    -rw-r--r--  1 root root 3802 Apr 17 02:56 fullchain5.pem
    -rw-r--r--  1 root root 3272 Aug 19  2020 privkey1.pem
    -rw-r--r--  1 root root 3272 Oct 18  2020 privkey2.pem
    -rw-r--r--  1 root root 3272 Dec 17 17:39 privkey3.pem
    -rw-r--r--  1 root root 3272 Feb 15 19:19 privkey4.pem
    -rw-r--r--  1 root root 3272 Apr 17 02:56 privkey5.pem
    Code:
    root@s1:~# ls -la /usr/local/ispconfig/interface/ssl/
    total 68
    drwxr-x--- 2 root      root      4096 Jun  4 23:37 .
    drwxr-x--- 9 ispconfig ispconfig 4096 Aug 14  2020 ..
    -rwxr-x--- 1 root      root        45 Aug 14  2020 empty.dir
    lrwxrwxrwx 1 root      root        51 Aug 19  2020 ispserver.crt -> /etc/letsencrypt/live/s1.snowweb.info/fullchain.pem
    -rwxr-x--- 1 root      root      2017 Aug 14  2020 ispserver.crt-200819111429.bak
    -rwxr-x--- 1 root      root      1687 Aug 14  2020 ispserver.csr
    lrwxrwxrwx 1 root      root        49 Aug 19  2020 ispserver.key -> /etc/letsencrypt/live/s1.snowweb.info/privkey.pem
    -rwxr-x--- 1 root      root      3247 Aug 14  2020 ispserver.key-200819111446.bak
    -rwxr-x--- 1 root      root      3311 Aug 14  2020 ispserver.key.secure
    -rw------- 1 root      root      7074 Apr 17 02:56 ispserver.pem
    -rw------- 1 root      root      7204 Aug 19  2020 ispserver.pem-201117131402.bak
    -rw------- 1 root      root      7208 Nov 17  2020 ispserver.pem-210116160626.bak
    -rw------- 1 root      root      7078 Jan 16 16:06 ispserver.pem-210317183053.bak
    -rw------- 1 root      root      7074 Mar 17 18:30 ispserver.pem-210417025626.bak
    So we're planning to do (before we commence ISPConfig update):
    rm /etc/postfix/smtpd.cert
    rm /etc/postfix/smtpd.key

    using nano /etc/dovecot/dovecot.conf
    remove:
    ssl_cert = </etc/postfix/smtpd.cert
    ssl_key = </etc/postfix/smtpd.key

    rm /etc/ssl/private/pure-ftpd.pem

    rm /usr/local/ispconfig/interface/ssl/*
    cp /etc/letsencrypt/archive/s1.snowweb.info/cert5.pem /usr/local/ispconfig/interface/ssl/cert.pem
    cp /etc/letsencrypt/archive/s1.snowweb.info/chain5.pem /usr/local/ispconfig/interface/ssl/chain.pem
    cp /etc/letsencrypt/archive/s1.snowweb.info/fullchain5.pem /usr/local/ispconfig/interface/ssl/ispserver.crt
    cp /etc/letsencrypt/archive/s1.snowweb.info/privkey5.pem /usr/local/ispconfig/interface/ssl/privkey.pem

    Please can you advise what command I need to issue to:
    Thanks
     
  18. snowweb

    snowweb Member

    After another four more attempts to update ISPConfig from 3.1 to 3.2 we have rolled back to before we began.

    It seems that one time, we got the s1.snowweb.info config right but then apache failed to start because it had an issue with the next domain. We tried removing all other .vhost files from sites-enabled but then the apache still wouldn't start due to an issue with the certificate. If apache won't start, then it can't validate the domain to issue the certificate and if it can't issue the certificate, you can't start apache.

    Last September we did a full server migration from another hosting company and control panel and installed ISPConfig, thinking that we would not need to do another migration for at least 5 years, if ever, since there was an upgrade path. But it seems we were mistaken and there is no way for us to upgrade. We have to do another full migration just nine months later.

    It's possible that our problems started on the day we installed ISPConfig, since according to our bash_history file, the following commands were used, which were the official procedure at that time were as follows:
    Code:
    wget -O ispconfig.tar.gz https://git.ispconfig.org/ispconfig/ispconfig3/repository/archive.tar.gz?ref=stable-3.1
    tar xfz ispconfig.tar.gz
    cd ispconfig3*/install/
    php -q install.php
    We were fully expecting that the resulting installation would be a "stable" release, but somehow, when we checked after the migration was complete, we had ended up with "3.1dev". Maybe this is part of our problem and perhaps why local websites are unable to send mail ever since.

    The other contributing factor (and most likely) is that we used a method endorsed here for securing the web sites with SSL. A great tutorial written by ahrasis, but unfortunately the advise on the forum when migrating from 3.1 to 3.2, is that what was accomplished in that procedure, has to be reverted in order for the upgrade to succeed (unless you want to forever manage your certificates manually) and there is no tested procedure that we could find to accomplish this. If there is, we could still use it. This is likely the biggest or possibly the entire reason why we're unable to upgrade.

    Anyway, we want to thank Till, ahrasis and Jesse for all your tireless inputs and your patience.

    Sabbath today. Will sleep now and will plan next steps on Sunday. Thanks again.
     
  19. Th0m

    Th0m ISPConfig Developer Staff Member ISPConfig Developer

    I wan to start this with one advice. Don't freak out when something goes wrong and roll back, but look into a fix. Those rollbacks can cause problems and you should be able to fix such issues.

    You are currently installing a dev version indeed, follow the Perfect server guide for your OS to use the correct version.

    The "old" method for securing your panel with a SSL cert can be used with 3.2 if wanted/necessary.

    It might be useful for you to hire someone to discuss this with directly who can solve issues for you. Especially if you are a hosting company with quite some clients, I suspect the cost is relatively low compared to the headaches it will prevent ;)
     
    ahrasis likes this.
  20. ahrasis

    ahrasis Well-Known Member HowtoForge Supporter

    I agree with @Th0m.

    In any event, ISPConfig development version is never a problem and one should be able to change from development to stable quite easily in most cases.

    The problem you have is undoing the previous setup in securing your ISPConfig server but once you settled this, you should be able to update your server to latest version of ISPConfig 3.2 and secure it during the same process.

    It is true that undoing the same will break the apache web server as in my script it removes the LE certs for the server but the ISPConfig 3.2 update should fix it by choosing to create SSL during the process. The step here is very crucial where you need to make sure that creating the new LE certs for your server is successful, or otherwise apache will fail to restart and the server will not be secured by LE certs.

    The root of my reasoning to delete the old LE certs for the server via my script was that the LE certs renewal conf file will not be changed and thus some ISPConfig new scripts will not be added to it if it is not deleted. At the same time, creation or renewal of new LE certs would failed without it, so the whole LE certs folder for the server need to be deleted.

    I already advised about the LE limit for creating certs for your server FQDN, that might be hit if you are not careful, so always check the LE logs if there is failure to create the new certs and determine what is its true cause(s). Once you fixed that problem(s), you may try ISPConfig update and choose creating new SSL, again, but do watch for the limit.

    Only when you have a good running Ubuntu 18.04 server with ISPConfig 3.2 that you should proceed to upgrade to Ubuntu 20.04 and the steps for this have been discussed very much in details above including making sure your server follow the ISPConfig PST for Ubuntu 20.04 and then run ISPConfig update again thereafter.

    I wish you the best of luck for tomorrow.
     
    Last edited: Jun 5, 2021

Share This Page