Pre-upgrade 3.1 > 3.2 & Ubuntu 18.04 > 20.04 Questions

Sorry. Just realized we could check that! No. There is nothing in those directories (except empty.dir)

 

Since the upgrade (which we'll need to roll-back shortly), the mail clients are failing to accept the new certificate, saying it's untrusted.

 

ls -la /var/www/clients/client2/web2/ssl/

Code:

root@s1:~# ls -la /var/www/clients/client2/web2/ssl/
total 92
drwxr-xr-x  2 root root    4096 Sep 13  2020 .
drwxr-xr-x 11 web2 client2 4096 Aug 20  2020 ..
lrwxrwxrwx  1 root root      47 Sep 13  2020 s1.snowweb.info-le.bundle -> /etc/letsencrypt/live/s1.snowweb.info/chain.pem
-r--------  1 root root    1647 Aug 23  2020 s1.snowweb.info-le.bundle.old.20200823173906
-r--------  1 root root    1647 Aug 25  2020 s1.snowweb.info-le.bundle.old.20200825124410
-r--------  1 root root    1647 Aug 26  2020 s1.snowweb.info-le.bundle.old.20200826194315
-r--------  1 root root    1647 Aug 26  2020 s1.snowweb.info-le.bundle.old.20200826194610
-r--------  1 root root    1647 Aug 27  2020 s1.snowweb.info-le.bundle.old.20200827150634
-r--------  1 root root    1647 Sep 13  2020 s1.snowweb.info-le.bundle.old.20200913204505
lrwxrwxrwx  1 root root      51 Sep 13  2020 s1.snowweb.info-le.crt -> /etc/letsencrypt/live/s1.snowweb.info/fullchain.pem
-r--------  1 root root    3932 Aug 23  2020 s1.snowweb.info-le.crt.old.20200823173906
-r--------  1 root root    3932 Aug 25  2020 s1.snowweb.info-le.crt.old.20200825124410
-r--------  1 root root    3932 Aug 26  2020 s1.snowweb.info-le.crt.old.20200826194315
-r--------  1 root root    3903 Aug 26  2020 s1.snowweb.info-le.crt.old.20200826194610
-r--------  1 root root    3903 Aug 27  2020 s1.snowweb.info-le.crt.old.20200827150634
-r--------  1 root root    3903 Sep 13  2020 s1.snowweb.info-le.crt.old.20200913204505
lrwxrwxrwx  1 root root      49 Sep 13  2020 s1.snowweb.info-le.key -> /etc/letsencrypt/live/s1.snowweb.info/privkey.pem
-r--------  1 root root    3272 Aug 23  2020 s1.snowweb.info-le.key.old.20200823173906
-r--------  1 root root    3272 Aug 25  2020 s1.snowweb.info-le.key.old.20200825124410
-r--------  1 root root    3272 Aug 26  2020 s1.snowweb.info-le.key.old.20200826194315
-r--------  1 root root    3272 Aug 26  2020 s1.snowweb.info-le.key.old.20200826194610
-r--------  1 root root    3272 Aug 27  2020 s1.snowweb.info-le.key.old.20200827150634
-r--------  1 root root    3272 Sep 13  2020 s1.snowweb.info-le.key.old.20200913204505
lrwxrwxrwx  1 root root      51 Sep 13  2020 www.s1.snowweb.info-le.bundle -> /etc/letsencrypt/live/www.s1.snowweb.info/chain.pem
-r--------  1 root root    1647 Sep 13  2020 www.s1.snowweb.info-le.bundle.old.20200913204404
lrwxrwxrwx  1 root root      55 Sep 13  2020 www.s1.snowweb.info-le.crt -> /etc/letsencrypt/live/www.s1.snowweb.info/fullchain.pem
-r--------  1 root root    3916 Sep 13  2020 www.s1.snowweb.info-le.crt.old.20200913204404
lrwxrwxrwx  1 root root      53 Sep 13  2020 www.s1.snowweb.info-le.key -> /etc/letsencrypt/live/www.s1.snowweb.info/privkey.pem
-r--------  1 root root    3272 Sep 13  2020 www.s1.snowweb.info-le.key.old.20200913204404

ls -la /usr/local/ispconfig/interface/ssl/

Code:

root@s1:~# ls -la /usr/local/ispconfig/interface/ssl/
total 36
drwxr-x---  2 root      root      4096 Jun  4 00:20 .
drwxr-x--- 10 ispconfig ispconfig 4096 Jun  4 00:14 ..
-rwxr-x---  1 root      root        45 Jun  4 00:21 empty.dir
-rwxr-x---  1 root      root      2078 Jun  4 00:20 ispserver.crt
-rwxr-x---  1 root      root      1720 Jun  4 00:20 ispserver.csr
-rwxr-x---  1 root      root      3247 Jun  4 00:20 ispserver.key
-rwxr-x---  1 root      root      3311 Jun  4 00:19 ispserver.key.secure
-rwxr-x---  1 root      root      5325 Jun  4 00:20 ispserver.pem

Not sure why is says "total 36"? I only count 8 entries.

Thanks Till

 

Code:

root@s1:~# ls -l /etc/letsencrypt/live/s1.snowweb.info/
ls: cannot access '/etc/letsencrypt/live/s1.snowweb.info/': No such file or directory

Code:

root@s1:~# ls -tlr /etc/letsencrypt/archive/s1.snowweb.info/
ls: cannot access '/etc/letsencrypt/archive/s1.snowweb.info/': No such file or directory

 

Just noticed this though...

Code:

root@s1:~# ls -l /etc/letsencrypt/live/
total 68
drwxr-xr-x 2 root root 4096 Apr 29 03:00 audio4design.com
drwxr-xr-x 2 root root 4096 May  6 03:00 cepher-bible-dist.info
drwxr-xr-x 2 root root 4096 May  1 03:00 rjen.net
drwxr-xr-x 2 root root 4096 Apr 19 02:25 snowkids.fun
drwxr-xr-x 2 root root 4096 Apr 21 03:00 snowkids.fun-0001
drwxr-xr-x 2 root root 4096 Apr 25 03:00 snowweb.net
drwxr-xr-x 2 root root 4096 Apr 25 03:00 snowweb.net-0001
drwxr-xr-x 2 root root 4096 Apr 29 03:00 webmail.audio4design.com
drwxr-xr-x 2 root root 4096 Apr 30 03:00 webmail.dentaltravel.com.ph
drwxr-xr-x 2 root root 4096 Apr 30 03:00 webmail.oralconfidence.com
drwxr-xr-x 2 root root 4096 May  1 03:01 webmail.rjen.net
drwxr-xr-x 2 root root 4096 Apr 21 03:00 webmail.snowkids.fun
drwxr-xr-x 2 root root 4096 Aug 27  2020 webmail.snowweb.net
drwxr-xr-x 2 root root 4096 Apr 24 02:32 webmail.snowweb.net-0001
drwxr-xr-x 2 root root 4096 May 30 02:54 webmail.zconnect.ph
drwxr-xr-x 2 root root 4096 May  9 03:00 www.s1.snowweb.info
drwxr-xr-x 2 root root 4096 May 30 02:54 zconnect.ph

Note the www.s1.snowweb.info near the bottom.

 

Code:

root@s1:~# ls -l /etc/letsencrypt/live/www.s1.snowweb.info/
total 4
lrwxrwxrwx 1 root root  43 May  9 03:00 cert.pem -> ../../archive/www.s1.snowweb.info/cert5.pem
lrwxrwxrwx 1 root root  44 May  9 03:00 chain.pem -> ../../archive/www.s1.snowweb.info/chain5.pem
lrwxrwxrwx 1 root root  48 May  9 03:00 fullchain.pem -> ../../archive/www.s1.snowweb.info/fullchain5.pem
lrwxrwxrwx 1 root root  46 May  9 03:00 privkey.pem -> ../../archive/www.s1.snowweb.info/privkey5.pem
-rw-r--r-- 1 root root 682 Sep 10  2020 README

 

Code:

root@s1:~# ls -tlr /etc/letsencrypt/archive/www.s1.snowweb.info/
total 84
-rw-r--r-- 1 root root 2269 Sep 10  2020 cert1.pem
-rw-r--r-- 1 root root 3272 Sep 10  2020 privkey1.pem
-rw-r--r-- 1 root root 3916 Sep 10  2020 fullchain1.pem
-rw-r--r-- 1 root root 1647 Sep 10  2020 chain1.pem
-rw-r--r-- 1 root root 3272 Nov 10  2020 privkey2.pem
-rw-r--r-- 1 root root 3920 Nov 10  2020 fullchain2.pem
-rw-r--r-- 1 root root 1647 Nov 10  2020 chain2.pem
-rw-r--r-- 1 root root 2273 Nov 10  2020 cert2.pem
-rw-r--r-- 1 root root 3272 Jan  9 03:00 privkey3.pem
-rw-r--r-- 1 root root 3785 Jan  9 03:00 fullchain3.pem
-rw-r--r-- 1 root root 1586 Jan  9 03:00 chain3.pem
-rw-r--r-- 1 root root 2199 Jan  9 03:00 cert3.pem
-rw-r--r-- 1 root root 3272 Mar 10 03:00 privkey4.pem
-rw-r--r-- 1 root root 3790 Mar 10 03:00 fullchain4.pem
-rw-r--r-- 1 root root 1586 Mar 10 03:00 chain4.pem
-rw-r--r-- 1 root root 2204 Mar 10 03:00 cert4.pem
-rw-r--r-- 1 root root 3272 May  9 03:00 privkey5.pem
-rw-r--r-- 1 root root 5949 May  9 03:00 fullchain5.pem
-rw-r--r-- 1 root root 3750 May  9 03:00 chain5.pem
-rw-r--r-- 1 root root 2199 May  9 03:00 cert5.pem

Sorry, realized you probably want that!

 

I think you've hit the bulls-eye there Jesse. Well spotted.

So I was attempting to undo the steps of the tutorial Securing ISPConfig 3.1 With a Free Let's Encrypt SSL Certificate when I deleted the directory but I guess it's not right in this instance.

The question is, do I just completely omit the command I used

or do I just need to be more specific regarding what it should delete (I'm not sure what that is either!)?

Thanks Till and Jesse. We really appreciate your patience and support.

We're planning to give this another try tonight (8 hours time) but we're thinking that since support for 18.04 is built-in for 3.2, we will just update ISPConfig tonight and will let the dust settle for a few days before going for the OS upgrade.

 

I'm really determined to try to get a manageable server this time round (one where things don't just suddenly stop working) and I think I need ISPConfig to manage the SSL renewal situation for best chance for that to happen. I also want to end up with a pretty standard setup which is easier to get support for if needed and easier to upgrade in the future (full server migrations are such a hassle), so do forgive me, but I'm going to try to press to get this working. Thanks for understanding.

I'll try just not deleting that directory and hopefully the setup will still create the certificates and link them up correctly. I already un-did the rest of ahrasis setup.

 

Got it! Thanks for the steps Till. We'll give that a try later.

 

I am not sure what you did which causing the failures but if you run ISPConfig update and choose to create SSL for your server during the update process, even if you have deleted the LE certs before that update, that should in theory create new certs for the server and I do see that in your list.

The only problem is if you run that delete command again after the update deliberately or by mistake.

Do note also that the creation of the certs per that server FQDN has limit and you might hit that limit, so do check your logs if your LE certs creation failed, in case you are trying to create the new ones.

 

Before I start, this is what I have in the following directories (this is different to what I sent yesterday because tonight it's before I start):

Code:

root@s1:~# ls -la /etc/letsencrypt/live/s1.snowweb.info/
total 12
drwxr-xr-x  2 root root 4096 Apr 17 02:56 .
drwx------ 21 root root 4096 Oct  1  2020 ..
lrwxrwxrwx  1 root root   39 Apr 17 02:56 cert.pem -> ../../archive/s1.snowweb.info/cert5.pem
lrwxrwxrwx  1 root root   40 Apr 17 02:56 chain.pem -> ../../archive/s1.snowweb.info/chain5.pem
lrwxrwxrwx  1 root root   44 Apr 17 02:56 fullchain.pem -> ../../archive/s1.snowweb.info/fullchain5.pem
lrwxrwxrwx  1 root root   42 Apr 17 02:56 privkey.pem -> ../../archive/s1.snowweb.info/privkey5.pem
-rw-r--r--  1 root root  682 Aug 19  2020 README

Code:

root@s1:~# ls -la /etc/letsencrypt/archive/s1.snowweb.info/
total 88
drwxr-xr-x  2 root root 4096 Apr 17 02:56 .
drwx------ 21 root root 4096 Oct  1  2020 ..
-rw-r--r--  1 root root 2285 Aug 19  2020 cert1.pem
-rw-r--r--  1 root root 2289 Oct 18  2020 cert2.pem
-rw-r--r--  1 root root 2220 Dec 17 17:39 cert3.pem
-rw-r--r--  1 root root 2216 Feb 15 19:19 cert4.pem
-rw-r--r--  1 root root 2216 Apr 17 02:56 cert5.pem
-rw-r--r--  1 root root 1647 Aug 19  2020 chain1.pem
-rw-r--r--  1 root root 1647 Oct 18  2020 chain2.pem
-rw-r--r--  1 root root 1586 Dec 17 17:39 chain3.pem
-rw-r--r--  1 root root 1586 Feb 15 19:19 chain4.pem
-rw-r--r--  1 root root 1586 Apr 17 02:56 chain5.pem
-rw-r--r--  1 root root 3932 Aug 19  2020 fullchain1.pem
-rw-r--r--  1 root root 3936 Oct 18  2020 fullchain2.pem
-rw-r--r--  1 root root 3806 Dec 17 17:39 fullchain3.pem
-rw-r--r--  1 root root 3802 Feb 15 19:19 fullchain4.pem
-rw-r--r--  1 root root 3802 Apr 17 02:56 fullchain5.pem
-rw-r--r--  1 root root 3272 Aug 19  2020 privkey1.pem
-rw-r--r--  1 root root 3272 Oct 18  2020 privkey2.pem
-rw-r--r--  1 root root 3272 Dec 17 17:39 privkey3.pem
-rw-r--r--  1 root root 3272 Feb 15 19:19 privkey4.pem
-rw-r--r--  1 root root 3272 Apr 17 02:56 privkey5.pem

Code:

root@s1:~# ls -la /usr/local/ispconfig/interface/ssl/
total 68
drwxr-x--- 2 root      root      4096 Jun  4 23:37 .
drwxr-x--- 9 ispconfig ispconfig 4096 Aug 14  2020 ..
-rwxr-x--- 1 root      root        45 Aug 14  2020 empty.dir
lrwxrwxrwx 1 root      root        51 Aug 19  2020 ispserver.crt -> /etc/letsencrypt/live/s1.snowweb.info/fullchain.pem
-rwxr-x--- 1 root      root      2017 Aug 14  2020 ispserver.crt-200819111429.bak
-rwxr-x--- 1 root      root      1687 Aug 14  2020 ispserver.csr
lrwxrwxrwx 1 root      root        49 Aug 19  2020 ispserver.key -> /etc/letsencrypt/live/s1.snowweb.info/privkey.pem
-rwxr-x--- 1 root      root      3247 Aug 14  2020 ispserver.key-200819111446.bak
-rwxr-x--- 1 root      root      3311 Aug 14  2020 ispserver.key.secure
-rw------- 1 root      root      7074 Apr 17 02:56 ispserver.pem
-rw------- 1 root      root      7204 Aug 19  2020 ispserver.pem-201117131402.bak
-rw------- 1 root      root      7208 Nov 17  2020 ispserver.pem-210116160626.bak
-rw------- 1 root      root      7078 Jan 16 16:06 ispserver.pem-210317183053.bak
-rw------- 1 root      root      7074 Mar 17 18:30 ispserver.pem-210417025626.bak

So we're planning to do (before we commence ISPConfig update):
rm /etc/postfix/smtpd.cert
rm /etc/postfix/smtpd.key

using nano /etc/dovecot/dovecot.conf
remove:
ssl_cert = </etc/postfix/smtpd.cert
ssl_key = </etc/postfix/smtpd.key

rm /etc/ssl/private/pure-ftpd.pem

rm /usr/local/ispconfig/interface/ssl/*
cp /etc/letsencrypt/archive/s1.snowweb.info/cert5.pem /usr/local/ispconfig/interface/ssl/cert.pem
cp /etc/letsencrypt/archive/s1.snowweb.info/chain5.pem /usr/local/ispconfig/interface/ssl/chain.pem
cp /etc/letsencrypt/archive/s1.snowweb.info/fullchain5.pem /usr/local/ispconfig/interface/ssl/ispserver.crt
cp /etc/letsencrypt/archive/s1.snowweb.info/privkey5.pem /usr/local/ispconfig/interface/ssl/privkey.pem

Please can you advise what command I need to issue to:

Thanks

 

After another four more attempts to update ISPConfig from 3.1 to 3.2 we have rolled back to before we began.

It seems that one time, we got the s1.snowweb.info config right but then apache failed to start because it had an issue with the next domain. We tried removing all other .vhost files from sites-enabled but then the apache still wouldn't start due to an issue with the certificate. If apache won't start, then it can't validate the domain to issue the certificate and if it can't issue the certificate, you can't start apache.

Last September we did a full server migration from another hosting company and control panel and installed ISPConfig, thinking that we would not need to do another migration for at least 5 years, if ever, since there was an upgrade path. But it seems we were mistaken and there is no way for us to upgrade. We have to do another full migration just nine months later.

It's possible that our problems started on the day we installed ISPConfig, since according to our bash_history file, the following commands were used, which were the official procedure at that time were as follows:

Code:

wget -O ispconfig.tar.gz https://git.ispconfig.org/ispconfig/ispconfig3/repository/archive.tar.gz?ref=stable-3.1
tar xfz ispconfig.tar.gz
cd ispconfig3*/install/
php -q install.php

We were fully expecting that the resulting installation would be a "stable" release, but somehow, when we checked after the migration was complete, we had ended up with "3.1dev". Maybe this is part of our problem and perhaps why local websites are unable to send mail ever since.

The other contributing factor (and most likely) is that we used a method endorsed here for securing the web sites with SSL. A great tutorial written by ahrasis, but unfortunately the advise on the forum when migrating from 3.1 to 3.2, is that what was accomplished in that procedure, has to be reverted in order for the upgrade to succeed (unless you want to forever manage your certificates manually) and there is no tested procedure that we could find to accomplish this. If there is, we could still use it. This is likely the biggest or possibly the entire reason why we're unable to upgrade.

Anyway, we want to thank Till, ahrasis and Jesse for all your tireless inputs and your patience.

Sabbath today. Will sleep now and will plan next steps on Sunday. Thanks again.

 

I agree with @Th0m.

In any event, ISPConfig development version is never a problem and one should be able to change from development to stable quite easily in most cases.

The problem you have is undoing the previous setup in securing your ISPConfig server but once you settled this, you should be able to update your server to latest version of ISPConfig 3.2 and secure it during the same process.

It is true that undoing the same will break the apache web server as in my script it removes the LE certs for the server but the ISPConfig 3.2 update should fix it by choosing to create SSL during the process. The step here is very crucial where you need to make sure that creating the new LE certs for your server is successful, or otherwise apache will fail to restart and the server will not be secured by LE certs.

The root of my reasoning to delete the old LE certs for the server via my script was that the LE certs renewal conf file will not be changed and thus some ISPConfig new scripts will not be added to it if it is not deleted. At the same time, creation or renewal of new LE certs would failed without it, so the whole LE certs folder for the server need to be deleted.

I already advised about the LE limit for creating certs for your server FQDN, that might be hit if you are not careful, so always check the LE logs if there is failure to create the new certs and determine what is its true cause(s). Once you fixed that problem(s), you may try ISPConfig update and choose creating new SSL, again, but do watch for the limit.

Only when you have a good running Ubuntu 18.04 server with ISPConfig 3.2 that you should proceed to upgrade to Ubuntu 20.04 and the steps for this have been discussed very much in details above including making sure your server follow the ISPConfig PST for Ubuntu 20.04 and then run ISPConfig update again thereafter.

I wish you the best of luck for tomorrow.