Sorry. Just realized we could check that! No. There is nothing in those directories (except empty.dir)
Since the upgrade (which we'll need to roll-back shortly), the mail clients are failing to accept the new certificate, saying it's untrusted.
Please post the result of: ls -la /var/www/clients/client2/web2/ssl/ From which exact ISPConfig version did you upgrade? Do you have an ispserver.bundle file in /usr/local/ispconfig/interface/ssl/ that contains the SSL chain certificate?
ls -la /var/www/clients/client2/web2/ssl/ Code: root@s1:~# ls -la /var/www/clients/client2/web2/ssl/ total 92 drwxr-xr-x 2 root root 4096 Sep 13 2020 . drwxr-xr-x 11 web2 client2 4096 Aug 20 2020 .. lrwxrwxrwx 1 root root 47 Sep 13 2020 s1.snowweb.info-le.bundle -> /etc/letsencrypt/live/s1.snowweb.info/chain.pem -r-------- 1 root root 1647 Aug 23 2020 s1.snowweb.info-le.bundle.old.20200823173906 -r-------- 1 root root 1647 Aug 25 2020 s1.snowweb.info-le.bundle.old.20200825124410 -r-------- 1 root root 1647 Aug 26 2020 s1.snowweb.info-le.bundle.old.20200826194315 -r-------- 1 root root 1647 Aug 26 2020 s1.snowweb.info-le.bundle.old.20200826194610 -r-------- 1 root root 1647 Aug 27 2020 s1.snowweb.info-le.bundle.old.20200827150634 -r-------- 1 root root 1647 Sep 13 2020 s1.snowweb.info-le.bundle.old.20200913204505 lrwxrwxrwx 1 root root 51 Sep 13 2020 s1.snowweb.info-le.crt -> /etc/letsencrypt/live/s1.snowweb.info/fullchain.pem -r-------- 1 root root 3932 Aug 23 2020 s1.snowweb.info-le.crt.old.20200823173906 -r-------- 1 root root 3932 Aug 25 2020 s1.snowweb.info-le.crt.old.20200825124410 -r-------- 1 root root 3932 Aug 26 2020 s1.snowweb.info-le.crt.old.20200826194315 -r-------- 1 root root 3903 Aug 26 2020 s1.snowweb.info-le.crt.old.20200826194610 -r-------- 1 root root 3903 Aug 27 2020 s1.snowweb.info-le.crt.old.20200827150634 -r-------- 1 root root 3903 Sep 13 2020 s1.snowweb.info-le.crt.old.20200913204505 lrwxrwxrwx 1 root root 49 Sep 13 2020 s1.snowweb.info-le.key -> /etc/letsencrypt/live/s1.snowweb.info/privkey.pem -r-------- 1 root root 3272 Aug 23 2020 s1.snowweb.info-le.key.old.20200823173906 -r-------- 1 root root 3272 Aug 25 2020 s1.snowweb.info-le.key.old.20200825124410 -r-------- 1 root root 3272 Aug 26 2020 s1.snowweb.info-le.key.old.20200826194315 -r-------- 1 root root 3272 Aug 26 2020 s1.snowweb.info-le.key.old.20200826194610 -r-------- 1 root root 3272 Aug 27 2020 s1.snowweb.info-le.key.old.20200827150634 -r-------- 1 root root 3272 Sep 13 2020 s1.snowweb.info-le.key.old.20200913204505 lrwxrwxrwx 1 root root 51 Sep 13 2020 www.s1.snowweb.info-le.bundle -> /etc/letsencrypt/live/www.s1.snowweb.info/chain.pem -r-------- 1 root root 1647 Sep 13 2020 www.s1.snowweb.info-le.bundle.old.20200913204404 lrwxrwxrwx 1 root root 55 Sep 13 2020 www.s1.snowweb.info-le.crt -> /etc/letsencrypt/live/www.s1.snowweb.info/fullchain.pem -r-------- 1 root root 3916 Sep 13 2020 www.s1.snowweb.info-le.crt.old.20200913204404 lrwxrwxrwx 1 root root 53 Sep 13 2020 www.s1.snowweb.info-le.key -> /etc/letsencrypt/live/www.s1.snowweb.info/privkey.pem -r-------- 1 root root 3272 Sep 13 2020 www.s1.snowweb.info-le.key.old.20200913204404 ls -la /usr/local/ispconfig/interface/ssl/ Code: root@s1:~# ls -la /usr/local/ispconfig/interface/ssl/ total 36 drwxr-x--- 2 root root 4096 Jun 4 00:20 . drwxr-x--- 10 ispconfig ispconfig 4096 Jun 4 00:14 .. -rwxr-x--- 1 root root 45 Jun 4 00:21 empty.dir -rwxr-x--- 1 root root 2078 Jun 4 00:20 ispserver.crt -rwxr-x--- 1 root root 1720 Jun 4 00:20 ispserver.csr -rwxr-x--- 1 root root 3247 Jun 4 00:20 ispserver.key -rwxr-x--- 1 root root 3311 Jun 4 00:19 ispserver.key.secure -rwxr-x--- 1 root root 5325 Jun 4 00:20 ispserver.pem Not sure why is says "total 36"? I only count 8 entries. Thanks Till
What do you see from 'ls -l /etc/letsencrypt/live/s1.snowweb.info/' and 'ls -tlr /etc/letsencrypt/archive/s1.snowweb.info/' ?
Code: root@s1:~# ls -l /etc/letsencrypt/live/s1.snowweb.info/ ls: cannot access '/etc/letsencrypt/live/s1.snowweb.info/': No such file or directory Code: root@s1:~# ls -tlr /etc/letsencrypt/archive/s1.snowweb.info/ ls: cannot access '/etc/letsencrypt/archive/s1.snowweb.info/': No such file or directory
Just noticed this though... Code: root@s1:~# ls -l /etc/letsencrypt/live/ total 68 drwxr-xr-x 2 root root 4096 Apr 29 03:00 audio4design.com drwxr-xr-x 2 root root 4096 May 6 03:00 cepher-bible-dist.info drwxr-xr-x 2 root root 4096 May 1 03:00 rjen.net drwxr-xr-x 2 root root 4096 Apr 19 02:25 snowkids.fun drwxr-xr-x 2 root root 4096 Apr 21 03:00 snowkids.fun-0001 drwxr-xr-x 2 root root 4096 Apr 25 03:00 snowweb.net drwxr-xr-x 2 root root 4096 Apr 25 03:00 snowweb.net-0001 drwxr-xr-x 2 root root 4096 Apr 29 03:00 webmail.audio4design.com drwxr-xr-x 2 root root 4096 Apr 30 03:00 webmail.dentaltravel.com.ph drwxr-xr-x 2 root root 4096 Apr 30 03:00 webmail.oralconfidence.com drwxr-xr-x 2 root root 4096 May 1 03:01 webmail.rjen.net drwxr-xr-x 2 root root 4096 Apr 21 03:00 webmail.snowkids.fun drwxr-xr-x 2 root root 4096 Aug 27 2020 webmail.snowweb.net drwxr-xr-x 2 root root 4096 Apr 24 02:32 webmail.snowweb.net-0001 drwxr-xr-x 2 root root 4096 May 30 02:54 webmail.zconnect.ph drwxr-xr-x 2 root root 4096 May 9 03:00 www.s1.snowweb.info drwxr-xr-x 2 root root 4096 May 30 02:54 zconnect.ph Note the www.s1.snowweb.info near the bottom.
Code: root@s1:~# ls -l /etc/letsencrypt/live/www.s1.snowweb.info/ total 4 lrwxrwxrwx 1 root root 43 May 9 03:00 cert.pem -> ../../archive/www.s1.snowweb.info/cert5.pem lrwxrwxrwx 1 root root 44 May 9 03:00 chain.pem -> ../../archive/www.s1.snowweb.info/chain5.pem lrwxrwxrwx 1 root root 48 May 9 03:00 fullchain.pem -> ../../archive/www.s1.snowweb.info/fullchain5.pem lrwxrwxrwx 1 root root 46 May 9 03:00 privkey.pem -> ../../archive/www.s1.snowweb.info/privkey5.pem -rw-r--r-- 1 root root 682 Sep 10 2020 README
Code: root@s1:~# ls -tlr /etc/letsencrypt/archive/www.s1.snowweb.info/ total 84 -rw-r--r-- 1 root root 2269 Sep 10 2020 cert1.pem -rw-r--r-- 1 root root 3272 Sep 10 2020 privkey1.pem -rw-r--r-- 1 root root 3916 Sep 10 2020 fullchain1.pem -rw-r--r-- 1 root root 1647 Sep 10 2020 chain1.pem -rw-r--r-- 1 root root 3272 Nov 10 2020 privkey2.pem -rw-r--r-- 1 root root 3920 Nov 10 2020 fullchain2.pem -rw-r--r-- 1 root root 1647 Nov 10 2020 chain2.pem -rw-r--r-- 1 root root 2273 Nov 10 2020 cert2.pem -rw-r--r-- 1 root root 3272 Jan 9 03:00 privkey3.pem -rw-r--r-- 1 root root 3785 Jan 9 03:00 fullchain3.pem -rw-r--r-- 1 root root 1586 Jan 9 03:00 chain3.pem -rw-r--r-- 1 root root 2199 Jan 9 03:00 cert3.pem -rw-r--r-- 1 root root 3272 Mar 10 03:00 privkey4.pem -rw-r--r-- 1 root root 3790 Mar 10 03:00 fullchain4.pem -rw-r--r-- 1 root root 1586 Mar 10 03:00 chain4.pem -rw-r--r-- 1 root root 2204 Mar 10 03:00 cert4.pem -rw-r--r-- 1 root root 3272 May 9 03:00 privkey5.pem -rw-r--r-- 1 root root 5949 May 9 03:00 fullchain5.pem -rw-r--r-- 1 root root 3750 May 9 03:00 chain5.pem -rw-r--r-- 1 root root 2199 May 9 03:00 cert5.pem Sorry, realized you probably want that!
So your vhost file specifies a certificate file: Code: SSLCertificateFile /var/www/clients/client2/web2/ssl/s1.snowweb.info-le.crt Which is a symlink: Code: s1.snowweb.info-le.crt -> /etc/letsencrypt/live/s1.snowweb.info/fullchain.pem To a file in a directory which does not exist: Code: ls: cannot access '/etc/letsencrypt/live/s1.snowweb.info/': No such file or directory Presumably this directory does exist before you started your updates? Removing that would not be due to upgrading ISPConfig; I don't expext it comes from upgrading Ubuntu (eg. certbot package) either. Your earlier steps mentioned this, which is probably the cause:
I think you've hit the bulls-eye there Jesse. Well spotted. So I was attempting to undo the steps of the tutorial Securing ISPConfig 3.1 With a Free Let's Encrypt SSL Certificate when I deleted the directory but I guess it's not right in this instance. The question is, do I just completely omit the command I used or do I just need to be more specific regarding what it should delete (I'm not sure what that is either!)? Thanks Till and Jesse. We really appreciate your patience and support. We're planning to give this another try tonight (8 hours time) but we're thinking that since support for 18.04 is built-in for 3.2, we will just update ISPConfig tonight and will let the dust settle for a few days before going for the OS upgrade.
I would just leave out that command, I guess you can even leave the ahrasis setup in place as it is, just don't choose to create a new SSL cert during ISPConfig update.
I'm really determined to try to get a manageable server this time round (one where things don't just suddenly stop working) and I think I need ISPConfig to manage the SSL renewal situation for best chance for that to happen. I also want to end up with a pretty standard setup which is easier to get support for if needed and easier to upgrade in the future (full server migrations are such a hassle), so do forgive me, but I'm going to try to press to get this working. Thanks for understanding. I'll try just not deleting that directory and hopefully the setup will still create the certificates and link them up correctly. I already un-did the rest of ahrasis setup.
This will most likely fail, as the ispconfig certs are symlinks in /usr/local/ispconfig/interface/ssl/ and these symlinks point to a non existent location then. What you might do is this: Replace the symlinks in /usr/local/ispconfig/interface/ssl/ with files that contain the certs from /etc/letsencrypt/live/s1.snowweb.info/ folder before you delete the le folder (use certbot delete command to safely remove the cert), so basically you mimic that these certs are no LE certs by making a copy of them instead of symlinking to LE. Then you can let the ispconfig installer create new LE based certs during update, which will replace the copies you made earlier with symlinks to a new LE cert.
I am not sure what you did which causing the failures but if you run ISPConfig update and choose to create SSL for your server during the update process, even if you have deleted the LE certs before that update, that should in theory create new certs for the server and I do see that in your list. The only problem is if you run that delete command again after the update deliberately or by mistake. Do note also that the creation of the certs per that server FQDN has limit and you might hit that limit, so do check your logs if your LE certs creation failed, in case you are trying to create the new ones.
Before I start, this is what I have in the following directories (this is different to what I sent yesterday because tonight it's before I start): Code: root@s1:~# ls -la /etc/letsencrypt/live/s1.snowweb.info/ total 12 drwxr-xr-x 2 root root 4096 Apr 17 02:56 . drwx------ 21 root root 4096 Oct 1 2020 .. lrwxrwxrwx 1 root root 39 Apr 17 02:56 cert.pem -> ../../archive/s1.snowweb.info/cert5.pem lrwxrwxrwx 1 root root 40 Apr 17 02:56 chain.pem -> ../../archive/s1.snowweb.info/chain5.pem lrwxrwxrwx 1 root root 44 Apr 17 02:56 fullchain.pem -> ../../archive/s1.snowweb.info/fullchain5.pem lrwxrwxrwx 1 root root 42 Apr 17 02:56 privkey.pem -> ../../archive/s1.snowweb.info/privkey5.pem -rw-r--r-- 1 root root 682 Aug 19 2020 README Code: root@s1:~# ls -la /etc/letsencrypt/archive/s1.snowweb.info/ total 88 drwxr-xr-x 2 root root 4096 Apr 17 02:56 . drwx------ 21 root root 4096 Oct 1 2020 .. -rw-r--r-- 1 root root 2285 Aug 19 2020 cert1.pem -rw-r--r-- 1 root root 2289 Oct 18 2020 cert2.pem -rw-r--r-- 1 root root 2220 Dec 17 17:39 cert3.pem -rw-r--r-- 1 root root 2216 Feb 15 19:19 cert4.pem -rw-r--r-- 1 root root 2216 Apr 17 02:56 cert5.pem -rw-r--r-- 1 root root 1647 Aug 19 2020 chain1.pem -rw-r--r-- 1 root root 1647 Oct 18 2020 chain2.pem -rw-r--r-- 1 root root 1586 Dec 17 17:39 chain3.pem -rw-r--r-- 1 root root 1586 Feb 15 19:19 chain4.pem -rw-r--r-- 1 root root 1586 Apr 17 02:56 chain5.pem -rw-r--r-- 1 root root 3932 Aug 19 2020 fullchain1.pem -rw-r--r-- 1 root root 3936 Oct 18 2020 fullchain2.pem -rw-r--r-- 1 root root 3806 Dec 17 17:39 fullchain3.pem -rw-r--r-- 1 root root 3802 Feb 15 19:19 fullchain4.pem -rw-r--r-- 1 root root 3802 Apr 17 02:56 fullchain5.pem -rw-r--r-- 1 root root 3272 Aug 19 2020 privkey1.pem -rw-r--r-- 1 root root 3272 Oct 18 2020 privkey2.pem -rw-r--r-- 1 root root 3272 Dec 17 17:39 privkey3.pem -rw-r--r-- 1 root root 3272 Feb 15 19:19 privkey4.pem -rw-r--r-- 1 root root 3272 Apr 17 02:56 privkey5.pem Code: root@s1:~# ls -la /usr/local/ispconfig/interface/ssl/ total 68 drwxr-x--- 2 root root 4096 Jun 4 23:37 . drwxr-x--- 9 ispconfig ispconfig 4096 Aug 14 2020 .. -rwxr-x--- 1 root root 45 Aug 14 2020 empty.dir lrwxrwxrwx 1 root root 51 Aug 19 2020 ispserver.crt -> /etc/letsencrypt/live/s1.snowweb.info/fullchain.pem -rwxr-x--- 1 root root 2017 Aug 14 2020 ispserver.crt-200819111429.bak -rwxr-x--- 1 root root 1687 Aug 14 2020 ispserver.csr lrwxrwxrwx 1 root root 49 Aug 19 2020 ispserver.key -> /etc/letsencrypt/live/s1.snowweb.info/privkey.pem -rwxr-x--- 1 root root 3247 Aug 14 2020 ispserver.key-200819111446.bak -rwxr-x--- 1 root root 3311 Aug 14 2020 ispserver.key.secure -rw------- 1 root root 7074 Apr 17 02:56 ispserver.pem -rw------- 1 root root 7204 Aug 19 2020 ispserver.pem-201117131402.bak -rw------- 1 root root 7208 Nov 17 2020 ispserver.pem-210116160626.bak -rw------- 1 root root 7078 Jan 16 16:06 ispserver.pem-210317183053.bak -rw------- 1 root root 7074 Mar 17 18:30 ispserver.pem-210417025626.bak So we're planning to do (before we commence ISPConfig update): rm /etc/postfix/smtpd.cert rm /etc/postfix/smtpd.key using nano /etc/dovecot/dovecot.conf remove: ssl_cert = </etc/postfix/smtpd.cert ssl_key = </etc/postfix/smtpd.key rm /etc/ssl/private/pure-ftpd.pem rm /usr/local/ispconfig/interface/ssl/* cp /etc/letsencrypt/archive/s1.snowweb.info/cert5.pem /usr/local/ispconfig/interface/ssl/cert.pem cp /etc/letsencrypt/archive/s1.snowweb.info/chain5.pem /usr/local/ispconfig/interface/ssl/chain.pem cp /etc/letsencrypt/archive/s1.snowweb.info/fullchain5.pem /usr/local/ispconfig/interface/ssl/ispserver.crt cp /etc/letsencrypt/archive/s1.snowweb.info/privkey5.pem /usr/local/ispconfig/interface/ssl/privkey.pem Please can you advise what command I need to issue to: Thanks
After another four more attempts to update ISPConfig from 3.1 to 3.2 we have rolled back to before we began. It seems that one time, we got the s1.snowweb.info config right but then apache failed to start because it had an issue with the next domain. We tried removing all other .vhost files from sites-enabled but then the apache still wouldn't start due to an issue with the certificate. If apache won't start, then it can't validate the domain to issue the certificate and if it can't issue the certificate, you can't start apache. Last September we did a full server migration from another hosting company and control panel and installed ISPConfig, thinking that we would not need to do another migration for at least 5 years, if ever, since there was an upgrade path. But it seems we were mistaken and there is no way for us to upgrade. We have to do another full migration just nine months later. It's possible that our problems started on the day we installed ISPConfig, since according to our bash_history file, the following commands were used, which were the official procedure at that time were as follows: Code: wget -O ispconfig.tar.gz https://git.ispconfig.org/ispconfig/ispconfig3/repository/archive.tar.gz?ref=stable-3.1 tar xfz ispconfig.tar.gz cd ispconfig3*/install/ php -q install.php We were fully expecting that the resulting installation would be a "stable" release, but somehow, when we checked after the migration was complete, we had ended up with "3.1dev". Maybe this is part of our problem and perhaps why local websites are unable to send mail ever since. The other contributing factor (and most likely) is that we used a method endorsed here for securing the web sites with SSL. A great tutorial written by ahrasis, but unfortunately the advise on the forum when migrating from 3.1 to 3.2, is that what was accomplished in that procedure, has to be reverted in order for the upgrade to succeed (unless you want to forever manage your certificates manually) and there is no tested procedure that we could find to accomplish this. If there is, we could still use it. This is likely the biggest or possibly the entire reason why we're unable to upgrade. Anyway, we want to thank Till, ahrasis and Jesse for all your tireless inputs and your patience. Sabbath today. Will sleep now and will plan next steps on Sunday. Thanks again.
I wan to start this with one advice. Don't freak out when something goes wrong and roll back, but look into a fix. Those rollbacks can cause problems and you should be able to fix such issues. You are currently installing a dev version indeed, follow the Perfect server guide for your OS to use the correct version. The "old" method for securing your panel with a SSL cert can be used with 3.2 if wanted/necessary. It might be useful for you to hire someone to discuss this with directly who can solve issues for you. Especially if you are a hosting company with quite some clients, I suspect the cost is relatively low compared to the headaches it will prevent
I agree with @Th0m. In any event, ISPConfig development version is never a problem and one should be able to change from development to stable quite easily in most cases. The problem you have is undoing the previous setup in securing your ISPConfig server but once you settled this, you should be able to update your server to latest version of ISPConfig 3.2 and secure it during the same process. It is true that undoing the same will break the apache web server as in my script it removes the LE certs for the server but the ISPConfig 3.2 update should fix it by choosing to create SSL during the process. The step here is very crucial where you need to make sure that creating the new LE certs for your server is successful, or otherwise apache will fail to restart and the server will not be secured by LE certs. The root of my reasoning to delete the old LE certs for the server via my script was that the LE certs renewal conf file will not be changed and thus some ISPConfig new scripts will not be added to it if it is not deleted. At the same time, creation or renewal of new LE certs would failed without it, so the whole LE certs folder for the server need to be deleted. I already advised about the LE limit for creating certs for your server FQDN, that might be hit if you are not careful, so always check the LE logs if there is failure to create the new certs and determine what is its true cause(s). Once you fixed that problem(s), you may try ISPConfig update and choose creating new SSL, again, but do watch for the limit. Only when you have a good running Ubuntu 18.04 server with ISPConfig 3.2 that you should proceed to upgrade to Ubuntu 20.04 and the steps for this have been discussed very much in details above including making sure your server follow the ISPConfig PST for Ubuntu 20.04 and then run ISPConfig update again thereafter. I wish you the best of luck for tomorrow.