certbot -> acme.sh

Hi, piping in late, but I just wanted to say that replacing certbot with acme.sh was a nightmare!
I have been upgrading ISPConfig for years now and had no idea that acme.sh was supported at all. It's not obvious at all that 'replacing the SSL certificate' for the ISPConfig virtual host will also switch it from certbot to acme.sh (and possibly vice-versa). In fact, if it weren't for this thread, I would have never figured that out. It really needs to be explained more fully, starting with the message displayed by ispconfig_update.sh, and then moving on to the tutorials as well...
Ok, I know, I'm just complaining because it has been a tough experience for me. Here's the kind of trouble I've encountered: the first thing was to uninstall certbot; of course I kept a backup (since I had no idea if ispconfig_update.sh would do it for me or not).
Then I installed acme.sh, hoping to get the configuration right: with luck, I could get it running from /etc/letsencrypt and it would 'reuse' the same certificate files. That wasn't the case
Then I basically reinstalled ISPConfig over the existing installation (using --force, as suggested), while being careful to answer 'yes' to 'replacing the SSL certificate'. This actually seemed to detect that certbot had left some residual/vestigial configuration (because it apparently created a new backup, I think) but went on to integrate acme.sh with ISPConfig3.
So far, I was hopeful that this would actually work! Not by far. The first thing that ISPConfig3 needs to have is a valid SSL certificate for its own virtual host. At that stage, I was unlucky: it seems that I had a custom vhost template that was out of date, and which didn't redirect properly to the ACME directory. Although there was a perfectly valid certificate on disk, obviously it wasn't on the right path; thus, acme.sh had to request a new one, but, because the vhost template was outdated (I thought that the host template for the ISPConfig backend was separate from the others... apparently not... or maybe at this stage the installer still hadn't changed the templates?... I don't know), Let's Encrypt couldn't validate that certificate, and ISPConfig3 proceeded to create a self-signed certificate... which I didn't want to.
Ok, so I aborted the installation and tried to get a new certificate via acme.sh — this time, using DNS as the validation mechanism — and now I knew where it was supposed to be installed. It was also the perfect occasion to test out the famous Cloudflare integration!
No such luck, though. Something goes wrong when I use my key & email, and acme.sh does not change anything at Cloudflare whatsoever, thus popping up the error that it couldn't find a TXT record on the DNS domain which I'm using... all right, time to do it manually. acme.sh has this humorous switch called --yes-I-know-dns-manual-mode-enough-go-ahead-please which actually makes it behave in the expected way: it starts the whole process, then aborts telling me what should be the content of the TXT record for proper validation, I go over to Cloudflare to promptly add it, and run acme.sh again with the --renew option, as suggested by the script. And, indeed, this worked! For good measure, since I install ISPConfig3 on a subdomain, I thought it would be better to get a certificate for the domain itself as well — going the same route, i.e. manual validation. Success, once again!
Now it was just a question of doing a lot of symbolic links — since the old templates (not yet overwritten!) still pointed to certbot's certificate layout, and acme.sh has different names and paths for everything... But at last that meant I could get the site up and running, with valid SSL certificates, and the installation could proceed. There was a nice touch of automatically placing the same certificate on Postfix (yay!) — not a moment too sooner, as I was starting to accumulate error messages from multiple mail servers complaining about the 'wrong' certificate and refusing to deliver mail...
Reloading all the services is something that takes a lot of time (I have a huge database for a particular application which is not managed via ISPConifg3... and shutting MariaDB down takes æons), but, eventually, I had everything up again — just all the SSL certificates on the 'wrong' place, that is.
As suggested, I next proceeded to do a 'resync' for all websites, hoping to get ISPConfig3 checking for certificates for each one of them, and storing them in the appropriate places. And, indeed, this is what ISPConfig3 tried to do. Unfortunately, it was still using my old, outdated custom template... and that meant that none of the domains could be validated via acme.sh! Arrrgh!
With all sites down — and ISPConfig3 painfully plodding along each one of them and trying to get new certificates, and failing... — I then went to the custom template, compared it to the current one, and made all necessary changes, saved it... and crossed my fingers. This seemed to have fixed things... because ISPConfig3 was still going through the list of domains, and now they were being correctly validated. There was probably an issue with the first domains on the list, which I forced a resync by going to the ISPConfig3 and checking the Let's Encrypt checkbox. Fortunately, there weren't many; fortunately, this eventually worked (resync, as explained elsewhere, can take a lot of time to finish); and as far as I can see, ISPConfig3 managed to change all certificates correctly.
Whew. There were a few glitches with one or two domains — they had unusual configurations and were to be retired anyway — but it was reasonably easy to get rid of them (this affected the certificates issued for them; they needed to be re-issued, in a different way; but acme.sh seemed to be able to handle all of that). Things seem to be working well now. But you can imagine my state of panic with all sites down for an extended period of time and no 'easy' solution at hand! (well, except restoring everything from the backups, but I'm a stubborn girl and hate to 'go back').

 

So, what could be done to improve this terrible experience of mine?

1. Inform people somewhere that they now have an option to abandon certbot and get acme.sh instead. Although it has been so often claimed that the tutorials already say that, I seem to be particularly unlucky with my searches; the only tutorial I found mentioning acme.sh was https://www.howtoforge.com/getting-started-with-acmesh-lets-encrypt-client/ — useful by itself, but it assumes a stable, finished server configuration, without any previous certbot installation.
2. As far as I could search, Ubuntu 20.04.X does not include acme.sh in any of its many packages (it has several alternatives to certbot, though), meaning that there is no other choice but to install it manually, as per the tutorial mentioned above. However, the 'correct' options are far from obvious, especially if you're used to doing backups from the 'standard' directories. I know that there is a current trend of installing all these nifty tools simply somewhere inside root's home directory, but I'm a traditionalist: root's home directory, for me, is supposed to be empty, and not be used for installed applications. There are several good reasons why the Un*x filesystem 'standard' has been around for (literally) 50 years: it allows system administrators to limit certain directories only for running applications, others for containing data, and optimise accordingly (e.g. placing all ../bin directories in a read-only partition). Again, this is not really ISPConfig3's fault; the acme.sh developers just wanted to avoid having to deal with an unknown filesystem organisation (for instance, you'll never know how the next macOS organisation will be... Apple is always shuffling things around in a non-standard way... and this also applies, to a degree, to many embedded Linux systems I've toyed with, including my home NAS). Nevertheless, ISPConfig3 could configure everything related to acme.sh inside its own directories (e.g. /usr/local/ispconfig/), as it does for so many other things, eventually adding a few symlinks to /etc/letsencrypt. In other words: there should be a mechanism for installing non-standard applications that are directly managed/used by ISPConfig3 and which are not packaged by the many distros in a uniform way — that's the Principle of Least Surprise, which I love, and which has been deeply engraved in the folds of my poor brain back in those university classes dealing with good programming practices (disclaimer: at that time, object-oriented programming was a new thing — aye, that tells you something about my age! — so I can imagine that things are being handled very differently today...).
3. There should be a way to engage acme.sh's fallback ability and its 'manual mode' at least for the ISPConfig3 vhost. Once that is fixed, Postfix will work as well (if using the same certificate), and all the remaining steps in ispconfig_update.sh will complete successfully. While a reasonable compromise is to generate a self-signed certificate for the ISPConfig3 vhost, it should be reasonably simple to automatically revert it at a later stage to a valid certificate. I'm well aware that there are many tutorials out there which explain how to do it manually.
4. A pet peeve of mine: certbot seemed to drop the file certbot.timer under /etc/systemd/system/timers.target.wants, as expected from a systemd configuration. I'm not going to enter the discussion about systemd versus /etc/init versus any other way of organising things — I might not be exactly very happy with the way systemd works, but slowly every distro (even for embedded Linux systems!) is 'migrating' to systemd (well, except for Apple...), so I guess we'll be stuck with it for a while... My point is just that acme.sh adds a line to crontab -l instead. Again, to comply with the Principle of Least Surprise, I think that this ought to be moved to the correct systemd directory... I'm not even sure what added that line on crontab —was it the acme.sh automatic configuration, or ISPConfig3 when doing the overall installation? I haven't checked, but since ISPConfig3 has its own job queue mechanism (and there is nothing wrong with that!), I'd assume that the non-standard configuration comes from acme.sh instead. It's not a question of being 'very easy' to move the corn job to its proper place; it's just that, when new versions of either ISPConfig3 or acme.sh come out, they'll very likely break such a configuration. One of the many, many, many advantages of ISPConfig3 is its predictably — i.e. we know where things are supposed to be (and that's one reason why I could so quickly locate the problem in my own buggy custom vhost configuration template — because basically I knew that it couldn't be anywhere else).

Alas, I know I've bothered you all way too much. Hopefully, though, others who read this thread may get a glimpse of what they're going to face when migrating from certbot to acme.sh. I'm well aware that there is no official 'migration tool' for that (and probably never will be!), but it would be nice to have it at least properly sketched out on the tutorials, somewhere...
Cheers, have a wonderful Valentine's Day!

 

There is, if I want to use ISPConfig Autoinstaller to install a new setup and use Migration Tool to transfer data from the old. If the SOURCE system uses Certbot, I can not use Autoinstaller since as far as I can see it does not support installing Certbot. So I would need some way to move from certbot to acme.sh. If there is no way to do that, I can not use Autoinstaller.

 

Sorry for the bad experience you had. I can feel with you, I did that a year ago: https://www.howtoforge.com/communit...ertificates-with-ispconfig.83692/#post-400168
I changed, because acme.sh has a lot of advantages (for example easy DNS challenge with ispconfig and support of ECC certs). I would not recommend doing that to anyone. If there is need, one has to understand, that certbot needs to be completely removed and every certificate reissued with acme.sh.

 

There was a migration question by @Taleman which I posted an answer involving migration to acme.sh from certbot.

In my theory you could do that by making a full backup of /etc/letsencrypt and /etc/nginx@apache2/sites-enabled, totally purge certbot, certbot-auto and letsencrypt from your system, removing the earlier said (both) folders, install acme.sh, force update ispconfig, choose create ssl for it during that process, get into your control panel and resync.

In my mind that should work but do note @till observation in the said thread that LE limit might be hit.