Let's Encrypt!

I'm trying to set up "letsencrypt" support for my cluster.

The thing is, I originally skipped setting up letsencrypt when initially setting up the cluster, as we were intending to sell SSL certificates and free ones would have stepped on the toes of that. But we've reviewed our plans and it makes sense to offer free SSL certificates from "let's encrypt" to everyone - and those who have more specific requirements, such as wildcards or organisation validation can pay extra for that.

Anyway, I've followed the instructions in the manual to install "certbot-auto" from the /opt/certbot directory and this went along fine without errors.

But when I try ticking the "Let's Encrypt SSL" box on a website, I get emails from the server that the domains could not be verified. And attempting to go to "https://website.tld" results in a "403 forbidden" error page.

Is this because I installed certbot after ISPConfig and it's not configured? Or is there some other issue?

 

Looking in the log files, I'm getting "Client does not support any combination of challenges that will satisfy the CA" as the error.

Seaching on this, I found that there's been an active incident with Let's Encrypt that's forced them to disable the TLS-SNI challenge completely:

https://letsencrypt.status.io/pages/incident/55957a99e800baa4470002da/5a55777ed9a9c1024c00b241

Their recommendation is to upgrade to the latest version of "certbot" (they provide a PPA for Ubuntu). But the thing is, the rename from "letsencrypt" to "certbot" isn't, I presume, something that ISPConfig takes account of, yes?

 

Excellent. I'l try that this morning.

One thing - as I've got a clustered setup - is that I presume I need to put "/etc/letsencrypt" onto the shared storage, so that all the nodes can access the SSL certificates? Or is this automatically done by ISPConfig in a mirrored multiserver configuration and so I should avoid erroneously sharing things twice for no reason?

 

Hmm, I've got the latest certbot, but it's still not working.

I can see from the logs that it's attempting the "http-01" test, trying to access a file - random filename - under "/.well-known" of the web root. When I browse to that directory, I do indeed find a ".well-known" hidden directory there, albeit that it's empty - no files inside - by the time I go to check (does certbot delete the temporary files afterwards?).

One thing of note is that "/var/www" is mounted from the storage server over NFS. Also, I tried mounting "/etc/letsencrypt" over NFS from the storage server too, so that the SSL certificates and keys would be available to all the nodes in the cluster. Could either of these things be affecting certbot's operation, perhaps?

I don't get it, really. Because the websites that I'm trying are definitely publicly accessible - and have been for many months, so it's not a DNS propagation delay or anything - and certbot is capable of creating that ".well-known" directory, as I can see that's there, so it really shouldn't have a problem creating a file in there, which would be publicly accessible to the "letsencrypt" servers, that verifies ownership of the domain.

But I'm reading "authorisation failure" in the logs.

 

And that would be the "acme" directory under /usr/local/ispconfig/interface, yes?

 

Okay, almost there.

I spotted in the "letsencrypt.log" that it was saying "please choose an account". And, lo and behold, looking in the "/etc/letsencrypt" subdirectories, there was more than one account in there.

I deleted one of these accounts - an old one from a month ago, so it must have been created from my initial testing - and we have success. I can now get Letsencrypt SSL certificates via ISPConfig. I can see the "pem" files in the "/etc/letsencrypt" directory. And the "Let's Encrypt SSL" tick box stays on.

It works!

But I have another issue with a few of the websites. These were some of the first websites I created and the cluster has been re-arranged a little since. So, basically, I created these websites on "storage0" - and that's the server they're listed under on the "websites" list page of ISPConfig - but things have been moved around in the meantime and the websites are now moved to "storage1" and that's the server that's being mirrored.

So the problem is that when I create an SSL certificate for these older sites that are on "storage0", then it's the "storage0" server that's updating its Apache configuration with the new SSL settings. But the web cluster now mirrors "storage1" and not "storage0" - we shifted all the websites to "storage1" to be the dedicated "websites server" - and so the other nodes are not mirroring the new SSL settings.

So I have an SSL certificate now but, for these older sites, it's not being used because the cluster has been re-arranged since they were created and this means the Apache SSL configuration that's being updated by ISPConfig is not what's being mirrored, as they're not technically on the mirror server (which is now "storage1" but they were created back when I only had "storage0").

Thus, as these websites are officially on "storage0" then it's "storage0" which is running certbot and updating its SSL configuration. But the server that's being mirrored has now become "storage1" since expanding the cluster.

I'm thinking that, basically, I go into "dbispconfig" and manually change the websites to be on "storage1" (due to using shared storage across the cluster, this makes no difference to accessing the website files - they all locally map /var/www to the shared storage - but it does make a difference to which Apache configuration is altered / mirrored to all the other servers in the cluster).

But, well, this seems like it's the sort of action that, were I wrong about this presumption of ISPConfig's operation, would seriously mess things up.

Also, it's always those little "gotchas" that get you, isn't it? Like, earlier, I'd already worked out that I probably needed to share "/etc/letsencrypt" so all the nodes could access the SSL certificates. But I didn't know about the "/usr/local/ispconfig/interface/acme" directory where certbot's tokens go.

Thus, before I embark on this database editing adventure, I thought I'd get your confirmation that I am, in fact, barking up the right tree with what I need to do here, and if there are any other things I need to change / update to make this work.

 

Yeah, I realise it's an unusual setup.

I built it all one way and then, with the addition of new servers, it was decided to restructure things in a different way and now I've got to twist things around to make this new way work.

Anyway, I guess what I'm asking is about the fields in the "dbispconfig" database that need changing in the master database.

I can see that there's a "server_id" under "web_domain", so I guess that needs changing to the new server ID I want. I notice that "web_traffic" is indexed by domain name and not server ID, so that should be okay to leave alone?

There's a "server_ID" for "web_database" but I presume that's the ID of the DB server (which is a different machine and that's not changed). And the "server_ID" under "web_database_user" is zero - which not being a valid server ID is, I guess, a programmer's way of saying "N/A" or that the user is not restricted to any particular DB server.

So, from my investigations of the database fields, I'm thinking that a change of "server_id" in "web_domain" will "move" the websites to the other server.

I guess what I'm asking here is "am I missing something?". If I just change that number to the ID of the new server in the master database then it should be "moved" over to that server? And the other servers - in their cron jobs - do a query on "server_id" to find changes to the master database, so this will propagate out to the cluster, so that, when the configuration is changed, the new server picks it up as its responsibility? Anything else obvious that I'm missing here?

 

Cool. I'll try it on one site to see if it works and¸ if it does, manually fix it all up.

Thanks for answering my often rather mad questions.

 

Yeah, it works now. Thanks a bunch.