Create Lets Encrypt SSL Certs via Certbot DNS Validation in Acme v02

Since certbot in Ubuntu 16.04 is upgraded to version 22, it is now ready to use Acme v2. I believe ISPConfig developers are already working on this but everybody have to be patient since it may not be out in the near future. As I am currently using CloudFlare as my dns server, I would like to share some tips/tricks that I recently did on my ubuntu nginx webserver in order to issue a Let's Encrypt wildcard SSL certs for one of my domains.

Firstly, other than installing the default certbot via "apt -y install python-certbot-nginx", I have to install cloudflare plugin for it too. This I did by running "apt -y install python3-certbot-dns-cloudflare python3-cloudflare". This plugin is essential for this tip/trick.

Secondly, create a hidden folder accessible only by root user and file for the required credentials to be filled in.

Code:

mkdir /etc/letsencrypt/.secrets
chown root:root /etc/letsencrypt/.secrets
chmod 600 /etc/letsencrypt/.secrets
# Create the credential file
nano /etc/letsencrypt/.secrets/domain.tld.ini

Thirdly, add the required credential inside the file. You obtained this from CloudFlare control panel for your domain.

Code:

# CloudFlare API key information
dns_cloudflare_api_key = xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
dns_cloudflare_email = [email protected]

And lastly, run certbot using the cloudflare plugin for the wanted domain(s) using dns validation to issue your domain certs, including for its wildcard subdomain, if you want.

Code:

certbot certonly \
  --dns-cloudflare \
  --dns-cloudflare-credentials ~/etc/letsencrypt/.secrets/domain.tld.ini \
  --server https://acme-v02.api.letsencrypt.org/directory \
  --agree-tos \
  --dns-cloudflare-propagation-seconds 60 \
  --rsa-key-size 4096 \
  --renew-hook letsencrypt_renew_hook.sh \
  -d domain.tld \
  -d *.domain.tld

This new certs will be defaulted to the same usual Let's Encrypt folder which you can manually use with ISPConfig. Its renewal file should look like this:

Code:

# renew_before_expiry = 30 days
version = 0.22.0
archive_dir = /etc/letsencrypt/archive/domain.tld
cert = /etc/letsencrypt/live/domain.tld/cert.pem
privkey = /etc/letsencrypt/live/domain.tld/privkey.pem
chain = /etc/letsencrypt/live/domain.tld/chain.pem
fullchain = /etc/letsencrypt/live/domain.tld/fullchain.pem

# Options used in the renewal process
[renewalparams]
account = xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
server = https://acme-v02.api.letsencrypt.org/directory
authenticator = dns-cloudflare
installer = None
dns_cloudflare_credentials = /etc/letsencrypt/domain.tld.ini

I think renewing via certbot renew should cover renewing this as well after the lapse of 60 days and before 90 days but I cannot confirm its renewal will work since mine is not subject to any renewal yet. So, please be attentive to friendly-reminding-email from Let's Encrypt just in case its renewal somehow failed after 60 days as you may need to do it manually.

 

In the addition to the above, since I think many ISPConfig servers use Bind, we may use certbot dns_rfc2136 plugin in almost similar way as above.

The idea is to firstly install Bind plugin and then create the TSIG base files (key and private) for the dns server, for examples Kdns.server.tld.+165+28266.key and Kdns.server.tld.+165+28266.private via the followings:

Code:

# Install certbot and its bind plugin, if you haven't.
apt -y install certbot
apt -y install python3-certbot-dns-rfc2136
# Get into bind directory and issue two TSIG base files.
cd /etc/bind
dnssec-keygen -a HMAC-SHA512 -b 512 -n HOST $(hostname -f).

Secondly, create the TSIG credential file for the dns server as follows:

Code:

# Define the intended domain accordingly.
domain4le=test.domain.tld <-- change this to your domain
dnsserverip=192.168.0.101 <-- change this to your dns server ip
# We now create tsig key credential file.
Kfile=$(ls K$(hostname -f).*.key)
Secret=$(cat $Kfile | awk '{print $7, $8}')
cat <<EOF > tsig.$(hostname -f).key
key "$(hostname -f)." {
  algorithm HMAC-SHA512;
  secret "$Secret";
};

zone "$domain4le." IN {
  type master;
  file "pri.$domain4le";
  update-policy {
   grant $(hostname -f). name _acme-challenge.$domain4le. txt;
  };
};
EOF

Thirdly, add the above credential file at the end of the targeted domain bind file pri.test.domain.tld in /etc/bind as follows:

Code:

# Add it in the domain dns / bind file.
cat <<EOF >> pri.$domain4le

\$INCLUDE tsig.$(hostname -f).key
EOF

Fourthly, make sure you are created the required secrets folder to save the secrets as follows:

Code:

# Create and protect .secrets folder
if [ -d "/etc/letsencrypt/.secrets" ]; then
  mkdir /etc/letsencrypt/.secrets
fi
chown root:root /etc/letsencrypt/.secrets
chmod 600 /etc/letsencrypt/.secrets

Fithfly, enter the secrets folder and use the ealier created TSIG credentials to fill the ini file as follows:

Code:

# Enter the .secrets folder & create the credential file.
cd /etc/letsencrypt/.secrets
cat <<EOF > $domain4le.ini
# Target ISPConfig DNS server
dns_rfc2136_server = $dnsserverip
# Target ISPConfig DNS port
dns_rfc2136_port = 53
# TSIG key name
dns_rfc2136_name = $(hostname -f).
# TSIG key secret
dns_rfc2136_secret = $Secret
# TSIG key algorithm
dns_rfc2136_algorithm = HMAC-SHA512
EOF

Lastly, issue LE SSL certs via the following command:

Code:

certbot certonly \
  --dns-rfc2136 \
  --dns-rfc2136-credentials ~/etc/letsencrypt/.secrets/domain.tld.ini \
  --server https://acme-v02.api.letsencrypt.org/directory \
  --agree-tos \
  --dns-rfc2136-propagation-seconds 60 \
  --rsa-key-size 4096 \
  -d domain.tld \
  -d *.domain.tld

It will be nice if I can write at least this up in a proper ISPConfig way, so that it can be contributed to the git but my knowledge is quite limited. Hopefully in ISPConfig version 3.2 we will see an option to obtain Let's Encrypt SSL certs via dns to certain extends.

Edited: The above code for both ISPConfig and CloudFlare is now ready to be rewritten in ISPConfig.

 

Now I can confirm that the renewal of my domain and its wildcard via cloudflare dns is working.

For those who are using the same method but got error in renewal for the domain and its wildcard, they should check the CAA entry which basically should consist of two CAA as follows:

Code:

CAA    domain.tld    0    issue "letsencrypt.org"    automatic
CAA    domain.tld    0    issuewild"letsencrypt.org"    automatic

The first is for the domain and the second is for its wildcard. That means if you want to create certs for sub.domain.tld only, you will need to create a CAA entry just for it. You can check you CAA entry via command "dig caa domain.tld" or via online service like at https://dnsspy.io/labs/caa-validator.

I think I can now try to write the this modification for letsencrypt in ISPConfig related files.

 

I just updated post #2 above with @till suggestions and with some extra, and I think they are now ready to be re-written for ISPConfig. Hopefully I am free to write them soon.

 

1. The following script is self-explanatory for the purpose of testing whether the code can be automatically run from ISPConfig Perfect Server (single server setup with dns capability) before expanding the same to be used in a multi server setup.

2. This script should be run before issuing certbot certonly command as described in post #2 above.

3. It is still a work in progress because at the end they will partly be converted to php code plus kept in database and files when it's (hopefully) ready.

4. Those who are interested may test this script on their test server at their own risk.

Code:

# 0 - Install certbot and its bind plugin, if you haven't.
apt -y install certbot
apt -y install python3-certbot-dns-rfc2136

# 1 - Define domain, tsig and ip.
domain=test.domain.tld
tsig=$(hostname -f)
dnsserverip=192.168.0.101


# 2 - Get to bind directory & issue 2 TSIG base files.
cd /etc/bind
dnssec-keygen -a HMAC-SHA512 -b 512 -n HOST $tsig.


# 3 - We now create tsig key file.
kfile=$(ls K$tsig.*.key)
secret=$(cat $kfile | awk '{print $7, $8}')
conf=named.conf.local
local=$(cat $conf)
cat <<EOF > $conf
key "$tsig." {
  algorithm hmac-sha512;
  secret "$secret";
};
$local
EOF
sed -i "s/file \"\/etc\/bind\/pri.$domain\";/allow-update {key $tsig.;};\n\tfile \"\/etc\/bind\/pri.$domain\";/g" $conf

# Add "key dns.server.tld." in Update ACL to make key permanent.
# Check its content if you want.
# cat $conf


# 4 - Add it in the domain dns / bind file.
cat <<EOF >> pri.$domain

\$INCLUDE /etc/bind/$kfile
EOF

# Again, check its content if you want.
# cat pri.$domain


# 5 - Create and protect .secrets folder
sdir=/usr/local/ispconfig/interface/ssl/.secrets
if [ ! -d "$sdir" ]; then
   mkdir $sdir
fi
chown root:root $sdir
chmod 600 $sdir

# 6 - Enter .secrets folder, create credential & chmod it 600
cd $sdir
cat <<EOF > $domain.ini
# Target ISPConfig DNS server
dns_rfc2136_server = $dnsserverip
# Target ISPConfig DNS port
dns_rfc2136_port = 53
# TSIG key name
dns_rfc2136_name = $tsig.
# TSIG key secret
dns_rfc2136_secret = $secret
# TSIG key algorithm
dns_rfc2136_algorithm = HMAC-SHA512
EOF
chmod 600 $domain.ini

# Again, check its content if you want.
# cat $domain.ini
# cd /etc/bind
# cat $conf
# cat pri.$domain

# 7 - Restart bind9 service
service bind9 restart

 

I am establishing two dns server on Hetzner VPS to perform a proper test for implementing the above on ISPConfig based on Debian Jessie 8.4 Cluster Server tutorial and Debian Squeeze DNS Server tutorial but using Ubuntu 18.04 Nginx Server Setup instead.

I hope I can conclude this plugin soonest possible on my limited available free time.

 

All test using certbot dns-rfc2136 (bind) plugin end up with: "Unable to determine base domain for _acme-challenge.test.domain.tld using names: ['_acme-challenge.test.domain.tld', 'test.domain.tld', 'domain.tld', 'tld']". I am using the latest version but this seems to be a reported bug which is not clear whether it was or will be fixed, so I'll drop any attempt to use this plugin for the time being.

The good news is I managed to simplify the process without using any plugin for issuing a dns validated LE ssl certs. This following code been tested working great for a single domain without any subdomain in a single ISCPConfig server setup, using certbot without any plugin.

1. Run nano /etc/bind/auth.sh to create it and add the following bash code.

Code:

#!/bin/bash

letoken=$CERTBOT_VALIDATION
pridomain=pri.$domain
pricatch=$(cat $pridomain)
cat <<EOF > $pridomain
$pricatch
_acme-challenge.$domain. 60     TXT     "$letoken"
EOF
service bind9 restart
sleep 10

2. Run nano /etc/bind/clean.sh to create it and add the following bash code.

Code:

#!/bin/bash

sed -i '/acme-challenge/d' pri.$domain
sed -i '3d' auth.sh
sed -i '3d' clean.sh
service bind9 restart

3. Make them both executable.

Code:

chmod +x /etc/bind/auth.sh && chmod +x /etc/bind/clean.sh

4. Run the following code after changing test.domain.tld to your own domain.

Code:

cd /etc/bind
domain=test.domain.tld
sed -i "s/letoken=/domain=$domain\nletoken=/g" auth.sh
sed -i -e "3idomain=$domain" clean.sh

5. Issue LE ssl certs via the following commands:

Code:

certbot certonly \
  --preferred-challenges dns \
  --manual \
  --manual-public-ip-logging-ok \
  --manual-auth-hook /etc/bind/auth.sh \
  --manual-cleanup-hook /etc/bind/clean.sh \
  --server https://acme-v02.api.letsencrypt.org/directory \
  --agree-tos \
  --rsa-key-size 4096 \
  -d $domain

I believe it is a matter of passing some arguments via ssh / plugins to the ISPCOnfig main dns sever from any other server in a multi server setup to achieve the same result. I'll try to venture into this later when I have a little bit more of free time.

 

I am able to extend the above hook to issue LE ssl certs to a remote server in a ISPConfig multiserver setup. For this extension to successfully works for you, the remote server must be given an access to the ISPConfig dns (bind) server via ssh keyless login in order to remotely run certain commands. The steps for it must be done in the remote server as follows:

1. In the remote server, create /etc/bind/auth.sh file with the following code:

Code:

#!/bin/bash

cat <<EOF > letoken
$CERTBOT_VALIDATION
EOF
scp letoken [email protected]:/etc/bind/
ssh [email protected] 'cd /etc/bind; CERTBOT_VALIDATION=$(cat letoken); . auth.sh'

2. Then, at the same server, create /etc/bind/clean.sh as follows:

Code:

#!/bin/bash

ssh [email protected]'cd /etc/bind; . clean.sh; rm letoken'

3. Specify the domain that you intend to get its LE ssl certs and run ssh remote command to change the default auth.sh and clean.sh similar to post #9 above as follows:

Code:

ssh [email protected] 'cd /etc/bind; domain=test.domain.tld; sed -i "s/letoken=/domain=$domain\nletoken=/g" auth.sh; sed -i -e "3idomain=$domain" clean.sh

4. Finally, issue a certbot command to request LE ssl certs for the targeted domain:

Code:

certbot certonly \
  --preferred-challenges dns \
  --manual \
  --manual-public-ip-logging-ok \
  --manual-auth-hook /etc/bind/auth.sh \
  --manual-cleanup-hook /etc/bind/clean.sh \
  --server https://acme-v02.api.letsencrypt.org/directory \
  --agree-tos \
  --rsa-key-size 4096 \
  -d $domain

If nothing goes wrong, you should be getting your desired LE ssl certs to your remote server for whatever purpose you wish to use it for, whether for email , mysql, even another dns or web server.

This above trick is basically grabbing certbot token for validation and pass it to dns server temporarily, that is until the LE verificationfor TXT for the targeted domain is finished; and afterwards, it will be removed.

Please note that for this dns LE ssl certs to get properly renewed, you must make sure that command for renewal (certbot renew --dry-run) is setup as a cron job and run at least once in 24 hours especially for server(s) other than the ISPConfig web server.

Hopefully, when I am done re-writing this in php for ISPConfig, there is no need for you to do so manually as all will be done by ISPConfig automatically.

 

This looks to be a good start! Thanks for all the research you've performed thus far, @ahrasis .

I'd love to see support for DNS challenge-response built into ISPConfig, and I'd love even more to see support for hosting-provider-specific plugins.

For example, I would prefer to do something like this in the relevant cron script:

Code:

certbot certonly \
  --dns-digitalocean \
  --dns-digitalocean-credentials ~/.secrets/certbot/digitalocean.ini \
  -d example.com

This works really well for me and doesn't require all the hoop-jumping with NGINX just to perform the challenge-response.

How difficult would it be to add support for something like this in ISPConfig?

 

Actually it is not that difficult but ISPConfig current direction is to use acme.sh in the near future, instead of certbot, though certbot will still be supported for certain period times and as another ordinary user I am not so sure whether all previous discussions regarding certbot is considered material anymore.

 

Thank you for the reply @ahrasis , much appreciated.

In your opinion, if I want to use the method I describe in my previous post, what is the "safest"/"cleanest" means by which to implement that?

Currently, it appears that ISPConfig has its own cron script that checks for updates to Let's Encrypt certificates.

But I hesitate simply to disable that script, because perhaps it contains important functionality that my very simple script (noted in my previous post) does not address.

 

Current ISPConfig cron script should work fine for all renewals including dns-challenge as certbot will be looking for settings it has created i.e. in the renewal file of the related domain.

 

Hmm. Are you referring to, e.g., the files located in /etc/letsencrypt/renewal/?

If I examine one of those files, it contains the following:

Code:

# renew_before_expiry = 30 days
version = 0.32.0
archive_dir = /etc/letsencrypt/archive/example.com
cert = /etc/letsencrypt/live/example.com/cert.pem
privkey = /etc/letsencrypt/live/example.com/privkey.pem
chain = /etc/letsencrypt/live/example.com/chain.pem
fullchain = /etc/letsencrypt/live/example.com/fullchain.pem

# Options used in the renewal process
[renewalparams]
account = ...
server = https://acme-v01.api.letsencrypt.org/directory
authenticator = webroot
rsa_key_size = 4096
post_hook = echo '1' > /usr/local/ispconfig/server/le.restart
[[webroot_map]]
example.com = /usr/local/ispconfig/interface/acme

Should I add the DNS-related information to this file manually? If I do, will ISPConfig overwrite it if I make a change to the associated website's configuration via the ISPConfig interface, or upgrade ISPConfig (or even Reconfigure Services)?

 

Yes. If you follow my suggestion when creating the certs using dns-challenge, it should look something like this:

Code:

# renew_before_expiry = 30 days
version = 0.29.1
archive_dir = /etc/letsencrypt/archive/domain.tld
cert = /etc/letsencrypt/live/domain.tld/cert.pem
privkey = /etc/letsencrypt/live/domain.tld/privkey.pem
chain = /etc/letsencrypt/live/domain.tld/chain.pem
fullchain = /etc/letsencrypt/live/domain.tld/fullchain.pem

# Options used in the renewal process
[renewalparams]
account = xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
dns_cloudflare_credentials = /etc/letsencrypt/domain.tld.ini
server = https://acme-v02.api.letsencrypt.org/directory
authenticator = dns-cloudflare
rsa_key_size = 4096

You should not be adding this manually.

 

Thank you for the continued assistance @ahrasis !

I see. So, I created a new certificate on the CLI, with the options I want to use, and that domain's configuration file now looks like this:

Code:

# renew_before_expiry = 30 days
version = 0.31.0
archive_dir = /etc/letsencrypt/archive/example.com
cert = /etc/letsencrypt/live/example.com/cert.pem
privkey = /etc/letsencrypt/live/example.com/privkey.pem
chain = /etc/letsencrypt/live/example.com/chain.pem
fullchain = /etc/letsencrypt/live/example.com/fullchain.pem

# Options used in the renewal process
[renewalparams]
dns_digitalocean_credentials = /root/.secrets/certbot/digitalocean.ini
server = https://acme-v02.api.letsencrypt.org/directory
authenticator = dns-digitalocean
account = xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

That all seems correct.

But if I now go into the ISPConfig interface and attempt to check the "Let's Encrypt SSL" box for the website, saving the change fails because ISPConfig is executing the following command in the background (I obtained this from the System Log with Debug-level logging enabled):

Code:

/usr/bin/letsencrypt certonly -n --text --agree-tos --expand --authenticator webroot --server https://acme-v02.api.letsencrypt.org/directory --rsa-key-size 4096 --email [email protected] --domains example.com --domains www.example.com --webroot-path /usr/local/ispconfig/interface/acme

which results in the following error log entry:

Code:

Let's Encrypt Cert file: does not exist.

Oddly, the Let's Encrypt log in /var/log/letsencrypt/letsencrypt.log does not indicate any type of error; it looks successful.

And, furthermore, if I literally copy/paste the exact command from ISPConfig's system log, it works as expected:

Code:

# /usr/bin/letsencrypt certonly -n --text --agree-tos --expand --authenticator webroot --server https://acme-v02.api.letsencrypt.org/directory --rsa-key-size 4096 --email [email protected] --domains example.com --domains www.example.com --webroot-path /usr/local/ispconfig/interface/acme
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator webroot, Installer None
Cert not yet due for renewal
Keeping the existing certificate

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Certificate not yet due for renewal; no action taken.
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

Given all of the above, I'm not sure how else to troubleshoot why ISPConfig fails to update the same website through the web interface.

 

It doesn't work the way you described above since the UI website settings for LE are meant only for using webroot and not dns-challenge.

So while your first step is correct, you should not do the UI steps onwards, as it "may" create another certs using webroot, on top of current dns-challenge based certs.

You can check your LE folder to confirm whether now both exist or just one, because if there are two, you should decide to delete the one created using webroot altogether.

As a note, the certs you created using certbot 0.31 has bugs where it failed to mention your domain name inside the renewal file and as for that you may want to try to use git-stable of ISPConfig before 3.1.14 is ready.

 

There are no duplicate certificates in my LE directory.

Are you certain that LE would create duplicates, given the use-case I describe? It almost seems as though LE would simply determine that a certificate already exists for the domain for which I enable LE via the ISPConfig interface and skip it.

In fact, that's exactly what the output from my previous post implies; that's precisely the command that ISPConfig executes when I save the website after checking Let's Encrypt SSL, and there appears not to be a problem. Why ISPConfig rolls-back the changes with a cryptic message, "Let's Encrypt Cert file: does not exist." is another matter.

I'm content with not using the ISPConfig interface for LE certificates anymore, so I can use the DNS challenge method, but how then should I enable SSL using the LE certificates for ISPConfig-managed websites?

Would I need to enable LE for all websites that require it using the ISPConfig interface before switching the challenge-response mechanism to DNS "manually"?

 

No you would not need to enable LE via UI but you may need a custom vhost config for your websites.

This is a bit tricky but can be done.

 

I really don't like the idea of maintaining a custom vhost config for my websites... there must be a better way.

Upon digging a bit further, it looks like that error I'm seeing in the Debug Log is generated here:

https://git.ispconfig.org/ispconfig...r/server/lib/classes/letsencrypt.inc.php#L480

Notice that the

Code:

$crt_tmp_file

evaluates to an empty string when the message is rendered. The value looks to be defined on https://git.ispconfig.org/ispconfig...r/server/lib/classes/letsencrypt.inc.php#L413 :

PHP:

        if($server_type != 'apache' || version_compare($app->system->getapacheversion(true), '2.4.8', '>=')) {
            $crt_tmp_file = $le_files['fullchain'];
        } else {
            $crt_tmp_file = $le_files['cert'];
        }


So, for whatever reason,

Code:

$le_files['fullchain']

appears to be empty. The value shouldn't be empty, or the error-handling should be improved so that it doesn't print an empty string when the certificate file for which it's checking on https://git.ispconfig.org/ispconfig...r/server/lib/classes/letsencrypt.inc.php#L434 doesn't exist.

The whole thing feels buggy.