LetsEncrypt auto-renew not working: how to solve this?

Discussion in 'Installation/Configuration' started by MarsWarrior, Jan 11, 2017.

  1. MarsWarrior

    MarsWarrior New Member

    Currently all my SSL certificates are out of date.
    The reason for this seems to be this issue: https://git.ispconfig.org/ispconfig/ispconfig3/issues/4380

    There is also a priority thread about this.

    I want to solve this ASAP, but don't understand the steps to take.

    Could someone explain the steps to solve this manually?

    THese are the errors I get from the letsencrypt log:
    2017-01-11 02:00:23,344:DEBUG:certbot.plugins.webroot:Creating root challenges validation dir at /usr/local/ispconfig/interface/acme/.well-known/acme-challenge
    2017-01-11 02:00:23,388:DEBUG:certbot.plugins.webroot:Attempting to save validation to /usr/local/ispconfig/interface/acme/.well-known/acme-challenge/dFv...

    FailedChallenges: Failed authorization procedure. sub.domain.nl (http-01): urn:acme:error:unauthorized :: The client lacks sufficient authorization :: Invalid response from http://sub.domain.nl/.well-known/acme-challenge/dFv...
    <!DOCTYPE html>
    <html lang="en">
    <meta http-equiv="content-type" content="text/html; charset=utf-8">
    <title>Page not"

    I assume the .well-known path is somewhere rewritten to the /user/local path instead of to the root of the /web/ folder for that domain??
    Last edited: Jan 11, 2017
  2. till

    till Super Moderator Staff Member ISPConfig Developer

    The issue you linked is not about failing renewal, it is about successfully renewal but a wrong symlink and it is limited for a special configuration using relative symlinks, so it does not apply to a default ISPConfig setup. According to your log, the renewal fails because lets encrypt is unable to reach a sub domain, so your problem is different

    Your log file says that lets encrypt tried to reach sub.domain.nl from outside to verify the SSL cert and this domain could not be reached. So the first step is to check if DNS for sub.domain.nl is correct and pointing to the right server
  3. MarsWarrior

    MarsWarrior New Member

    Ah. Sorry for the mix-up then ;)

    The subdomain does exist: I can reach it from the outside and about three months ago I was able to request the LetsEncrypt certificates.
    - subdomain is a proxy for another webserver
    - http is directed to https

    This is in the vhost for port 80:
    RewriteEngine on
    RewriteCond %{REQUEST_URI} ^/\.well-known/acme-challenge/
    RewriteRule ^ - [END]
    RewriteCond %{HTTPS} off
    RewriteRule (.*) https://%{HTTP_HOST}%{REQUEST_URI} [R=301,L]

    What I don't understand is that the request is to http://sub.domain.nl/.well-known/acme-challenge/dFv..., but that the log says that the actual location is at /usr/isponfig... Is that somewhere rewritten / linked, as there is no .well-known folder in the rootfolder of the domain.

    I disabled the proxy in the *.80 vhost config as I thought that that could be the problem, but that didn't solve the renewal problem.
    Last edited: Jan 11, 2017
  4. till

    till Super Moderator Staff Member ISPConfig Developer

    Yes, the path gets rewritten to a central directory where all tokens for all domains get created, you can find the rewrite rule in the apache ispconfig.conf file.
  5. MarsWarrior

    MarsWarrior New Member

    Ok. I tested access to http://sub.domain.nl/.well-known/acme-challenge/

    I get a 403 and an ISPConfig generated screen with:
    The following error occurred:
    You are not permitted to access the requested URL.

    Please contact the webmaster with any queries.

    This generates the following error in the subdomains access.log file:
    [Wed Jan 11 14:22:42.699626 2017] [autoindex:error] [pid 15385] [client] AH01276: Cannot serve directory /usr/local/ispconfig/interface/acme/.well-known/acme-challenge/: No matching DirectoryIndex (index.html,inde$

    The access from last night was (access.log file for sub domain) a 404 result: - - [11/Jan/2017:03:00:27 +0100] "GET /.well-known/acme-challenge/pgAzyYZTi9mllkG3RVjWztUCAzjBHkxTdq10X9F58jU HTTP/1.1" 404 927 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"

    So the main folder gives me an permission error, but LetsEncrypt validation gets a 404.

    What's the next thing I could check??
  6. till

    till Super Moderator Staff Member ISPConfig Developer

    The permission error is fine as you should not be able to get a file list for that folder (and that's what you request by accessing it with a browser). letsencrypt creates a token file and then tries to access it in the folder, so if you want to try what letsencrypt is doing, then you must create a file in that folder and then try to access that file with a browser.

    My guess is that it is related to the proxy redirect, I know that you disabled it but there must be still some config that causes the request from letsencrypt to be redirected to a wrong folder or server as it worked at the time you created the letsencrypt cert, so the change must be made after that date. did you try a manual letsencrypt renewal after you disabled the proxy? The command is either:

    certbot -n renew


    letsencrypt -n renew
  7. MarsWarrior

    MarsWarrior New Member

    I don't want to solve it manually, as this does not solve the problem.

    I did make a file named 'testfile' with the contents "hello" in /usr/local/ispconfig/interface/acme/.well-known/acme-challenge/test#

    and can access that file via:
    http://sub.domain.nl/.well-known/acme-challenge/test/testfile, where the "hello" is shown in the browser window!

    THis works for ALL the disables proxies. It doens't for enabled proxies, so disabling the proxies in the *.80 vhost configuration is essential for this to work!

    I can also access the empty.dir file (it is downloaded by my browser).

    So no problems with the proxy. But what's the problem then??
    Last edited: Jan 11, 2017
  8. till

    till Super Moderator Staff Member ISPConfig Developer

    That's the same command that ispconfig runs once a night. But you can wait until tomorrow of course until the command has been run by the nightly cronjob.
  9. MarsWarrior

    MarsWarrior New Member

    Aha, then I might try it manually!.

    Next thing I see in the logging:
    - dns-01 challenge fails
    - challenge files are removed
    - then the http-01 challenge fails with a 404 on that file that is just removed by the failing dns-01 challenge

    2017-01-11 02:00:26,810:INFO:certbot.auth_handler:Cleaning up challenges
    2017-01-11 02:00:26,810:DEBUG:certbot.plugins.webroot:Removing /usr/local/ispconfig/interface/acme/.well-known/acme-challenge/dFvpmcL2MnrXEfnVp57WQ9e7xmFEOyKdFUO-Bjw1GHU
    2017-01-11 02:00:26,811:INFO:certbot.plugins.webroot:Unable to clean up challenge directory /usr/local/ispconfig/interface/acme/.well-known/acme-challenge
    2017-01-11 02:00:26,811:DEBUG:certbot.plugins.webroot:Error was: [Errno 39] Directory not empty: '/usr/local/ispconfig/interface/acme/.well-known/acme-challenge'
    2017-01-11 02:00:26,811:WARNING:certbot.renewal:Attempting to renew cert from /etc/letsencrypt/renewal/sub.domain.nl-0001.conf produced an unexpected error: Failed authorization procedure. sub.domain.nl (http-01): urn:acme:
    error:unauthorized :: The client lacks sufficient authorization :: Invalid response from http://sub.domain.nl/.well-known/acme-challenge/dFvpmcL2MnrXEfnVp57WQ9e7xmFEOyKdFUO-Bjw1GHU: "

    <!DOCTYPE html>
    <html lang="en">
    <meta http-equiv="content-type" content="text/html; charset=utf-8">
    <title>Page not". Skipping.

    Is that something?
    Last edited: Jan 11, 2017
  10. till

    till Super Moderator Staff Member ISPConfig Developer

    DNS challenge is not used, so it is ok that it fails, but the http challenge should work, that's the token download attempt. Does it still fail for the domain where you removed the proxy?
  11. MarsWarrior

    MarsWarrior New Member

    Yes. I just added the logging in my previous post.

    Look at this line: 2017-01-11 02:00:26,810:DEBUG:certbot.plugins.webroot:Removing /usr/local/ispconfig/interface/acme/.well-known/acme-challenge/dFvpmcL2MnrXEfnVp57WQ9e7xmFEOyKdFUO-Bjw1GHU

    That is done before the http-01 challenge so it seems!

    It is a subdomain where I removed the proxy, ie where I proved that access to the testfile is working.
  12. till

    till Super Moderator Staff Member ISPConfig Developer

    Please try to disable the http to https redirect and check if it works then.
  13. MarsWarrior

    MarsWarrior New Member

    Disables https redirect.
    At the same time ssl and letsencrypt where unchecked. Can't check letsencrypt anymore. If I enable it, it is disabled again after saving...

    THe only thing I did after the initial letsencrypt is upgrade ISPCOnfig from 3.1 to the 3.1.p.1

    And I can't find letsencrypt and certbot commands to manually renew. Where are these located?
    Last edited: Jan 11, 2017
  14. MarsWarrior

    MarsWarrior New Member

    Status update of this problem :)

    The proxy is definately the source of the problem. Completely removing the proxy (in the website options tab in ISPConfig) resulted in a renewal of the LetsEncrypt certificate.

    After that, I tried to find a solution that would result in:
    - No proxy for port 80
    - A proxy for port 443

    My first try was (with the incredible bad Apache documentation) to use a check on the port number:
    <If "%{SERVER_PORT} == '443'">

    But it appears that the ProxyPass directive is NOT accepted within that If block :mad:

    So I changed that using the <Location> block. Below should disable the proxy for LetsEncrypt challenges:
    <Location "/.well-known/acme-challenge/">
    ProxyPass "!"
    ProxyPassReverse "!"​

    I still have to run some tests to make sure that this works.
    The problem is that if I add a domain, and tick the SSL and LetsEncrypt checkboxes and continue to the other tab to enter the proxy details, ISPConfig already starts to issue the certificate (the red circle at the top is already blinking while I'm still entering data for the domain and I haven't hit the "Save" button yet!!!).

    So I'm not 100% sure that the above solution is indeed the right one!
    No Idea if I should use a wildcard orso...
  15. MarsWarrior

    MarsWarrior New Member

    Last night's log in the sub.domain.nl access log, 404's as before with a proxied subdomain: - - [25/Jan/2017:03:02:20 +0100] "GET /.well-known/acme-challenge/rC6... HTTP/1.1" 404 1547 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; $ - - [25/Jan/2017:03:02:24 +0100] "GET /.well-known/acme-challenge/Acen... HTTP/1.1" 404 1548 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; $

    I Changed the proxy settings yesterday afternoon and re-enabled HTTPS in ISPConfig giving this log: - - [25/Jan/2017:16:12:10 +0100] "GET /.well-known/acme-challenge/TEpe... HTTP/1.1" 200 450 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +$

    But still, the certificate is old... NO renewal has taken place! The latest access logfile doesn't contain any GET from LetsEncrypt.

    • This subdomain has a -0001.conf file in the /etc/letsencrypt/renewal folder. The other subdomains don't have that.
    • The LetsEncrypt logfile tells me that renewal has taken place:
    • 2017-01-26 02:00:30,859:DEBUG:certbot.storage:Writing new private key to /etc/letsencrypt/archive/sub.domain.nl-0001/privkey2.pem.
      2017-01-26 02:00:30,859:DEBUG:certbot.storage:Writing certificate to /etc/letsencrypt/archive/sub.domain.nl-0001/cert2.pem.
      2017-01-26 02:00:30,859:DEBUG:certbot.storage:Writing chain to /etc/letsencrypt/archive/sub.domain.nl-0001/chain2.pem.
      2017-01-26 02:00:30,859:DEBUG:certbot.storage:Writing full chain to /etc/letsencrypt/archive/sub.domain.nl-0001/fullchain2.pem.
      2017-01-26 02:00:31,669:DEBUG:certbot.storage:Writing new config /etc/letsencrypt/renewal/sub.domain.nl-0001.conf.new.
    So, what's going wrong in this case???
  16. MarsWarrior

    MarsWarrior New Member

    New day, new results!

    Another domain went well last night: the certificate was renewed :)

    However, https did'nt work for that domain o_O
    It appears that ISPConfig simply 'forgets' to write the *:443 part in the vhost file after some change in the configuration of the domain. Only the *:80 part is present. I have to disable and enable SSL and LetsEncrypt checks before ISPConfig does write a complete vhost file again.

    So, the renewal was with a domain that had only port 80 enabled. The proxy & location statements where in it though, so I hope that this solution does work. Not writing the port 443 part must be a bug I guess...

    I'm running the latest ISPConfig v3.1.2

Share This Page