[solved] Letsencrypt writes certs but checkbox gets empty after a while

Discussion in 'Installation/Configuration' started by linus, Apr 5, 2017.

  1. linus

    linus Member

    The Let's Encrypt works well, except for that after a while, some minutes or sooner, the gui forgets the tick in the Let's Encrypt checkbox.

    upload_2017-4-5_18-41-14.png
    The certificate still works on the site and the debug log shows:

    Code:
    05.04.2017-18:32 - DEBUG - Restarting httpd: systemctl reload apache2.service
    05.04.2017-18:32 - DEBUG - Remove Lock: /usr/local/ispconfig/server/temp/.ispconfig_lock
    05.04.2017-18:32 - DEBUG - Calling function 'check_phpini_changes' from plugin 'webserver_plugin' raised by action 'server_plugins_loaded'.
    05.04.2017-18:32 - DEBUG - Found 1 changes, starting update process.
    05.04.2017-18:32 - DEBUG - Calling function 'ssl' from plugin 'apache2_plugin' raised by event 'web_domain_update'.
    05.04.2017-18:32 - DEBUG - Calling function 'update' from plugin 'apache2_plugin' raised by event 'web_domain_update'.
    05.04.2017-18:32 - DEBUG - Create Let's Encrypt SSL Cert for: intra.example.com
    05.04.2017-18:32 - DEBUG - Let's Encrypt SSL Cert domains: intra.example.com
    05.04.2017-18:32 - DEBUG - exec: /usr/bin/letsencrypt certonly -n --text --agree-tos --expand --authenticator webroot --server https://acme-v01.api.letsencrypt.org/directory --rsa-key-size$...
    05.04.2017-18:32 - DEBUG - Enable SSL for: intra.example.com
    05.04.2017-18:32 - DEBUG - Writing the vhost file: /etc/apache2/sites-available/intra.example.com.vhost
    05.04.2017-18:32 - DEBUG - Processed datalog_id 16331
    05.04.2017-18:32 - DEBUG - Calling function 'restartHttpd' from module 'web_module'.
    05.04.2017-18:32 - DEBUG - Restarting httpd: systemctl reload apache2.service
    05.04.2017-18:32 - DEBUG - Remove Lock: /usr/local/ispconfig/server/temp/.ispconfig_lock
    05.04.2017-18:33 - DEBUG - Calling function 'check_phpini_changes' from plugin 'webserver_plugin' raised by action 'server_plugins_loaded'.
    05.04.2017-18:33 - DEBUG - Remove Lock: /usr/local/ispconfig/server/temp/.ispconfig_lock
    
    After saving from the GUI the mysql table shows correct (I assume) (as long as the tick remains correct):
    upload_2017-4-5_18-43-33.png
    But after a while it shows:
    upload_2017-4-5_18-43-55.png

    In the log I just see entries like this when it forgets:

    Code:
    05.04.2017-18:44 - DEBUG - Calling function 'check_phpini_changes' from plugin 'webserver_plugin' raised by action 'server_plugins_loaded'.
    05.04.2017-18:44 - DEBUG - Remove Lock: /usr/local/ispconfig/server/temp/.ispconfig_lock
    finished.
    
    I did go to tools resync websites and I thought that fixed it because it remembered it for a longer time (many minutes) but then it forgot again.

    I tried also with different domains, the other domain also lost its tick from the SSL-checkbox after a while. The domains have been migrated using the migration tool from another server. The problem also seem to occur to new domains.

    The system is Debian 8 and Ispconfig 3.1.2, certbot version is 0.9.3.
     
  2. linus

    linus Member

    What part of Ispconfig would check the Letsencrypt status and enter a "n" to the ssl_letsencrypt field in the database? Why would it update it after the changes have successfully been written both to the cert-files and to the database?
     
  3. linus

    linus Member

    I also downloaded the latest ispconfig and run php update.php and reconfigure services, but both ssl and let's encrypt-checkboxes still become empty.
     
  4. linus

    linus Member

    Does it have to do with this part:

    plugins-available/apache2_plugin.inc.php
    Code:
        $success = $this->_exec($letsencrypt . " certonly -n --text --agree-tos --expand --authenticator webroot --server https://acme-v01.api.letsencrypt.o$
                                    }
                                    if(!$success) {
                                            // error issuing cert
                                            $app->log('Let\'s Encrypt SSL Cert for: ' . $domain . ' could not be issued.', LOGLEVEL_WARN);
                                            $data['new']['ssl_letsencrypt'] = 'n';
                                            if($data['old']['ssl'] == 'n') $data['new']['ssl'] = 'n';
                                            /* Update the DB of the (local) Server */
                                            $app->db->query("UPDATE web_domain SET `ssl` = ?, `ssl_letsencrypt` = ? WHERE `domain` = ?", $data['new']['ssl'], 'n', $data['new']['domain']);
                                            /* Update also the master-DB of the Server-Farm */
                                            $app->dbmaster->query("UPDATE web_domain SET `ssl` = ?, `ssl_letsencrypt` = ? WHERE `domain` = ?", $data['new']['ssl'], 'n', $data['new']['domain'])$
                                    }
    
    From the debug I checked that when trying to generate the certificate it gives not yet due for renewal and the $success variable is 1. So The update web_domain should not be run. Is there any other place where the ssl_letsencrypt field is modified?
     
  5. till

    till Super Moderator Staff Member ISPConfig Developer

    Just a guess, are you sure that you allowed letsencrypt for the client that owns this website (see client limits)? If you as admin enable letsencrypt and then your client logs in and has no letsencrypt permission, then the cert will get disabled.
     
  6. linus

    linus Member

    Till, your lucky guess was right, the client had not the permissions, but even when I now gave the right permissions it still unchecks ssl and letsencrypt for the sites during the first server.sh run. All certificates are ok.

    upload_2017-4-6_13-25-5.png


    and now

    upload_2017-4-6_13-38-14.png

    But I don't believe the client have has logged in while I tested. I think it forgets every time server.sh is run.
    Should I resync clients / web again or anyhing else I could try, now I just made the setttings to the web again but it didn't last.
     
  7. till

    till Super Moderator Staff Member ISPConfig Developer

    How many minutes does it take until it gets disabled? the checkbox must be disabled when the ssl cert creation by LE fails, so my guess is that it gets disabled within one minute after you enabled it and this means that LE was not able to issue a cert and ispconfig reflects that failure by disabling the checkbox. In this case, look into the letsencrypt.log file to see why LE could not issue that cert.
     
  8. linus

    linus Member

    In the beginning I think it took many runs, but I can't recreate that anymore, maybe I was just wrong. Now the ticks disappear every time server.sh is run (every minute from cron).

    LE says it will not recreate the certificate as it not yet due.

    Ispconfig debug:
    Code:
    Cert not yet due for renewal
    Keeping the existing certificate
    
    letsencrypt.log
    Code:
    2017-04-06 10:30:52,766:DEBUG:parsedatetime:parse (bottom) [][30 days][][]
    2017-04-06 10:30:52,766:DEBUG:parsedatetime:weekday False, dateStd False, dateStr False, time False, timeStr False, meridian False
    2017-04-06 10:30:52,766:DEBUG:parsedatetime:dayStr False, modifier False, modifier2 False, units True, qunits False
    2017-04-06 10:30:52,766:DEBUG:parsedatetime:_evalString(30 days, time.struct_time(tm_year=2017, tm_mon=4, tm_mday=6, tm_hour=10, tm_min=30, tm_sec=52, tm_wday=3, tm_yday=96, tm_isdst=0))
    2017-04-06 10:30:52,766:DEBUG:parsedatetime:_buildTime: [30 ][][days]
    2017-04-06 10:30:52,766:DEBUG:parsedatetime:units days --> realunit days
    2017-04-06 10:30:52,766:DEBUG:parsedatetime:return
    2017-04-06 10:30:52,766:INFO:certbot.renewal:Cert not yet due for renewal
    2017-04-06 10:30:52,766:INFO:certbot.main:Keeping the existing certificate
    
    I think it looks good, and I as I said, the $success variable is 1 when I test by running server.sh manually, so it shouldn't run the update_db, but somewhere it does.
     
  9. till

    till Super Moderator Staff Member ISPConfig Developer

    Do you use a custom vhost template file which has not been adjusted for the current ISPConfig version yet?
     
  10. linus

    linus Member

    No, not anymore, before the migration there was a varnish-adaption, but the conf-custom files were not moved.

    /usr/local/ispconfig/server# ls -la /usr/local/ispconfig/server/conf-custom/*/*
    Code:
    -rwxr-x--- 1 root root  45 Apr  5 21:55 /usr/local/ispconfig/server/conf-custom/error/empty.dir
    -rwxr-x--- 1 root root  45 Apr  5 21:55 /usr/local/ispconfig/server/conf-custom/index/empty.dir
    -rwxr-x--- 1 root root  45 Apr  5 21:55 /usr/local/ispconfig/server/conf-custom/install/empty.dir
    -rwxr-x--- 1 root root  45 Apr  5 21:55 /usr/local/ispconfig/server/conf-custom/mail/empty.dir
    -rwxr-x--- 1 root root 283 Feb  4 15:11 /usr/local/ispconfig/server/conf-custom/mail/welcome_email_en.txt
    -rwxr-x--- 1 root root 265 Feb  4 15:12 /usr/local/ispconfig/server/conf-custom/mail/welcome_email_fi.txt
    
     
  11. linus

    linus Member

    I also tried to make a new client with the correct limits from the beginning. No change. The strange thing is that the certs get written and after that they don't get written as they are not due yet, but Ispconfig falsely thinks LE and SSL is not enabled anymore.
     
  12. till

    till Super Moderator Staff Member ISPConfig Developer

    ISPConfig disables LE because the LE cert can not be found. Maybe LE renamed the cert. The cert has to be in this place:

    /etc/letsencrypt/live/DOMAIN/cert.pem

    where DOMAIN is the domain name of the website.
     
  13. florian030

    florian030 ISPConfig Developer ISPConfig Developer

    Did you installed the latest stable-branch? There is a check for le-certs that tries to connect to the domains. This fails on some systems.
     
  14. linus

    linus Member

    Thank you Till and florian030 for your time and help.

    ls -la /etc/letsencrypt/live/example.com/*
    lrwxrwxrwx 1 root root 39 Apr 5 21:31 /etc/letsencrypt/live/example.com/cert.pem -> ../../archive/example.com/cert1.pem
    lrwxrwxrwx 1 root root 40 Apr 5 21:31 /etc/letsencrypt/live/example.com/chain.pem -> ../../archive/example.com/chain1.pem
    lrwxrwxrwx 1 root root 44 Apr 5 21:31 /etc/letsencrypt/live/example.com/fullchain.pem -> ../../archive/example.com/fullchain1.pem
    lrwxrwxrwx 1 root root 42 Apr 5 21:31 /etc/letsencrypt/live/example.com/privkey.pem -> ../../archive/example.com/privkey1.pem

    I downloaded through sourceforge two days ago via the link on ispconfig.org. The installation files have the following dates:
    Code:
    /tmp/ispconfig3_install/install# ls -la
    total 6060
    drwxrwxr-x 8 root root     300 Apr  5 21:55 .
    drwxrwxr-x 9 root root     300 Jan 24 19:45 ..
    drwxrwxr-x 3 root root     160 Jan 24 19:45 apps
    drwxrwxr-x 5 root root     100 Jan 24 19:45 dist
    -rw-r--r-- 1 root root 6122352 Apr  5 21:54 existing_db.sql
    -rw-rw-r-- 1 root root   28019 Jan 24 19:45 install.php
    drwxrwxr-x 3 root root     140 Jan 24 19:45 lib
    -rw-r--r-- 1 root root     971 Apr  5 21:55 named.conf.options~
    drwxrwxr-x 2 root root      80 Jan 24 19:45 patches
    -rw-rw-r-- 1 root root    5280 Jan 24 19:45 setrights.php
    drwxrwxr-x 3 root root     140 Jan 25 17:27 sql
    drwxrwxr-x 2 root root    1640 Jan 25 17:27 tpl
    -rw-rw-r-- 1 root root    4093 Jan 24 19:45 uninstall-fedora.php
    -rw-rw-r-- 1 root root    4266 Jan 24 19:45 uninstall.php
    -rw-rw-r-- 1 root root   25385 Jan 24 19:45 update.php
    
     
  15. till

    till Super Moderator Staff Member ISPConfig Developer

    Try an update to latest git-stable with:

    ispconfig_update.sh

    choose "git-stable" as source. Git stable is not the dev branch, it is the stable code plus the bugfixes of the next release, so you don't have to worry that new untested features get installed.
     
  16. linus

    linus Member

    Thank you, I should have known git-stable was safe to upgrade to. Now I did and reconfigured services.
    I tought it was working because when I run server.sh manually the checkboxes worked like they should, but after a while they disappeared (probably after the cron jobs) as before.

    Is there anything else I could try or test?
     
  17. florian030

    florian030 ISPConfig Developer ISPConfig Developer

    Server.sh creates the LE-cert. If this fails, the checkbox is unset.
    Maybe check if domains are reachable to avoid letsencrypt verification errors is not working on your server (i have the same problem here).
    The file_get_contents('http://' . $temp_domain . '/.well-known/acme-challenge/' . $le_rnd_file)-call is not working (at least) with NAT.
    You can try to change the webserver-plugin in server/plugins-enabled. For nginx (~ line 1300) change it to:
    Code:
      foreach($temp_domains as $temp_domain) {
    //  $le_hash_check = trim(@file_get_contents('http://' . $temp_domain . '/.well-known/acme-challenge/' . $le_rnd_file));
    //  if($le_hash_check == $le_rnd_hash) {
      $le_domains[] = $temp_domain;
      $app->log("Verified domain " . $temp_domain . " should be reachable for letsencrypt.", LOGLEVEL_DEBUG);
    //  } else {
    //  $app->log("Could not verify domain " . $temp_domain . ", so excluding it from letsencrypt request.", LOGLEVEL_WARN);
    //  }
      }
    
    You should find the same code in the apache-plugin.
     
  18. linus

    linus Member

    Thank you for your help again! Now I removed the verification from /usr/local/ispconfig/server/plugins-enabled/apache2_plugin.inc.php

    Code:
        foreach($temp_domains as $temp_domain) {
                            //      $le_hash_check = trim(@file_get_contents('http://' . $temp_domain . '/.well-known/acme-challenge/' . $le_rnd_file));
                            //      if($le_hash_check == $le_rnd_hash) {
                                            $le_domains[] = $temp_domain;
                                            $app->log("Verified domain " . $temp_domain . " should be reachable for letsencrypt.", LOGLEVEL_DEBUG);
                            //      } else {
                            //              $app->log("Could not verify domain " . $temp_domain . ", so excluding it from letsencrypt request.", LOGLEVEL_WARN);
                            //      }
                            }
    
    
    But the problem is still there, and I think I haven't seen any clear problems in the /var/log/ispconfig/ispconfig.log (like "could not verify the domain") in debug mode.

    ispconfig.log
    Code:
     
    
    08.04.2017-11:02 - DEBUG - Verified domain example.com should be reachable for letsencrypt.
    08.04.2017-11:02 - DEBUG - Create Let's Encrypt SSL Cert for: example.com
    08.04.2017-11:02 - DEBUG - Let's Encrypt SSL Cert domains:  --domains example.com
    08.04.2017-11:02 - DEBUG - exec: /usr/bin/letsencrypt certonly -n --text --agree-tos --expand --authenticator webroot --server https://acme-v01.api.letsencrypt.org/directory --rsa-key-size$
    08.04.2017-11:02 - DEBUG - Enable SSL for: example.com
    08.04.2017-11:02 - DEBUG - Writing the vhost file: /etc/apache2/sites-available/example.com.vhost
    08.04.2017-11:02 - DEBUG - Processed datalog_id 16431
    08.04.2017-11:02 - DEBUG - Calling function 'restartHttpd' from module 'web_module'.
    08.04.2017-11:02 - DEBUG - Restarting httpd: systemctl reload apache2.service
    08.04.2017-11:02 - DEBUG - Remove Lock: /usr/local/ispconfig/server/temp/.ispconfig_lock
    08.04.2017-11:03 - DEBUG - Calling function 'check_phpini_changes' from plugin 'webserver_plugin' raised by action 'server_plugins_loaded'.
    08.04.2017-11:03 - DEBUG - Remove Lock: /usr/local/ispconfig/server/temp/.ispconfig_lock
    08.04.2017-11:04 - DEBUG - Calling function 'check_phpini_changes' from plugin 'webserver_plugin' raised by action 'server_plugins_loaded'.
    0
    
    Key factors so far:
    - Certificates are created correctly for every new site with the checkboxes
    - The https sites are working for the end users
    - In the log Let's encrypt tells (correctly) that the certificate is not due yet for renewal when checking again
    - If I run /usr/local/ispconfig/server/server.sh manually the ticks last ... until
    - ... the cron job makes the SSL and LE checkboxes emtpy (without proving any obvious help to the with error log) (also removing the SSL redirect option)
     
  19. linus

    linus Member

    Is there any other place that the letsencrypt field is updated to "n" than here: /usr/local/ispconfig/server/plugins-enabled/apache2_plugin.inc.php ?

    Because shouldn't I then see the log entry Let's Encrypt SSL Cert for: ... could not be issued? I don't see that in the log. And why does the cron job behave different than running server.sh manually?

    Code:
                   if(!$success) {
                                            // error issuing cert
                                            $app->log('Let\'s Encrypt SSL Cert for: ' . $domain . ' could not be issued.', LOGLEVEL_WARN);
                                            $data['new']['ssl_letsencrypt'] = 'n';
                                            if($data['old']['ssl'] == 'n') $data['new']['ssl'] = 'n';
                                            /* Update the DB of the (local) Server */
                                            $app->db->query("UPDATE web_domain SET `ssl` = ?, `ssl_letsencrypt` = ? WHERE `domain` = ?", $data['new']['ssl'], 'n', $data['new']['domain']);
                                            /* Update also the master-DB of the Server-Farm */
                                            $app->dbmaster->query("UPDATE web_domain SET `ssl` = ?, `ssl_letsencrypt` = ? WHERE `domain` = ?", $data['new']['ssl'], 'n', $data['new']['domain'])$
                                    }
                            //}
    
    The entries in the root crontab are
    Code:
    * * * * * /usr/local/ispconfig/server/server.sh 2>&1 | while read line; do echo `/bin/date` "$line" >> /var/log/ispconfig/cron.log; done
    * * * * * /usr/local/ispconfig/server/cron.sh 2>&1 | while read line; do echo `/bin/date` "$line" >> /var/log/ispconfig/cron.log; done
    
     
    Last edited: Apr 10, 2017
  20. florian030

    florian030 ISPConfig Developer ISPConfig Developer

    Are you using your own template in server/conf/conf-custom?
     

Share This Page