First -I have read through the Letsencrypt FAQ. The really cute part is that the base site certificate works fine, and I can generate certificates manually with certbot, so letsencrypt _in and of itself_ is fine. There's something glitchy elsewhere History for those unwilling or uninterested in reading the old post(s) on this topic. This is an old machine, and it's been upgraded several times over the years. It started having issues a long time ago, and I just worked around the problem, as it was only affecting the primary domain name -now it's affecting all the sites. (I think this machine started as an Ubuntu 12 system. It's never had a GUI) Here's where I'm seeing the errors for the site itself, when the auth tries to happen. [Wed Jan 01 19:30:13.293856 2025] [proxy_http:error] [pid 2017] [client 23.178.112.219:47335] AH01114: HTTP: failed to make connection to backend: 127.0.0.1 [Wed Jan 01 19:30:13.338552 2025] [proxy:error] [pid 2015] (111)Connection refused: AH00957: http: attempt to connect to 127.0.0.1:9999 (127.0.0.1) failed [Wed Jan 01 19:30:13.338577 2025] [proxy_http:error] [pid 2015] [client 23.178.112.213:50855] AH01114: HTTP: failed to make connection to backend: 127.0.0.1 [Wed Jan 01 23:35:05.613864 2025] [proxy:error] [pid 76208] (111)Connection refused: AH00957: http: attempt to connect to 127.0.0.1:9999 (127.0.0.1) failed [Wed Jan 01 23:35:05.613910 2025] [proxy_http:error] [pid 76208] [client 23.178.112.216:38729] AH01114: HTTP: failed to make connection to backend: 127.0.0.1 [Wed Jan 01 23:35:05.685907 2025] [proxy:error] [pid 76207] (111)Connection refused: AH00957: http: attempt to connect to 127.0.0.1:9999 (127.0.0.1) failed [Wed Jan 01 23:35:05.685940 2025] [proxy_http:error] [pid 76207] [client 23.178.112.217:60547] AH01114: HTTP: failed to make connection to backend: 127.0.0.1 -- This matches up with the errors in the logs (commented out domain name, but don't see a need to comment out IP) 2025-01-01 23:35:06,654:INFO:certbot._internal.auth_handler:Challenge failed for domain <customer>.com 2025-01-01 23:35:06,655:INFO:certbot._internal.auth_handler:Challenge failed for domain www.<customer>.com 2025-01-01 23:35:06,655:INFO:certbot._internal.auth_handler:http-01 challenge for <customer>.com 2025-01-01 23:35:06,655:INFO:certbot._internal.auth_handler:http-01 challenge for www.<customer>.com 2025-01-01 23:35:06,655EBUG:certbot._internal.display.obj:Notifying user: Certbot failed to authenticate some domains (authenticator: webroot). The Certificate Authority reported these problems: Domain: <customer>.com Type: unauthorized Detail: 216.74.247.27: Invalid response from http://<customer>.com/.well-known/acme-challenge/8_2JI46dr741msOPiY1oIaufH5U673eixaxNdmktczs: 503 Domain: www.<customer>.com Type: unauthorized Detail: 216.74.247.27: Invalid response from http://www.<customer>.com/.well-known/acme-challenge/WLSs-emHkw7jOfXzsdqe4MhUbUXxmj9pqU7gV385H44: 503 Hint: The Certificate Authority failed to download the temporary challenge files created by Certbot. Ensure that the listed domains serve their content from the provided --webroot-path/-w and that files created there can be downloaded from the internet. 2025-01-01 23:35:06,656EBUG:certbot._internal.error_handler:Encountered exception: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/certbot/_internal/auth_handler.py", line 90, in handle_authorizations self._poll_authorizations(authzrs, max_retries, best_effort) File "/usr/lib/python3/dist-packages/certbot/_internal/auth_handler.py", line 178, in _poll_authorizations raise errors.AuthorizationError('Some challenges have failed.') certbot.errors.AuthorizationError: Some challenges have failed. -- So, what I'm suspecting is that I have an _apache_ misconfiguration from an earlier upgrade. So, I have the 'proxy.conf' in mods-enabled, as well as the acme.conf in sites-enabled. (999-acme.conf there). acme.conf Alias /.well-known/acme-challenge /usr/local/ispconfig/interface/acme/.well-known/acme-challenge <Directory /usr/local/ispconfig/interface/acme/.well-known/acme-challenge> Require all granted <IfModule mpm_itk_module> AssignUserId ispconfig ispconfig </IfModule> </Directory> this is a duplicated section, sort of, in proxy.conf <IfModule mod_proxy.c> ProxyPass "/.well-known/acme-challenge/" "http://127.0.0.1:9999/.well-known/acme-challenge/" retry=1 ProxyPassReverse "/.well-known/acme-challenge/" "http://127.0.0.1:9999/.well-known/acme-challenge/" <Location "/.well-known/acme-challenge/"> ProxyPreserveHost On Order allow,deny Allow from all Require all granted </Location> (everything after this is commented out) So, what I'm hoping is that _someone_ here can point me to how to possibly get a working apache2 configuration, so that the certificates will start working. The system is due to be migrated to new hardware, but I'm trying to get everything up to the same level before I start trying to consolidate three ISPConfig systems into one (when they're all standalone right now) BW
That's not a file from ISPConfig, and this must cause any LE cert authorizations to fail as you redirect all traffic for cert authorizations to a service running on port 9999. This file must be deleted and Apache must be restarted if you want to be able to get LE certs again.
I'll give that a shot. I don't know where it came from, as I've never personally set up a reverse proxy on this system. It was all just upgrading with the standard ubuntu tools - but being now at 22.04, I've done it a number of times, and each step is a doozy. It even uninstalled roundcube rather than upgrade, so I had to reinstall it immediately thereafter. Thank you.
That definitely cleared out the proxy issue. Now, what I'm seeing is this from the debug log. "detail": "216.74.247.27: Invalid response from http://www.<customer>.com/.well-known/acme-challenge/j1P6df0Dn8bF0JQmQuivyZAzJ9WbLV14QPRYljYR93U: 404", I tried poking the hello.txt and the empty.dir that are in the /usr/local/ispconfig/interface/acme/.well-known/acme-challenge/ folder, and received the same 404 error. So, I'm pretty sure I'm still seeing a configuration problem somewhere in apache, as the logs show this for letsencrypt. 2025-01-02 10:18:05,471EBUG:acme.client:Storing nonce: LPSR-4-sls3aoiNttgVem7J8ibe0o0wouVrddwM5iJqz0ChvDuY 2025-01-02 10:18:05,472:INFO:certbot._internal.auth_handlererforming the following challenges: 2025-01-02 10:18:05,472:INFO:certbot._internal.auth_handler:http-01 challenge for <customer>.com 2025-01-02 10:18:05,472:INFO:certbot._internal.auth_handler:http-01 challenge for www.<customer>.com 2025-01-02 10:18:05,472EBUG:certbot._internal.plugins.webroot:Creating root challenges validation dir at /usr/local/ispconfig/interface/acme/.well-known/acme-challenge 2025-01-02 10:18:05,473EBUG:certbot._internal.plugins.webroot:Creating root challenges validation dir at /usr/local/ispconfig/interface/acme/.well-known/acme-challenge 2025-01-02 10:18:05,474EBUG:certbot._internal.plugins.webroot:Attempting to save validation to /usr/local/ispconfig/interface/acme/.well-known/acme-challenge/pkAzgGXqGkpK5iXyH1Am8A Zp2DzwePr6JXh_MuIWocM 2025-01-02 10:18:05,476EBUG:certbot._internal.plugins.webroot:Attempting to save validation to /usr/local/ispconfig/interface/acme/.well-known/acme-challenge/j1P6df0Dn8bF0JQmQuivyZ AzJ9WbLV14QPRYljYR93U 2025-01-02 10:18:05,476EBUG:acme.client:JWS payload: b'{}' 2025-01-02 10:18:05,488EBUG:acme.client:Sending POST request to https://acme-v02.api.letsencrypt.org/acme/chall/17103277/454474279995/NWQpgQ: And I'm seeing no errors there. My conclusion is that the process is running, but the redirect for .well-known isn't actually redirecting. I may need to see what a new clean build has, unless someone else has an easier suggestion. I certainly don't blame ISPConfig for this. It's just years of upgrades rather than new installs.
Your system shall have this file: root@server1:~# cat /etc/apache2/sites-available/acme.conf Code: Alias /.well-known/acme-challenge /usr/local/ispconfig/interface/acme/.well-known/acme-challenge <Directory /usr/local/ispconfig/interface/acme/.well-known/acme-challenge> Require all granted <IfModule mpm_itk_module> AssignUserId ispconfig ispconfig </IfModule> </Directory> and it must be enabled: root@server1:~# ls -la /etc/apache2/sites-enabled/999-acme.conf lrwxrwxrwx 1 root root 38 Sep 22 08:16 /etc/apache2/sites-enabled/999-acme.conf -> /etc/apache2/sites-available/acme.conf
Yes. that is there - I mentioned it in my first post. I did verify that your code is identical to what I have. The apachectl configtest verifies no syntax issues. No errors on startup in the main logs. Suggestions on where to go next, other than building the entire machine from scratch, or just migrating the system? (I'm concerned that migration would carry the problem over) I despise problems like this. I've been working with Linux for 34 years now, and upgrades are never fun, and I've even chewed out project developers for the attitudes "Oh, nobody would possibly be using older programs, because it's insecure!" (Not here. That attitude, however, is why so many servers had issues when they upgraded from 2.2 to 2.4, because they failed to actually document _changes_ that would happen. Going from Apache 1 to 2 was a cakewalk by comparison. Those folks actually put effort into it.)
Check the vhost file of the affected website and also any .htaccess files in that site if anything there redirects to the .well-known/ folder. Also, grep trough all files in /etc/apache2 if there are other configs that might affect accessing .well-known or .well-known/acme-challenge
Btw. For testing this is useful: touch /usr/local/ispconfig/interface/acme/.well-known/acme-challenge/acmetest.txt Now you can test it without trying to create a cert, just use: http://yourdomain.tld/.well-known/acme-challenge/acmetest.txt in your browser to test if acme validation is working again.
I did try that - i have a 'hello.txt' in there, and another file as well. I'll check again when I return. It seemed like the redirect (the acme part) is just being ignored. I'm going to dig into the full http logs later.
Since you are using certbot and ubuntu, but I am not sure what version you are using, also not sure any causes of your issues yet, I would like to know how you install that certbot and its version? I find that apt install for certbot is not working well in mine, so I removed it and reinstall via snap.
Currently Ubuntu 22.04, and certbot 1.21.0-1build1 I don't think it's a problem in certbot itself, or in letsencrypt. I think it's Apache related.
All right. For whatever reason, the server itself issues a 503 when trying to find files in the challenge folder, but the main thing I'm coming away with is that for whatever reason, the acme.conf is _not_ being acknowledged or used. The individual domain logs show accesses to their own folders, which of course are 404. So, it's definitely an Apache issue. Now, here's the diff for my apache.conf file and the package distribution .conf file. Code: apache2.conf:IncludeOptional mods-enabled/*.load apache2.conf:IncludeOptional mods-enabled/*.conf apache2.conf:IncludeOptional conf-enabled/*.conf apache2.conf:IncludeOptional sites-enabled/*.vhost apache2.conf.dpkg-dist:IncludeOptional mods-enabled/*.load apache2.conf.dpkg-dist:IncludeOptional mods-enabled/*.conf apache2.conf.dpkg-dist:IncludeOptional conf-enabled/*.conf apache2.conf.dpkg-dist:IncludeOptional sites-enabled/*.conf The only thing I can think of is _maybe_ needing 'sites-enabled' to be '.conf' rather than '.vhost'. I'm going to give that a shot and see what happens.
Reading through, it looks like it shouldn't matter at all; renamed the acme.conf to acme.vhost, just as test, and after reboot, same behaviour.
Now, there is _this_. It's the contents of the 000-default.conf file - which was created by ispconfig, IIRC. I may disable it and see what happens. (note the letsencrypt at the end)
A couple of things. Apparently ISPConfig, over the years, has changed file names - but the script never looked to see if those previous names existed. My constantly upgraded system, since 2014, no less, has a 2015 symlink of 000-ispconfig.vhost to an updated in 2017 000-ispconfig.vhost (in available). There's also a 000-ispconfig.conf that symlinks to ispconfig.conf - from 2023. The other is that all that proxy information that I was told to remove, because it wasn't ISPConfig? It was. It's installed as part of the Rspamd move. Apparently Rspamd _requires_ proxy. The relatively clean machine I was working on for migration is packed with 'proxy'. In fact, this page - https://www.howtoforge.com/replacing-amavisd-with-rspamd-in-ispconfig/ - says "we will create a website in ISPConfig and add a proxy configuration". The question becomes, why did the ispconfig installer add it to a machine that didn't have rspamd? No telling. In any case, I'm still certain it's an apache misconfiguration somewhere, and I'm attempting to cross-compare between the 'new' machine and this one, but it's a slog. Trying to decide which files should be retained isn't fun, and this is a production system.
Stripping out everything from the sites-enabled and available, then reconfiguring ispconfig on the 'clean' system, it regenerated just two files - the acme.conf, and the ispconfig.conf files, both linked to 999 and 000 respectively. Something I noticed - acme.conf demands the module 'mpm_itk_module'. This isn't installed as part of the setup, that I can see. that requires this line: sudo apt-get install libapache2-mpm-itk Now, I really don't know how it's used, just that it's there. Still bashing my head against a wall.
This file was not created by ISPConfig, nor is it managed by ISPConfig. ISPConfig cleans up its files properly. There must be a file 000-ispconfig.vhost and a 000-ispconfig.conf in sites-enabled that links to a file (without the 000-) in sites-available. I hope you did not installed libapache2-mpm-itk as it should not be installed and will break your system. The lines in acme.conf about mpm_itk_module are not for any normal setup, they are for some very old special setups and just a workaround. So do not try to install mpm_itk_module. The installer did not add it to your system. You now start to mix up all kinds of unrelated things. You read there is about adding a website in ISPConfig to access Rspamd GUI; it's not the proxy file you removed, which was not from ISPConfig as I mentioned already. And the ISPConfig installer never adds websites btw.
That is longer than I have been using Linux. In fact, it is longer than Linus Torvalds has been using Linux. Linux Initial release September 17, 1991; 33 years ago I have been using Linux since 1993. Or maybe 1992, I would need to check dates to be sure.
it's 2025. I've been working with Linux since University.. Okay, so I was working with UNIX to start. Linux came two years later, if that makes you happier. ULTRIX, specifically. The enormous stack of floppies you had to create to just get it online enough to reach an FTP server... However, I've been using it as a sysadmin, not as a developer/programmer. I can _fix_, but not actively create from scratch. My programming days were long long ago, and only as part of learning (FORTRAN and PASCAL). Till - I beg to differ. The machine with all of the proxy installs _from the base_ was installed using the ispconfig installer. https://www.howtoforge.com/ispconfig-autoinstall-debian-ubuntu/ Apache was installed from the get-go with it, by your installer, so it shouldn't be getting called by acme. That's the machine I'm using to try to cross compare to figure out why THIS machine is ignoring the acme section. And that's what is happening - the vhosts are all working, roundcube works, but the global 'everything .well_known should go here' is being ignored. the mpm_itk does coexist with mpm_prefork, apparently. It waits until called (I did spend half an hour doing comparisons). It had no effect on this problem either way. I did wipe out the apache sites-available and enabled folders on the test system (had no vms) and re-ran the installer with --force, and it ended up with the following. 000-default.conf acme.conf apps.vhost default-ssl.conf ispconfig.conf -- ispconfig.vhost was not regenerated ---- Again, on an existing system, is there a way to rebuild the sites-enabled/sites-available folders using the installer, so I can just remove all the old files and try building anew? I do understand I'm a very niche case, with this being a decade old install, but it is very frustrating. (probably more so for you, in many ways)
