Cannot assign valid SSL cert for new install.

I have major annoying issue with a fresh install of Debian 10, Apache2.4, BIND9 DNS, PureFTP, Postfix, MariaDB, Dovecot, RSPAMD and have been unable to get a valid cert assigned and always falls back to self-signed-cert. I have never lost access to to the control panel via port 8080. Firewall has TCP 80 & 443 with TCP/UDP 53 open to primary server as I am the SOA for my domain. I can create the "test.txt" file using the "touch" command and access it from outside my network via FQDN displaying a blank page as it should. Then I run "ispconfig_update.sh --force" select defaults until the section to "create new SSL certificate" where I select "yes" and it ultimately says the same error every time "server":Verify error:Fetching Connection refused". Log file shows the token is created, the host is verified, but when it attempts to write the token it shows "retrying post" repeatedly. I first get a trigger validation code=200, the log file says the CA is processing my order, then a couple more post retries with code=200, then it skips DNS _on_issue_err, retries one more post, then gives error=400. That's when it falls back to a new self-signed cert. I am going nuts trying to figure this out and any help or feedback would be appreciated. This is literally the only thing prevented me from going live. I can provide the log file if needed.

 

Common issue lately I supposed. Nothing much for all of us can do to help except to hunt for the cause via log files. I always suspect IPV6 issue or behind NAT router issue. Other than that, I am not so sure why. I use dns challenge and create cert before installing ISPConfig, so I do not know much about the LE cert issue that most of you were facing during ISPConfig install or update.

 

As an FYI, it won't let me post if I put anything in here referencing even a false link.

I am getting ready to try choosing the nightly build when I verify my DNS A, NS, SOA records are fully propagated (with some exceptions, I am sure). I am behind a FW/NAT device, but have TCP 80, 443, and TCP/UDP 53 open to internal IP. Only issue is opening ports to IPv6 addresses. Since I cannot do that at this moment until I have my commercial inet connection and new FW installed, I removed the AAAA records (at the recommendation of SCHAAL IT support) which did fix an access issue when using DNS resolution since almost all browsers, apps, and API's will attempt to use IPv6 if DNS returns a valid AAAA host record. And you are correct about my initially setting this up before I had any ports opened on my FW, but that was done knowingly because I assumed this would not be an issue down the road as stated in the documentation. Freaking hindsight. With that said, my hosts file on this server (node01) has total of 5 lines that reference itself, see below:

127.0.0.1 -> localhost.localdomain -> localhost
127.0.1.1 -> node01
(actual pub IPv4) x.x.x.x -> node01 with domain -> node01
(actual IPv6) 2xxx:x:x:x:x:x:x:x -> node01 with domain -> node01
::1 -> localhost ip6-localhost ip6-loopback

From browsers (and other apps/utils) outside my network (the internet), you can reach, see below:
node01/.well-known/acme-challenge/test.txt (I issued the touch cmd to create test.txt)
Websites I have created but I have an "Under Construction" HTML.
The websites are accessible over port 80 or 443. You just have to accept the security risk because of the self-signed cert issue.

With me having a cert issue, I will not go live until I have a valid certificate for email services, secure transactions for billing purposes, etc.

Before I perform the ispconfig_update.sh --force using the "nightly" build this time, do either of y'all see an issue or have any thoughts with the above hosts file config, the way I have my DNS configured with the absence of any AAAA records, or me being behind a FW/NAT device? I also have tried selecting the "Skip Lets Encrypt Check" in the server configuration but it made no difference.

 

It worked this time! I just had to change it from "stable" to "nightly" build!!!! Are you kidding me? Thank you very much Till! You too Ahrasis. I got the "touch" command from one of Ahrasis' posts and Till's recommendation to use the "nightly" build. Teamwork, thank you very much you two! This has had me running in circles and banging my head. It made no sense to me why only this unbelievably specific module wouldn't function when everything, I mean everything else works as they should. Thank you again!

 

Thank you for that update, Th0m. Now I just have to request valid certificates for each of the client websites, DNS zones where I am the SOA, and my Postfix/Dovecot mail server that will be hosting some or all of my clients who want my server(s) to host their email. I am not wrong when saying ISPConfig 3.2.5 can extend and provide this service to each and every client as the documentation suggests, correct? I do not want to say I can offer secure email services for each client if I cannot. It has been my understanding I can obtain LE wildcard certificates for customer websites and a valid certificate for my mail server(s), correct?

 

Thank you for your response Th0m. I have a DNS CAA record with LE as the CA for my domain and so do the client DNS zones without "Use Wildcard SSL" checked. To be clear, in the DNS Zone for my company hosting client websites and email for them and myself, I have the MX record for my domain pointed to "mail" then dot domain name. then a A record for "mail" pointed to the correct IPv4 address and when I check external DNS and certificate status for that MX host/domain name, it checks out. But for the client sites and DNS zones do I need to put either an MX record or an A record pointing to my domain? I wasn't thinking I do, but when they are sending and receiving email that have to be digitally signed they would need a DNS DKIM TXT record and a DNS SPF record in their DNS Zone, correct? Or do the client DNS zones not need MX records at all? Hopefully I am explaining this properly.

 

Thank you again, Th0m. All looks good on my side, so I am opening proper SSL/TLS ports in my firewall for IMAPS / SMTPS / POP3S. I do not want to allow non secure email connections, so I am only opening TCP 465, 587, 993, 995 which should cover all three protocols. And last question then I'll try and leave you be. I have noticed only loopback interfaces are listening for BIND9's "secure DNS" port 953 (when the industry standard DoT is 853 for TLS and 443 for HTTPS). I understand TCP 953 is in use because of BIND9's DNSSEC being enabled. However, in addition to DNSSEC, is there a way to lock down ISPConfig 3.2.5 DNS even further by only allowing secure DNS connections/transactions over TCP/UDP 853/443 instead of TCP/UDP 53 on interfaces other than the IPv4 and IPv6 loopback address? Thank you again for all your guidance.

 

Thank you Th0m. I still have an LE problem that one of y'all may be able to point me in the right direction. First off, I opened port 25 and 110 on my Internet FW/NAT device to my primary mail server for both SMTP and POP3. I am on the fence whether or not to keep pop3 service enabled and would rather my customers use IMAP when I am hosting their email. Customers who are heavily invested with and currently use Microsoft's cloud-hosted email services with their Office 365 exchange service, I'll most likely get the ISPConfig MS Exchange MTA module installed and configured so I won't lose potential customers because I don't support that feature. I will be supporting/integrating cloud based installs in the next phase anyway.

But to the certificate issue I am having is not with my main server (node01), but with my secondary server (node02). It still has a self-signed certificate. Even though the MariaDB/MySQL setup is a Master/Master and ISPConfig 3.2.5 is configured in a multi-server setup supporting the exact same services (with exception of the ISPConfig panel), a new SSL certificate for the second server (node02 DOT domain name) cannot be issues when running ispconfig_update.sh --force" (selecting "nightly" build option instead of "stable") because this server doesn't have the ISPConfig Panel enabled on it. The Master/Master database configuration appears to work properly. I've tested and verified within mysql -u root -p , enter the PW, then verify both statuses on both servers with "SHOW MASTER STATUS /G" and "SHOW SLAVE STATUS /G". Both reflect correct Master_Log_File and Read_Master_Log_Pos of each other. Whenever I try to authenticate to the email server using any client, there's two errors in the System log from "dovecot: doveadm (user@domain-name) stating "received an invalid SSL certificate: self signed certificate: /C=XX/ST=XX//L=City/O=Company Name/OU=Department/CN=node02 DOT domain name.

That seems to tell me node02 is still using the self signed cert. Is there any way to request an LE SSL Cert for my second server without using the "ispconfig_update.sh --force" selecting "nightly" like I did the primary node01? When I run the update command, it knows it doesn't run the ISPConfig Panel so it just says SSL certs already in place and moves on to the cert SYMLINK process and finishes up. Everything I seem to read say DO NOT request a certificate via command line, only via the update command or in the Panel, but I don't find anywhere to request a valid SSL cert for a second node or it's mail server. What am I missing here?

Mikeal

 

I haven't visited ISPConfig code for quite long time but I am sure last time I checked they were designed to create LE certs during install or update using webroot for web server or standalone for non-web server.

Are your servers behind nat router or using port forwarding or proxy etc?

By the way, though not advisable, actually you can request your own certs manually but the renew conf will then need to be modified accordingly so it will work with ISPConfig setup.

I used certbot with dns-challenge to create LE certs for my server and all its services and they are automatically updated after 60 days.

 

My primary server with the ISPConfig Panel (port 8080) finally was able to receive a valid cert the other day without me changing any configurations. The only thing I did was when running "ispconfig_update.sh --force" was change from downloading the [STABLE] image to the [NIGHTLY] image and the cert request was instantaneous. Now my websites and a client website as well as the "host panel over port 8080" work beautifully. I was even able to setup HTTPS rewrite so the website will always come up as HTTPS instead of HTTP. Being behind a FW and NAT device I know is not an issue, Network Engineering in routing/switching, Data Center, Compute, Wireless, Security, IPT/UC, etc is my specialty for the past 20+ years. In Linux I am not an expert but pretty good. My main issue is only having one public IPv4 address with my residential line until my business internet connection with plenty of IPv6 and IPv4 addresses is installed. This is why I am directing all my traffic to only the Primary server. Which works fine unless some function/module/plugin calls the second server. I think I just found it, too. There is a "dovecot_custom.conf.master" file which is called first and can overwrite the "dovecot.conf" file in directory "/etc/dovecot". At the end of "dovecot_custom.conf.master" file loads up the "ssl_client_ca_dir", "service_replicator", "aggregator", "listener", etc and at the last few lines, it specifies the "inet_listener" port and to use SSL with the next line being "plugin { mail_replica = tcps:node02 DOT domain name} So, now I know why node02 is being called so quickly when I try to authenticate. The question is do I comment it out temporarily to verify my email functions properly or find a way to have a valid certificate with the same External IP address assigned to node02 until my business circuit is installed with enough IPs for all the servers I am going to have.

 

It is kinda solvable issue but you may need some methods to properly setup and manage multiple ISPConfig servers behind a nat router because you may need to share port 80 currently used by your primary ISPConfig server with other servers in order for them to obtain LE certs for the servers as well as websites under them, if any.

I think your case is almost if not actually similar to @Chris_UK setup for his multiple ISPConfig servers, so I know such setup is possible and can be worked out.

 

I agree it is solvable with a certain setup, which I am waiting for a few devices to come in so I can perform SSL offloading, L3-7 load-balancing, In-line IPS, and NGFW. I Just need one full setup, I should've done this POC with only one server, which tomorrow I think I'll build one standalone and migrate all my data over to it with a "HOT STANDBY" in the event of a failure until my business circuit is in which will release the funding for all the devices I need instead of fighting a design that is going to change within a month anyway. This is strictly a POC to prove all the features I proposed operate with an incredibly small carbon/real estate footprint that will provide a whole lot of services. With my new IC engineering specs.Thank you again. You helped me make a important decision. I am going to finish up some documentation and prep this to go live tomorrow on a single host. I added too much complexity from the getgo when I did not need to. HA and redundancy was not the goal of the proof of concept, I ingrained network and systems fundamentals when I didn't need to. Thank you again, though. Between you, Th0m, and Till, I picked up a lot of info for problems that could arise much farther down the road if they even do.
Mikeal

 

There is no need to set up with just one server Thanks for the tag Ahrasis.

You actually have a couple of options, i've used both. A reverse proxy, seems like you have something in mind for,
Another option that I have used is what I call Acme sharing. Any server that does not have a public web interface (because as you will realise without a reverse proxy you can only have one on standard ports) can use an NFS share instead.
You set up the share on the public facing server, you want to share /usr/local/ispconfig/interface/acme

On the server that is requesting the cert, you run the installer in interactive mode and then when you get to the cert prompt you open another terminal and mount the share to the same location. You can't mount it before this point because its only just before the cert request that it gets created.

Once you have mounted and verified you can write to the share, send the installer back on its way. Bad for auto renew though. I am working on a solution to that so the share gets mounted just in time for the cert renewals and unmounted right after.

The reason for the dismount is its not strictly safe, I could only do it with a no_root_squash flag which means root has privileges on the share from the mount side. This is bad because it can lead to an exploit where the root account could be used to breach the sharing host.

Without the flag the root user on the mounting side is "squashed" to an unprivileged level, but that means that the ISPC installer can't write to the mount so cert generation will fail.

It all comes down to user:id mapping. If lets say "bob" on both ends of the share had the same user id and bob can write to the share from the host side, the bob can write on the client side, if they mismatch on any part, user id or user name then they can't write to the share.

Anyway, sorry about the ramble. I thought I would give you a little detail on the downside to using a share in this way. It can be used but at the moment I have only achieved this running manually.