LE certs for secondary servers.

Discussion in 'Installation/Configuration' started by Chris_UK, Jul 24, 2021.

  1. Chris_UK

    Chris_UK Active Member HowtoForge Supporter

    There is work around in this thread to obtain certificates on secondary servers that are unable to use one of the standard methods of LE.

    I have 5 VM's running on a host.

    ISPC panel with internal network access only.

    DNS is hosted by my domain provider so the bind server isn't actually doing anything at the moment other than holding the records that ISPC generates for when I decide to move things up a notch.

    I want to to get certs issues for dns, mx and db but the certs won't generate because there is no way for LE to perform its check as webserver picks up any requests, so for example, dns server generates a request, web server responds but doesnt have the corret credentials for the response. Result Failure.

    I could manually use dns record acme but with propagation times and a lack of free time in general, this isn't a good option either.

    So my question is this. Is there a way to hand this off to web server and have that pass the cert back to the dns server(read any secondary non web server)?
    Last edited: Jul 29, 2021
  2. Th0m

    Th0m ISPConfig Developer Staff Member ISPConfig Developer

  3. Chris_UK

    Chris_UK Active Member HowtoForge Supporter

    You mean create a site for using the mx host name? It's a separate vm and the mx vm is making the call to le so i'm not sure that will work?
  4. Th0m

    Th0m ISPConfig Developer Staff Member ISPConfig Developer

    I'm not sure if I am misunderstanding you or you are misunderstanding me/the system.

    The easiest way to retrieve a cert is obtaining it for a "site" and symlinking your software's (e.g. Postfix or Dovecot) certificate file to the certificate of the site. Web traffic for mx.example.com will go to the IP it is pointed to.

    Every VM needs it's own public IP, which is because of things like this. Using one IP and forwarding the designated ports to the correct VM will lead to more issues.
  5. Chris_UK

    Chris_UK Active Member HowtoForge Supporter

    Yes i'm discovering.

    The vm only runs a mail server so it works with port forwarding hence I can run several vms and only forward the required ports. so anything that comes into my router gets where it needs to.

    The problem here is that I dont have different public ip addresses. So to get a site cert and use it for mx, I would have to be running mx on the web server.

    To be honest my best approach here would be multiple ip addresses from my provider but that requires submitting justification, I could do that but no guarantees on it being approved. Another option would be to go back to a single server approach, its manageable but I just don't like the idea of a single vm crash taking everything down with it.

    The option i'm looking at for now is to hand off the request to the webserver so it can request and respond as required. Its probably going to need a look into the le code to see if it can be done.
  6. Th0m

    Th0m ISPConfig Developer Staff Member ISPConfig Developer

    I would strongly recommend you to use a VPS provider like Hetzner, which offers cheap hosting of virtual servers with their own public address. You can follow the multiserver guide to set it up.
  7. Chris_UK

    Chris_UK Active Member HowtoForge Supporter

    To be honest with you my whole use case for ISPC is a little unusual. Let me explain.

    Everything I do I could have don't without ISPC in the mix at all, I could have created the sites without it, installed certbot as a standalone, so on and so on, even the mail server, dns etc I could run standalone and I have done all of these in the past on VPS.

    I don't host websites for other people (not for some years anyway) it's all for my own personal usage with the exception of running a website and mail for my fishing business.

    So the reason I use ISPC is its free and makes things much simpler to manage, no need to manually config everything or write scripts for it. I want to do some web development for a certain platform.. no worries just throw up a new site on a subdomain of one of the two domains I currently have at my disposal. Besides my ISP is pretty cool about what I can do on my residential connection.

    With all that I saw some dirt cheap servers on ebay and after weighing up the cost of buying and running these servers (Less than £200 buy 2 reasonable spec servers, around £10 a month to run in electricity) I decided it was well worth it. I was already at that point paying out in excess of £15 per month for VPS to do what i'm doing now.

    So now I have a home lab, couple of hp 360 & 380 G7's cisco router and switch and a small 12u cab. All in this cost me more than I would be paying in three or four years of hosting but at the point I got so wrapped up in the possibilities that it made sense. My electricity usage is a little higher with the added hardware but its still cheaper than what I would be paying out to get the specs of these servers in vps form.

    I could COLO the servers and solve this but that's a considerable cost for what I get out of it. Plus my servers are in a different location to me which mean I have to travel if i can't fix an issue remotely.... also... I'm cheap.

    And here we are today, I am looking for a work around to a problem that most people will never face because they are willing to shell out a little extra cash... Did I mention I am english and spending lots of money is like cutting off an arm, haha.

    Anyway with the back story out of the way, here is what I see happening and I might yet write a module to handle this.

    • Secondary calls Primary - Hey man, get me a cert.
    • Primary has the required info of secondary because it's a know all- Hits up LE for a cert.
    • LE knows nothing of my weird setup and tells the primary to create the file which it does.
    • LE verifies the file and a cert is sent to primary.
    • Secondary for the nth time calls home - Hey man, got my cert yet.. Yes replies Primary
    • Secondary transfers the cert to itself and finally we have a winner.
    Now I know that is a little simplified but I would imagine completely do-able. See this is what happens.. the rabbit whole just opened up and its got me!

    Another thought just occurred to me.. I could just as easily run these private dns servers properly. That is if ISPC handles the changes to dns automatically.
    Last edited: Jul 25, 2021
    ahrasis likes this.
  8. ahrasis

    ahrasis Well-Known Member

    Seems your case is like mine except I am not running server from a real server hardware just converted gen4 i5 PCs.

    The default ISPConfig can already issue LE certs for all type of server on update or install as it use either standalone or webroot, so no web server is needed for other types of server.

    The request for LE server may have failed because that five types of server you have are all behind one public ip as you might have guessed.

    In.my mind the solution is either use dns challenge as you mentioned earlier or create one server as a proxy to all server but your module is also doable to me.

    I personally use the former via cloudflare dns server as to me it is the easiest thing to get LE certs via dns challenge.

    About the last part of dns thingy that you mentioned, I am not that clear, are you using public dynamic ip because if you do, normally it won't be possible to run mail service except via relay and it will also be very hard to run a public dns server on your own though it may seem possible.
  9. Chris_UK

    Chris_UK Active Member HowtoForge Supporter

    Hi ahrasis, I am glad to have somebody who knows my plight.

    I have a public static IP from my ISP. I just looked into a block of IP's again and they no longer support them on residential services which I am somewhat disheartened by, though this isn't really a problem I can switch to a business service. The cost is surprisingly no different, what is different is the support level and that you can get a block of IP's with justification. I suppose I have the justification with the requirements to get SSL.

    I used to run a dns server behind my providers router, it worked just fine except a weird case where the router rebooted if i made a certain dns query from within the router.. I dont know why, something about it the router didnt know how to handle and fritzed out I suspect with it being a consumer grade router. This prompted me to get the cisco routers in the first place and the problem went away.

    The reason I swapped to using the domain providers dns hosting was because I didnt have the required single ip per dns server, it made sense at the time but requires manual configuration, this of course breaks LE's dns method for auto renew so maybe I should just switch back to private dns and let ispc take care of the dns record. (I assume it does that)
    Last edited: Jul 25, 2021
  10. Chris_UK

    Chris_UK Active Member HowtoForge Supporter

    Actually as an aside/hope and prayer... does ISPC/LE have anything pre-baked that can work with Enom?
  11. ahrasis

    ahrasis Well-Known Member

    Then I think you should not have any problem running dns server or mail server using ISPConfig. You may have an internal backup / secondary dns server for your primary dns server or you may have it elsewhere like at Afraid.Org FreeDNS.

    Try looking at WHCMS and its ISPConfig plugin that I think should work with ENOM.
  12. Chris_UK

    Chris_UK Active Member HowtoForge Supporter

    Yeah whmcs is going to be a big no, their current repricing structure has put pay to that. I has their owned license and was able to buy an annual upgrade license for a reasonable amount so it was good. Now they still technically support my license but I cannot buy an upgrade license. So with that I am stuck with the last version available when my upgrade license expired. That puts me somewhere in the 5.x range if whmcs and thats not going to work for me. I don't want the latest and greatest but I do want function.

    Seeing as ISPC is comparable in its function to cpanel, maybe we should be looking as a community to work on the "client" side of things, what with whmcs seeming to want to price people out of the market.
  13. ahrasis

    ahrasis Well-Known Member

  14. Jesse Norell

    Jesse Norell ISPConfig Developer Staff Member ISPConfig Developer

    Note I haven't gotten through this whole thread yet, but two ways to solve the initial issue would be: 1) use a shared mount point for /usr/local/ispconfig/interface/acme/ so that when the letsencrypt client on the mail server creates a verification file, the webserver immediately has access to that same file and can serve it to the letsencrypt servers during the verification request, or 2) setup DNS based verification for your mail server's certificate(s) (manually, ISPConfig does not support configuring this currently).

    A third option might be to setup a vhost in the web server with the server name requested by the mail server and reverse proxy the request to the mail server (the letsencrypt client on the mail server could use standalone mode). This would actually rely on the fact that the reverse proxy config (at least for nginx) currently forwards the acme-challenge requests, which is the subject of a fix/enhancement. (This may be a good use case for keeping the current functionality available.)
    Last edited: Jul 26, 2021
    Chris_UK likes this.
  15. Chris_UK

    Chris_UK Active Member HowtoForge Supporter

    2nd option shared mount sounds just what I need. That's actually quite genius. I will look at that on my next day off.
  16. Chris_UK

    Chris_UK Active Member HowtoForge Supporter

    This guide will allow you to obtain an SSL certificate from Lets Encrypt for secondary servers that are prevented from doing so. or at least it's made difficult through the normal options. For my use case I have available a single public primary server/IP address and use port forwarding to access secondary servers, my DNS is not managed by ISPConfig but manually at my registrar.

    Difficulty - if you have a basic understanding of using the Linux command line then this will be very easy to configure. I really wanted to come back to this with an automatic script but that would have required digging into the installer and multiple servers in the mix makes that more difficult. So I come as close as I can with that and have plotted the order things should be done and where to pause the ISPConfig installation process.

    You should already have a propagated host record set up with your DNS provider for the hostname of the secondary server. You likely will not need to use this method if ISPConfig is handling your DNS.

    This should be performed on a fresh installation of an ISPConfig secondary server.
    When attempting this on an existing mail server i managed to baulk the certificates and was receiving mismatched cert errors in the postfix logs, maybe I could have redone it all by removing all the certs but that was not how i "fixed" the issue. I reinstalled the mail server after backing up all of vmail with rsync, I sent them back later after I had migrates all of my mail accounts to the new server. For that reason I recommend a fresh install.

    Tested on:
    A fresh installation of an ubuntu 20.04 ispconfig 3.2.5 as a secondary server. This should work in any version that has the automatic ispconfig installer.

    Finally a note on security: no_squash_root in the NFS setup of Host is not secure and should not be used where a share can be accessed from outside of a private network, doing so can allow root access to your servers. If you ispconfig user ids match on all servers you should remove that option from the configuration.

    is a publicly accessible web server with Lets Encrypt installed and fully functional.
    Client a fresh installation of ubuntu 20.04 using the ISPConfig 3.2 perfect multi server guide.
    IP Addresses place holders and should be adjusted as required.

    Host: | Setting up NFS
    sudo -s
    apt update
    apt install nfs-kernel-server
    echo '' >> /etc/exports
    echo '# Share the acme directory with the entire .1 subnet to allow secondary servers to hand off the acme challenge to this server. >> /etc/exports
    echo '/usr/local/ispconfig/interface/acme,no_root_squash,sync,no_subtree_check,crossmnt,fsid=0)' >> /etc/exports
    echo '# limit access to the acme shared directory to specific hosts, one ip address per block' >>
    echo '/usr/local/ispconfig/interface/acme,no_root_squash,sync,no_subtree_check,crossmnt,fsid=0),no_root_squash,sync,no_subtree_check,crossmnt,fsid=0)' >> /etc/exports
    # This should give not output unless there is a problem in the file.
    exportfs -ar
    # You should see output showing your new shared directory.
    exportfs -v
    # Create a file for testing later.
    touch /usr/local/ispconfig/interface/acme/host-test
    Firewall: Allow access to your NFS share only from designated IP/Range.
    ufw allow from to any port nfs # A single IP
    ufw allow from to any port nfs # Entire .1 subnet
    ufw status
    Client: or any in the .1 subnet if you used that line
    sudo -s
    apt update
    apt install nfs-common
    Using the ISPConfig perfect multi server guide (link at the end of this post) install the latest stable release. Follow the installer for the server you are installing (page 2+) until you reach the question about the ssl certificate and stop, do not answer the question yet, the installer will wait. Minimise the installer terminal for now, you will come back to it later.

    Mount the share from your web server. It is perfectly safe to mount over an existing directory, mounts only hide the existing directory, nothing is lost, it's just inaccessible while the mount is active.

    Client: | In a new terminal/putty, I will identify this as terminal 2 from here.
    sudo -s
    mount /usr/local/ispconfig/interface/acme

    Client: | (Terminal 2) & Host:
    # Client
    touch /usr/local/ispconfig/interface/acme/client-test
    # Client & Host: Run this command on each server, you should see the host-test & client-test files in both.
    ls -l /usr/local/ispconfig/interface/acme
    rm /usr/local/ispconfig/interface/acme/*-test
    Create a new website
    You will now need to create a site for your servers hostname in your primary ispconfig control panel. For simplicity I will assume that hostname -f gave an output of mail.example.com

    Server: web server with ip
    Domain: mail.example.com
    SSL: Leave unchecked.
    LE SSL: Leave unchecked.
    Save and wait for the process to complete.

    Generate the certificate
    Return to your ISPConfig installation terminal and answer yes to the lets encrypt ssl question and continue with your installation.

    Client: | Mount the share automatically on boot to allow renewals.
    echo '' >> /etc/fstab
    echo '# web server nfs share to allow acme challenge to succeed on this server' >> /etc/fstab
    echo '    /usr/local/ispconfig/interface/acme    nfs auto,nofail,noatime,nolock,intr,tcp,actimeo=1800 0 0' >> /etc/fstab
    The steps I have added around the automatic installer should add around 15 minutes to your installation time, I think I was able to perform the complete installation in under an hour, subsequent installs of this nature should be faster.

    Sources: A combination of these pages went into building the steps for this guide.
    Last edited: Jul 29, 2021
    ahrasis likes this.
  17. Chris_UK

    Chris_UK Active Member HowtoForge Supporter

    Just a quick thank you to jesse for the idea. This has solved a problem I have had getting LE certs into my home lab for quite a while.
    till likes this.

Share This Page