A Guide to using NGINX as a Reverse proxy for HTTP(S) and SSH

This guide will help you to set up a HTTP(S) & SSH reverse proxy using NGINX, Certbot and Certbot NGINX plugin. (Is that pronounced Engine X? Possibly).

SSH reverse proxy is needed when you want to SSH into multiple hosts that are behind a single public IP Address.

Prerequisite
This requires a minimal install of 18.04. For remote hosts you will need ssh access, local servers may use the console directly. From here on I presume that you have installed and have logged in to your new ubuntu host server.

You should know how to port forward to your nginx system.

Your login should be a non-root user. to get root privileges:

Code:

sudo -s

Ensure we get the latest certbot version (ubuntu default may be a few versions behind)

Code:

add-apt-repository ppa:certbot/certbot

Make sure our software repositories are updated and our system upgraded (upgrade may not be needed but worth checking):

Code:

apt update && apt upgrade

Install the applications we need:

Code:

 apt install -y nginx certbot python-certbot-nginx

If installing this on the ISPConfig master server which is possible you may already have certbot installed, just remove certbot from this line so that you install only nginx and python-certbot-nginx.


At this point we now have all that is needed to make this work. We want to disable the default server within nginx as its not needed and may get in our way if left active.

Code:

 unlink /etc/nginx/sites-enabled/default

And create our new host(s) the same base configuration will be used in all of the servers, you may have other services you wish to reverse proxy for, they are not covered here, but if I find need to do these, I will come back to this guide and update it.

Create your first reverse proxy server configuration file:

Code:

nano /etc/nginx/sites-available/domain.com.rproxy.conf

Copy the following code and paste it into your document, adjust the server_name and proxy_pass lines as needed:

Code:

server {
     server_name example.com www.example.com;
    # The internal IP of the VM that you want this server to point to
    location / {
         proxy_pass_header Authorization;
         proxy_pass http://192.168.1.100;
         proxy_set_header Host $host;
         proxy_set_header X-Real-IP $remote_addr;
         proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
         proxy_http_version 1.1;
         proxy_set_header Connection “”;
        proxy_buffering off;
        client_max_body_size 0;
        proxy_read_timeout 36000s;
        proxy_redirect off;
     }
    listen 80;
}

Save and close the file(s)

Now create symlinks to your configuration(s), this command will create all the links in one go is long as you have named them all with conf at the end:

Code:

ln -s /etc/nginx/sites-available/*conf /etc/nginx/sites-enabled

Test your configuration(s):

Code:

nginx -t

Assuming all went well and no errors were reported you can now restart nginx, otherwise fix the errors and retest, it may be a type on your edits:

Code:

systemctl restart nginx

Now its time to get your certificate for your server(s). I am assuming you want example.com and www.example.com
Repeat this for every domain that you added an nginx conf.

Code:

 certbot --nginx -d example.com -d www.example.com

You will answer a few questions as appropriate to your needs, if all is well you should be notified that nginx config was updated. You can check this yourself by opening the host(s).

Code:

nano /etc/nginx/sites-enabled/example.com.rproxy.conf

You may need to perform these steps after changing dns over, you can try without and see if you get a cert, if not you will need to make the dns change to point to your NGINX reverse proxy and then complete the certbot step(s) after propagation. After all of your changes you can check if your configurations worked by visiting your website in a browser.

One point I found was the proxy_pass line needed changing to https because I serve all domains entirely over ssl and the proxy was asking for http, this caused firefox to block resources because of mixed content, bad news. You should be able to set this line ahead of time if you need it that way.

I hope you find this guide useful, if you do please give it a thumbs up.

UPDATE::
So you want to ssh into your servers right? I mean it does make things easier, well with NGINX 1.9+ its possible, albeit with a caveat that you can only ssh into one server for one public ip.. Or can you.

Well as it turns out this is not the case, while SSH is a protocol without hostnames its not without IP addresses and ports, so this is what you can use make nginx reverse proxy all of your ssh servers. Well as many as the ports you can have +50k should do you well enough yes?

So there are some things you need to do and we will follow the standard method for http reverse proxying.

SSH into your reverse proxy server and make your self root: sudo -s

Code:

mkdir /etc/nginx/ssh_available
mkdir /etc/nginx/ssh_enabled

This is where things get a little tedious, but for brevity I selected a few ports in a range that match the final block of possible IPV4 server ips. You should just pick ports that you want and that are not already used by other services. Note that if you have IPV6 support then you do not need this method, you will find better solutions to this problem by searching IPV6 reverse proxy SSH.

Before you get into the next block of code, let me explain what each part is, the reverse proxy consists of two parts, the upstream and the server block. for ease of following this I used a number that matches the ip, but you could just type in the hostname.

The server block you should recognise already, we are just feeding in the appropriate upstream information.

So this is how the conf file should look (with your own edits of course).

Code:

nano /etc/nginx/ssh_available/ssh.reverse.proxy.conf

#copy paste and edit as needed
  upstream 001 { # you can use the hostname here
        server 192.168.1.1:22;
  }

  server {
    listen 22001;
    proxy_pass 001; # if you use a hostname then supply that here.
  }

  upstream 002 {
        server 192.168.1.2:22;
  }

  server {
    listen 22002;
    proxy_pass 02;
  }

  upstream 003 {
        server 192.168.1.3:22;
  }

  server {
    listen 22003;
    proxy_pass 003;
  }
# save and close

# don't forget to link it.
ln -s /etc/nginx/ssl_available/* /etc/nginx/ssl_enabled/

As i said, its tedious, a port for each ssh host and you will need a pair of upstream/server for each SSH host in your configuration file that you want to expose via your nginx reverse proxy, but when its all done your proxied hosts will be available over ssh remotely should you need it.

Now, all that we have so far is a configuration that nginx knows nothing about. We need to tell nginx to load it, but we don't want normal function http(s) as this will break the ssh and you wont be able to connect. What we need now is stream which is available for nginx versions 1.9 and above. Its one line in the stream block but it's important [see what I did there, okay i'l hang my head, you carry on] none the less.

The stream block is a base block, it must go into the nginx.conf alongside the other base blocks, http, mail etc etc.

Code:

nano /etc/nginx/nginx.conf
# ssh reverse proxy conf
stream {
    include /etc/nginx/ssh_enabled/*;
}

Save and close the file, and then , test with the following.

Code:

nginx -t

if all went well, you can restart nginx

Code:

systemctl restart nginx

The final step to bring all of this together, forward all of the ports in your router that you have defined in your server blocks to the nginx reverse proxy server.

UPDATE AGAIN:

You can proxy mysql connections, this is becoming more complex than I had imagined it to be, however using the same steps as used to proxy ssh connections, we can assign ports more ports to nginx to serve as portals to the database ports on our database servers.

So assuming we have a shared database server hosted behind our reverse proxy. We are going to want to let in our slaves. So, lets do that.
go back to the rproxy conf file and add the next code.

Code:

upstream mysql {
    server 192.168.1.21:3306;
}

server {
   listen 23456;
  proxy_pass mysql;
}

Now, when you try to connect from one of these hosts using the ispcsrvX user that ispconfig servers use, you will find you cannot connect, but when done over the command line you get an instant explanation.

ERROR 1045 (28000): Access denied for user 'ispcsrv8'@'r-proxy.lan' (using password: YES)

To remedy this, you can either, duplicate a user record in the mysql.user database and edit its host to match the one in the error, or you can run ispconfig_update.sh and reconfigure the credentials for the slave, this might be the better option if you have a lot of hosts to add to avoid any mistakes.

And now we need some security measures in place.
It's likely that we do not want to allow access to just anybody who can find the open database port. We will want to keep the circle tight and allow only our own servers to reach them. For this we add numbered rules to the ufw firewall. I will leave you to decide which other ports to close off.

Remember, these are to be run on your reverse proxy host.

Code:

ufw insert 1 allow from xxx.xxx.xxx.xxx to any port 23456 # database 1 rule for server access
ufw reload
ufw status numbered

Now lock the port down to everybody else. Change X to the next number after your last allow rule to the reverse proxy database port.

Code:

ufw insert X deny 23456
ufw reload
ufw status numbered

 

While I have your attention, Till created a feature request some time back, years actually. It's gotten no traction as yet but if you are interested in reverse proxy, you may be want to add your support here: Give the request a thumbs up, if you have a suggestion as to how it might be implemented, leave a comment also.

https://git.ispconfig.org/ispconfig/ispconfig3/issues/2829

 

Ive just found the plugin, im looking through it but as ive not looked into plugins so its going to take some time to work through this. I've done php work for a while now so it shouldn't be too difficult to run through and update it as needed.

I presume however that the plugin file itself is not the end here and more is needed to build a fully functional plugin. Time to learn the ISPC framework I think. Thanks for your pointers.

 

I hit the character limit! Maybe this should have been done as a howtoforge guide.

 

When attempting to connect to PHPMyAdmin through the reverse proxy I got a bad gateway, a little digging into the logs of the reverse proxy revealed the issue.

Code:

tail -fn100 /var/log/nginx/error.log

Revealed the issue.

Code:

upstream sent too big header while reading response header from upstream, client:

The problem was resolved after a short search and is due to buffering or lack of.

To fix this issue, add these lines into the location block of the affected host/websites NGINX configuration file.

Code:

        proxy_buffering on;
        proxy_buffer_size 128k;
        proxy_buffers 4 256k;
        proxy_busy_buffers_size 256k;

As always, nginx -t and if the configurations test well, systemctl restart nginx. You should now be able to load up phpmyadmin on the affected domain.

Its worth noting here that the values defined are very high. you can try smaller buffer values, and you should probably calculate exactly what you need in terms of buffer size and set it a few k higher, but for some reason phpmyadmin is exceeding the NGINX default of 4k.