nginx reverse proxy with multiple servers

renewed, see next entry please

 

Hi Mesiah,

Thank you very much for posting this, I've been trying to do something similar with not much success so your post definitely helps.
Have a few questions though:
If I already have IP address for my backend servers, for eg:
(webserver1.domain.com --> 192.168.0.10, webserver2.domain.com --> 192.168.0.11, webserver3.domain.com --> 192.168.0.12)
What would I change in the "passthru" config above?

Also, what does the following do? Is this an internal DNS? And assuming I don't have an internal DNS, what should this be?
upstream support_backend {
server 192.168.0.1:443; # or DNS Name
}
upstream intranet_backend {
server 192.168.0.2:443; # or DNS Name
}

Thanks,

 

Hello Adnan

I have now a far better structure for your reverse Proxy. Please follow the tutorial shown below as I'm starting from the beginnig:

I was in a need to place several webservers behind a reverse proxy. But without encrypt and reencrypt and just pass unencrypted traffic from the reverse proxy. So this Tutorial may help for some people.
issue:

1. Several Webservers with various applications on each are running behind a FW and responding only on Port 443
2. The Webservers have a wildcard Certificate, they are IIS Webservers(whoooho very brave), and have public IP addresses on each webserver
3. It is requested, that all webserver should not be exposed to the Internet and moved to a DMZ
4. Since IP4 addresses are short these days, it is not possible get more IPs addresses
5. Nginx should only passthrough the requests. No Certificate break, decrypt, re-encrypt between webserver and reverse proxy or whatsoever.

Solution:

1. All websservers should be moved to a "internal" DMZ
2. A single nginx reverse proxy should handle all requests based on the webservers DNS entries and map them. This will make the public IP4 address needs obsolete. The reverse proxy could be placed on external DMZ
3. All webservers would get a private IP
4. A wild certificate would be just fine to handle all aliases for DNS forwarding.

Steps to be done:

1. A single nginx RP should be placed on the external-DMZ.

2. Install nginx: - Install nginx on a fully patched debian with apt-get install nginx. At this Point you'll get Version 1.14 for nginx. Of course you may compile it too

1. If you have installed nginx by the apt-get way, it will be configured with the following modules, which you will need later: ngx_stream_ssl_preread, ngx_stream_map, and stream. Don't worry, they are already in the package. You may check with nginx -V

4. external DNS Configuration: - all DNS request from the Internet should point the nginx.

E.g
webserver1.domain.com --> nginx
webserver2.domain.com --> nginx
webserver3.domain.com --> nginx
webserver4.domain.com --> nginx


5. nginx configurations:

• Create following folders:
  • /etc/nginx/stream/available
  • /etc/nginx/stream/enabled
• Now we need three portions of conf file placed into ./available

a) The mappings file

00_prefix.conf

Code:

 
## this where the mappings are made to each server and port. Everytime you place a new wsebserver, you need to update this file:
## mapping of servers to upstream servers ###
## In this section, we have to pair frontend and backend servers
#  everytime we want have them reverse proxied
## e.g. <relaserver.domainname>:port   <backendservervariable_port>

 map $ssl_preread_server_name:$server_port $upstream {
 #######
 webserver01.domain.com:443 realserver01_backend_443;
 #######
 webserver02.domain.com:443 realserver02_backend_443;
 #######
 webserver03.domain.com:443 realserver03_backend_443;
 #######
 webserver04.domain.com:443 realserver04_backend_443;
 webserver04.domain.com:9098 realserver04_backend_9098;  ### This is some other Port in case you need it
}

b) Now the Ports conf file.

99_suffix.conf

Code:

server {
  listen 443;
  proxy_pass $upstream;
  ssl_preread on;
}

c) At least the conf file for each upstream

01_webserver01.domain.com.conf

Code:

# 01_webserver01.domain.com.conf
upstream  realserver02_backend_443 {
  hash $remote_addr consistent;
  server 192.168.1.1:443;  # -> this is the real backend IP of the webserver
}

02_webserver02.domain.com.conf

Code:

# 02_webserver02.domain.com.conf
upstream  realserver02_backend_443 {
  hash $remote_addr consistent;
  server 192.168.1.2:443;  # -> this is the real backend IP of the webserver
}

03_webserver03.domain.com.conf

Code:

# 03_webserver02.domain.com.conf
upstream  realserver03_backend_443 {
  hash $remote_addr consistent;
  server 192.168.1.3:443;  # -> this is the real backend IP of the webserver
}

04_webserver04.domain.com.conf

Code:

upstream realserver04_backend_443 {
  hash $remote_addr consistent;
  server 192.168.1.4:443;  # -> this is the real backend IP of the webserver
}

server {
  listen 9098;  # -> this is the  extra Port if you have other services on the same webserver
  proxy_pass $upstream;
  ssl_preread on;
}

The last config is made for a server, which is running a different service extra. So the listen 9098 directive must be added as a additional entry into the 04_webserver04.domain.com.conf

now make a symlink to the ./stream/enabled/ folder:

ln -s /etc/nginx/stream/available/*.conf /etc/nginx/stream/enabled/


6. Add a Port 80 forwarding to the realservers

For each Port 80 running on the backend server you need to place a conf file here /etc/nginx/sites-available and symlink it to /etc/nginx/sites-enabled

I show you one example:

realserver01.conf

Code:

server {
  server_name webserver01.domain.com;
 listen 80;

  location ~ {
    proxy_pass_header Authorization;
    proxy_pass http://192.168.1.1;
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_http_version 1.1;
    proxy_set_header Connection "";
    proxy_buffering off;
    client_max_body_size 0;
    proxy_read_timeout 36000s;
    proxy_redirect off;
  }
}

7. Modify your nginx.conf

Code:

user www-data;

worker_processes auto;
pid /run/nginx.pid;
include /etc/nginx/modules-enabled/*.conf;

events {
        # worker_connections 768;
         worker_connections  1024;
         multi_accept on;
         use  epoll;
}



http {
        ##
        # Basic Settings
        ##

        sendfile on;
        tcp_nopush on;
        tcp_nodelay on;
        keepalive_timeout 65;
        types_hash_max_size 2048;
        server_tokens off;

        # server_names_hash_bucket_size 64;
        # server_name_in_redirect off;

        include /etc/nginx/mime.types;
        default_type application/octet-stream;

        ##
        # SSL Settings
        ##

        # ssl_protocols TLSv1 TLSv1.1 TLSv1.2; # Dropping SSLv3, ref: POODLE
        # ssl_protocols TLSv1.1 TLSv1.2; # Dropping SSLv3, ref: POODLE
        # ssl_prefer_server_ciphers on;

        ##
        # Logging Settings
        ##

        #access_log /var/log/nginx/access.log;
        #error_log /var/log/nginx/error.log;

        ##
        # Gzip Settings
        ##

         gzip on;
         gzip_vary on;
         gzip_proxied any;
         gzip_comp_level 6;
        # gzip_buffers 16 8k;
        # gzip_http_version 1.1;
         gzip_types text/plain text/css application/json application/javascript text/xml application/xml application/xml+rss text/javascript;

        ##
        # Virtual Host Configs
        ##

        include /etc/nginx/conf.d/*.conf;
        include /etc/nginx/sites-enabled/*;
}

        stream {

 include /etc/nginx/stream/enabled/*.conf;

}


#mail {
#       # See sample authentication script at:
#       # http://wiki.nginx.org/ImapAuthenticateWithApachePhpScript
#
#       # auth_http localhost/auth.php;
#       # pop3_capabilities "TOP" "USER";
#       # imap_capabilities "IMAP4rev1" "UIDPLUS";
#
#       server {
#               listen     localhost:110;
#               protocol   pop3;
#               proxy      on;
#       }
#
#       server {
#               listen     localhost:143;
#               protocol   imap;
#               proxy      on;
#       }
#}

8. Place a optimize file for nginx into /etc/nginx/conf.d

optimize.conf

Code:

        ##

        # Advanced Settings
        ##

        client_body_buffer_size 10K;
        client_header_buffer_size 1k;
        client_max_body_size 1G;
        large_client_header_buffers 2 1k;

        ##
        # Buffering
        ##

        proxy_buffering    off;
        proxy_buffer_size  128k;
        proxy_buffers 100  128k;

9. Unlink the default virtual webserver rm /etc/nginx/sites-enabled/default

10. Test nginx

nginx -t and nginx -T

11. Reload nginx with systemctl reload nginx

Now to your questions:

1. Just follow the new tutorial. You have to configure the 00_prefix.conf and add additional upstream server files 0x_webserver0x.domain.conf everytime you have a new webserver.
2. These are the IP's of your backends. BTW you can use the host file in nginx and map the IP's to the names. No need to have a DNS. Of course you can just use the IP's. But don't forget to change the IP's in the upstream, when you change your realservers IP's
3. For sure you need a external DNS. Otherwise you cannot handle point 4

Rgds

Haydar

 

Hi Haydar,

Thanks for really cleaning this up and for making it more comprehensive. I've implemented what you've suggested above and it works! Thank you!

Had one more question. I want to install a Letsencrypt certificate for SSL. In your original text, you had stated:
Solution.4 A wild certificate would be just fine to handle all aliases for DNS forwarding.
So should I just do a wildcard certificate on the Nginx (i.e. a certificate for *.domain.com)? Or should I instead install the certificate on the backend server i.e certificate on webserver1.domain.com, and so on, on the other backend servers?

Not sure what the pros and cons of either approach is, but I thought I'd ask since I'm not entirely clear of where the SSL request terminates. My preference would be to have the entire communication be encrypted from client to server.

I also noticed that in the nginx.conf, all of the SSL directives are commented out. Don't we need this for SSL?
##
# SSL Settings
##

# ssl_protocols TLSv1 TLSv1.1 TLSv1.2; # Dropping SSLv3, ref: POODLE
# ssl_protocols TLSv1.1 TLSv1.2; # Dropping SSLv3, ref: POODLE
# ssl_prefer_server_ciphers on;

Adnan

PS: Noticed a typo in Section 5 above:
5. nginx configurations:

• Create following folders:
  • /etc/nginx/stream/stream/available => /etc/nginx/stream/available
  • /etc/nginx/stream/stream/enabled => /etc/nginx/stream/enabled

 

Hello Adnan:

Yes indeed all ssl directives are commented out. Why? Because I just wanted to act the nginx as a simple reverse forwarder. The SSL decryption must be done on the webservers themself. Of course the presentation of the ssl certificate must be done on the backend servers too. Why? Because of troubleshoot and security issues. I don't want to have the nginx change the decryption and encryption. So the web server admin should not blame me, if things might go wrong. Further I don't want to have the communication between nginx and web server unencrypted.

Of course if you like the other way, like have nginx decrypt the communication and forward the requests un-encrypted or re-encrypted, sure you may do so.

As for the wildcard certificate, yes you don't need a wild card certificate, every backend could have its own. My approach was to Implement the ssl certificate on the backend. However, it depends on your needs.

This Tutorial was generated by an Project of mine. So I thought it might be useful for others too.

Yes indeed everybody is configuring the nginx the way they like. An this was my way;-) So the configs depend on your needs.

BTW, Thanks for the typo reminder, I've corrected this section now

Rgds

Haydar

 

Thank you very much. This works perfectly and was exactly what I was looking for.

 

Hi Messiah,
Thanks for the extensive tutotrial. I found another typo: 00.prefix.conf instead of 00_prefix.conf.
I followed the tutorial, but when I test the nginx with sudo nginx -t I get the following error:
nginx: [emerg] could not build map_hash, you should increase map_hash_bucket_size: 32
nginx: configuration file /etc/nginx/nginx.conf test failed
I tried to modify the nginx.conf to avoid the error, but without success. Can you point me in the right direction? Thanks in advance!

 

Hey Emile,

please check in your nginx.conf against the correct implementation of directives


a quick google Search spit out this:

https://serverfault.com/questions/8...h-you-should-increase-map-hash-bucket-size-64
https://community.centminmod.com/th...ou-should-increase-map_hash_bucket_size.5221/

https://stackoverflow.com/questions...r-names-hash-bucket-size-32-did-this-but-no-e

 

Hey Mesiah,
Thanks for your quick response!
Of course I did google as well; my nginx.conf is a copy of yours above, extended with map_hash_bucket_size 32 just above the Basic settings but still the same error. I tried the google results as well but no success

 

Hello,

ok, I would disable in the stream/enabled settings and disable the stream directive in the nginx.conf and test the conf. I would check the nginx.conf one by one

 

Hi Mesiah,
I disabled the stream settings in the nginx.conf and now the test is successful. So there must be something wrong in the stream settings. I'll doublecheck them

 

BTW I would take out all the #### comments and activate each stream conf one by one, it might be a type or so

 

Hi Mesiah,
I don't see a typo in the attached files. Can you please check them? (I renamed the suffix from .conf to .txt otherwise I cannot upload them)

 

Please try to remove the # and comments from all your files and where is 99_ files and your nginx.conf

 

I did . In the attachment is the nginx.conf. The 99_suffix.conf is exactly the same as your post

 

include /etc/nginx/stream/enabled/* is this folder existing ? Otherwise I don't see anything weird,

BTW is your map and stream modul added to nginx installtion

check it with nginx -V against ngx_stream_ssl_preread, ngx_stream_map, and stream.

 

That folder exists.
In the attachment the output for nginx -V
I'm pretty new to nginx so I don't understand the modules

 

these modules are missing:

ngx_stream_ssl_preread, ngx_stream_map

This is mandatory, which was explained in my tutorial

 

Ah now I see. Can you please explain how to install / activate the two missing modules?

 

Reading message #3, they should be installed if following the tutorial.
If they are not, I would guess using apt-get install for them does the trick. If my guess is wrong, try Internet Search Engines with

Code:

installing ngx_stream_ssl_preread ngx_stream_map