Can anybody tell me how to install nginx as a reverse proxy beside an existing ISPConfig installation
You don't have to since ISPConfig nginx web server may do the same. Just set it up and choose nginx in the parameters. You may begin to use as nginx proxy the same thereafter.