How to prevent search engine from indexing of Website auto alias domains

Hi,

I use Website auto alias in ISPConfig and I've found that search engines are indexing domain aliases. This is not desired status. So I decided to use X-Robots-Tag HTTP header (more info on developers.google.com for example). When you have in Website auto alias for example:

[client_id].test.domain.ltd

so you can put this directive to apache directive settings:

Code:

# Prevent search engine indexing on temporary domain sites:
<IfModule setenvif_module>
  <IfModule headers_module>
    SetEnvIf Host test\.domain\.ltd$ temporary_domain 
    Header set X-Robots-Tag "noindex, nofollow" env=temporary_domain
  </IfModule>
</IfModule>

Apache mod headers and mod setenvif has to be activated for properly work.

It would be nice if this functioin was included in ISPConfig directly as optional

SupuS

 

I've added this to System >> Directive Snippets

Is that the right place?

 

Yes. You have to create directive snippet for Apache and than choose this directive for each web on which you want prevent indexing by crawlers.

 

Thanks SupuS

Thats the part I was missing. Enabling it in the site config.

Looks like it's working but on checking the headers, I don't get the X-Robots-Tag.

I added a new line to the directive like this to test it:

Code:

Header set X-Robots-Tag "noindex, nofollow" env=temporary_domain
Header set MyHeader "Hello. It took %D microseconds for Apache to serve this request."

The headers show:

Code:

MyHeader:	 Hello. It took D=4463 microseconds for Apache to serve this request.

So the directive is loaded. Should I be able to see the X-Robots-Tag in the headers?

 

Please check your setting in Server configuration => Website auto alias. You have to specify on which URL this directive will be launched. If you use for example:

[website_domain].test.testingdomain.tld

you have to set:

SetEnvIf Host test\.testingdomain\.tld$ temporary_domain

 

Perfect.

100% working.

 

I added this snippet and my sites and admin panel urls won't respond anymore. Any ideas how to recover? I didin't find a file where this is located in debian.

 

Directive snippets for Apache are saved in vhost files in:
/etc/apache2/sites-available/yourdomain.tld.vhost
Try to remove your snippet from vhost file of domain you edited and restart apache manualy.

 

I'm quite sure that snippet was not in my vhost file, but I found it in *.vhost.error file. I removed my corrupted vhost file from both sites-available and sites-enabled folders and tried to restart apache, not working:

Code:

root@dev-anttimakela:/etc/apache2/sites-available# systemctl status apache2.service
● apache2.service - LSB: Apache2 web server
   Loaded: loaded (/etc/init.d/apache2)
  Drop-In: /lib/systemd/system/apache2.service.d
           └─forking.conf
   Active: active (running) (Result: exit-code) since Wed 2017-09-27 00:07:36 CEST; 9min ago
  Process: 11025 ExecStop=/etc/init.d/apache2 stop (code=exited, status=1/FAILURE)
  Process: 11329 ExecReload=/etc/init.d/apache2 reload (code=exited, status=1/FAILURE)
  Process: 11038 ExecStart=/etc/init.d/apache2 start (code=exited, status=0/SUCCESS)
   CGroup: /system.slice/apache2.service
           └─31440 /usr/sbin/apache2 -k start

Sep 27 00:09:05 dev-anttimakela systemd[1]: Reloading LSB: Apache2 web server.
Sep 27 00:09:05 dev-anttimakela apache2[11149]: Reloading web server: apache2 failed!
Sep 27 00:09:05 dev-anttimakela apache2[11149]: Apache2 is not running ... (warning).
Sep 27 00:09:05 dev-anttimakela systemd[1]: apache2.service: control process exited, code=exited status=1
Sep 27 00:09:05 dev-anttimakela systemd[1]: Reload failed for LSB: Apache2 web server.
Sep 27 00:14:00 dev-anttimakela systemd[1]: Reloading LSB: Apache2 web server.
Sep 27 00:14:00 dev-anttimakela apache2[11329]: Reloading web server: apache2 failed!
Sep 27 00:14:00 dev-anttimakela apache2[11329]: Apache2 is not running ... (warning).
Sep 27 00:14:00 dev-anttimakela systemd[1]: apache2.service: control process exited, code=exited status=1
Sep 27 00:14:00 dev-anttimakela systemd[1]: Reload failed for LSB: Apache2 web server.
root@dev-anttimakela:/etc/apache2/sites-available# journalctl -xn
-- Logs begin at Tue 2017-09-19 21:05:02 CEST, end at Wed 2017-09-27 00:17:01 CEST. --
Sep 27 00:16:49 dev-anttimakela postfix/smtpd[11315]: disconnect from unknown[14.177.236.10]
Sep 27 00:17:01 dev-anttimakela CRON[11472]: pam_unix(cron:session): session opened for user root by (uid=0)
Sep 27 00:17:01 dev-anttimakela CRON[11470]: pam_unix(cron:session): session opened for user root by (uid=0)
Sep 27 00:17:01 dev-anttimakela CRON[11474]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Sep 27 00:17:01 dev-anttimakela CRON[11471]: pam_unix(cron:session): session opened for user root by (uid=0)
Sep 27 00:17:01 dev-anttimakela CRON[11473]: (root) CMD (/usr/local/ispconfig/server/server.sh 2>&1 | while read line; do echo `/bin/date` "$line" >> /var/log/i
Sep 27 00:17:01 dev-anttimakela CRON[11470]: pam_unix(cron:session): session closed for user root
Sep 27 00:17:01 dev-anttimakela CRON[11475]: (root) CMD (/usr/local/ispconfig/server/cron.sh 2>&1 | while read line; do echo `/bin/date` "$line" >> /var/log/isp
Sep 27 00:17:01 dev-anttimakela CRON[11471]: pam_unix(cron:session): session closed for user root
Sep 27 00:17:01 dev-anttimakela CRON[11472]: pam_unix(cron:session): session closed for user root

 

What's the output of: apachectl configtest

 

Rebooted my server, apache is working again and got back in ispconfig admin area. Solved, thanks.