Google Bot dossing my server

Discussion in 'Server Operation' started by maxxer, Feb 4, 2016.

  1. maxxer

    maxxer Member

    hi.
    I've a server with ~200 wordpress websites hosted. When Google Favicon bot pays us a visit it searches all the sites indexed on our IP, that means it's requesting ~200 favicons. Most of the sites doesn't have one, so the request is passed to wordpress which has to render a 404 page. This means in a matter of seconds dozens of php processes and mysql connections, which causes a server overload.
    In the past I mitigated this by limiting the number of mysql connection and it partially works, because the server is not crashing anymore, but when this happens it still gets overloaded and unresponsive for some time.
    Given that I could either
    1. create an empty favicon.ico file for every site
    2. add a custom htaccess rule for favicon.ico
    is there a better way of controlling this issue, which doesn't involve touching websites?

    Thanks
     
  2. ztk.me

    ztk.me ISPConfig Developer ISPConfig Developer

    well you forgot to add an entry to robots.txt to disallow indexing favicon for given sites.
    But yes, one of the three solutions need to be implemented, not much you could do.

    Create empty favicon
    Disallow bot to index favicon
    Check for request to non existing file && filename equals favicon.ico => returning "blank" 404 which could be implemented on vhost level or htaccess or even in the .php file rendering the 404 page.
     
  3. maxxer

    maxxer Member

    thanks for the feedback
     

Share This Page