I need help to debug a slow website

Discussion in 'Server Operation' started by vtapas, Jun 3, 2010.

  1. vtapas

    vtapas New Member

    Hi,I am running a website behind a reverse proxy in following environment.

    Code:
     Server A --------------------Server B (site running eduCommons on Plone)
    (running apache reverse        (back end server)
     proxy)
    Server A and B are on same physical machine.
    B is a guest Operating System on A.

    Operating System on Server B is CentOS 5.5
    Operating System on Server A is Ubuntu 10.04
    eduCommons is a content management system.
    You check about it here.


    recorded the website loading time in the Chrome developer tools time line section and
    based on what I observed

    following page elements take more than 100 ms on the website
    jquery checks
    113 ms http://192.168.1.5:8080/eduCommons/p...achekey0517.js

    244ms http://192.168.1.5:8080/eduCommons/p...achekey1443.js
    247ms http://192.168.1.5:8080/eduCommons/p...chekey5402.css

    like this there are some more css and page elements which have taken
    100ms or 200ms in their GET request from server.

    on this link
    http://www.chicagostyleseo.com/2010/03/caffeine’s-need-for-speedhow-to-know-if-your-site-is-slow/
    they mentioned what Google has defined as a slow website following
    words are mentioned.

    "Google has made it clear that any process that takes more than 100
    milliseconds (1/10 of a second) is too long"

    So that means the website is slow.

    In fact I have asked a few people to access the website on internet and they too reported the same.

    In a step by step way then to debug the problem I did following

    1) The site http://site5.abc.com was loading slowly
    (Some page elements showed 411ms in Google Developer tools snapshot here)
    2) From my laptop checked http://192.168.1.5:8080/eduCommons

    ( got a few page elements 211 ms and one was 770 ms a snapshot for the 700 ms attached here)

    3) Then to check if the site is loading slow on CentOS server itself I did


    ssh -L 8080:localhost:8080 [email protected]

    4) Then from my laptop when above ssh connection on step 3 was alive

    http://localhost:8080/eduCommons
    and saw a page element loading at 411 ms.


    If I am not wrong then by step 4 it is confirming that site is slow on server itself.


    I enabled %D in /etc/httpd/conf/httpd.conf
    Code:
    LogFormat "%a %D %>s %B" resp
    CustomLog /var/log/httpd/site_in_question-resp_log resp
    
    but after hitting the website from different ISPs I saw there was no log in
    Code:
    /var/log/httpd/site_in_question-resp_log resp
    as expected.
    I thought that it might be permission so it I changed
    Code:
    chown apache:apache /var/log/httpd/site_in_question-resp_log
    I did in vhost configuration of file on CentOS Guest OS on Ubuntu Server 10.04

    On the other hand I tried to do the same on reverse proxy server
    and then apache failed to restart.
    Upon checking I found
    a2enmod mod_log_config
    Code:
    
    ERROR: Module mod_log_config does not exist!
    
    Searched but any thing fruitful did not came.
    As per your apache documentation page I searched for this file
    Code:
    find / -name mod_log_config.c
    but it was not there.



    I would like to clear that the server is not running any thing other than eduCommons .
    The server on which I am running it is physically mine.
    8 GB RAM ,500 G hard disk and more than 256 kbps bandwidth.

    What more should I do to check it and how can I over come this problem?
     
  2. Rapid2214

    Rapid2214 New Member

    By this are you impling that external sites are slow?
    The most probable reason would be the proxy, try take that out the equation and see what you get.

    Good luck :D

    Marty
     
  3. vtapas

    vtapas New Member

    I could not understand your point completely.
    The people on internet will be able to access my website when in browser they type
    http://site5.abc.com

    above is just a dummy name.

    The site is hosted internally on a webserver.
    Any how I will give some more information after posting here I tried some thing

    After reading this page

    I added in /etc/httpd/conf/httpd.conf on Server B (CentOS Guest OS)
    Code:
    <IfDefine RequestTime>
         Header set X-Request-Received: %t
         Header set X-Request-Processing-Time: %D
    </IfDefine>
    
    I am able to see some response log. in /var/log/httpd/site-in-question-resp
    Now tell me what should I post

    Code:
    
    192.168.1.5 3062 403 5043
    192.168.1.5 2826 403 5043
    
    What does this tell me ?
    What more should I do to check it and how can I over come this problem?
     
  4. Rapid2214

    Rapid2214 New Member

    So the web server is slow on a local connection? Then you have a problem.
    Try 'top' or 'free' and see if the webservers have limited resorces also try 'df -h' too see if its a disk space problem.

    The 403 part of that message would indicate a permission error, in theroy you should get 200 :S


    Good luck :D

    Marty
     
  5. vtapas

    vtapas New Member

    Well I Can not say exactly what you mean by slow on a local connection if some one on my Lan is browsing the website he is getting instant response if another user outside my lan tries to access the website on internet then it is giving a problem.The server on which website is running has 8 GB Ram and 500 Gb hard disk which is 88% free.There is no heavy load since the site is under construction.
     
  6. Rapid2214

    Rapid2214 New Member

    Ok, thanks for clearing that up.
    Whats your upload speed? http://speedtest.net

    Normally the upload speed will be the problem if a wan connection is appearing slow.

    Pls post the 'Forum Link' here :D
     
  7. vtapas

    vtapas New Member

    Download speed 6.02 Mbps
    Upload speed 18.78 Mbps
     
  8. Rapid2214

    Rapid2214 New Member

    Oh man.
    Umm you post you apache config file '/etc/apache2/apache2.conf'
     
  9. vtapas

    vtapas New Member

    Ok on CentOS which in above diagram I gave as Sever B
    I posted the file here
    http://pastebin.com/8uwMxbPS
    and for Server A it is in /etc/apache2/sites-enabled

    site5.abc.com
    Code:
    ProxyPass  /  http://192.168.1.5:8080/eduCommons
    ProxyPassReverse / http://192.168.1.5:8080/eduCommons
    
     
  10. Rapid2214

    Rapid2214 New Member

    Code:
    #
    #Listen 12.34.56.78:80
    Listen 80
    #Listen 8080
    I see that you are requesting a page on port 8080 but the webserver isn't listening on that port, try uncomment it and restart the web server, only a suggestion.

    Good luck :D

    Marty
     
  11. vtapas

    vtapas New Member

    Exactly even I thought what you just said.But when I restarted httpd I got error.
    Code:
    
     service httpd restart
    Stopping httpd:                                            [  OK  ]
    Starting httpd: (98)Address already in use: make_sock: could not bind to address [::]:8080
    (98)Address already in use: make_sock: could not bind to address 0.0.0.0:8080
    no listening sockets available, shutting down
    Unable to open logs
                                                               [FAILED]
    
    This error came when I uncomment that Listen 8080 line.

    Let me know if you have any other suggestions.
     
  12. Rapid2214

    Rapid2214 New Member

    Awsome error, it means something else is listening on that port.
    to find out what it is run:

    Code:
    netstat -tap
    Your should get the service and pid on the right

    Thanks

    Marty
     
  13. vtapas

    vtapas New Member

    webcache in following output is referring to port 8080
    Code:
    
    # netstat -tap 
    Active Internet connections (servers and established)
    Proto Recv-Q Send-Q Local Address               Foreign Address             State       PID/Program name   
    tcp        0      0 site5.siteinquestion.:xprint-server *:*                         LISTEN      5224/python2.4      
    tcp        0      0 *:hello-port                *:*                         LISTEN      1742/rpc.statd      
    tcp        0      0 *:sunrpc                    *:*                         LISTEN      1706/portmap        
    tcp        0      0 *:webcache                  *:*                         LISTEN      5229/python2.4      
    tcp        0      0 site5.abc.com:smtp        *:*                         LISTEN      2036/sendmail: acce 
    tcp        0      0 site5.siteinquestion:x11-ssh-offset *:*                         LISTEN      18012/0             
    tcp        0      0 site5.siteinquestion.:xprint-server site5.abc.com:56072       ESTABLISHED 5224/python2.4      
    tcp        0      0 192.168.1.5:webcache     192.168.1.19:60533        ESTABLISHED 5229/python2.4      
    tcp        0      0 site5.abc.com:56072       site5.siteinquestion.:xprint-server ESTABLISHED 5229/python2.4      
    tcp        0      0 192.168.1.5:webcache     serverA.siteinquestion.com:55012 ESTABLISHED 5229/python2.4      
    tcp        0      0 192.168.1.5:webcache     serverA.siteinquestion.com:44424 ESTABLISHED 5229/python2.4      
    tcp        0      0 192.168.1.5:38380        someserver.siteinquestion.com:domain TIME_WAIT   -                   
    tcp        0      0 192.168.1.5:38381        someserver.siteinquestion.com:domain TIME_WAIT   -                   
    tcp        0      0 192.168.1.5:38378        someserver.siteinquestion.com:domain TIME_WAIT   -                   
    tcp        0      0 192.168.1.5:38379        someserver.siteinquestion.com:domain TIME_WAIT   -                   
    tcp        0      0 192.168.1.5:webcache     blackberry-e63a.somdomain.:35588 ESTABLISHED 5229/python2.4      
    tcp        0      0 *:http                      *:*                         LISTEN      18140/httpd         
    tcp        0      0 *:ssh                       *:*                         LISTEN      1984/sshd           
    tcp        0      0 localhost6.l:x11-ssh-offset *:*                         LISTEN      18012/0             
    tcp        0      0 *:https                     *:*                         LISTEN      18140/httpd         
    tcp        0      0 ::ffff:192.168.1.5:ssh   jishnub08:53195             ESTABLISHED 18012/0             
    
    Above 192.168.1.19 is my laptop
    192.168.1.5 is Server B
    192.168.1.1 is Server A
    by default eduCommons uses a webserver known as Plone it is running on Port 8080.
    Will it make sense If I try to make apache listen to port 8080.
     
    Last edited: Jun 3, 2010
  14. Rapid2214

    Rapid2214 New Member

    Proberly not.
    On the CentOS 5.5 Sever run
    Code:
    wget localhost:8080
    If it doesnt download instantly its the sever, if it works quickly, its the bridge between server A & B.

    Pls post the out put of the code :D

    Good luck

    Marty
     
  15. vtapas

    vtapas New Member

    Here is the output as you said
    Code:
    wget localhost:8080
    --2010-06-04 18:45:53--  http://localhost:8080/
    Resolving localhost... 127.0.0.1
    Connecting to localhost|127.0.0.1|:8080... connected.
    HTTP request sent, awaiting response... 200 OK
    Length: 2714 (2.7K) [text/html]
    Saving to: `index.html'
    
    100%[==============================================================================================================================>] 2,714       --.-K/s   in 0s      
    
    2010-06-04 18:45:53 (185 MB/s) - `index.html' saved [2714/2714]
    
    Apart from what you said here is what I did
    Code:
    ssh -L 8080:localhost:8080 root@servername
    
    and in browser tested
    Code:
    http://localhost:8080/VirtualHostBase/http/site5.abc.com:80/eduCommons/VirtualHostRoot
    This is what I is being served and makes my site slow.
    By default some how this is the only thing which is being served to the users
    on internet also.

    So the site is slow.
    Here is a snapshot of what I just said
    and to prove what makes it slow I again loaded the website
    in Chrome Developer Tools pressed Control+Shift+I
    and was able to record the time page elements were taking to load they showed some css or jquery took more than 100ms
    Here is another snapshot.

    What is happening is eduCommons is using a server Zope.
    Zope is running on port 8080 behind an Apache running on port 80.

    The Virtual Host Manager doesn't do anything unless it sees one of the following special path elements in a URL: VirtualHostBase sets the protocol and host, while VirtualHostRoot sets the path root.
    In the Proxy Pass directive I am passing the url to an internal IP
    where I can access the website on LAN like this
    http://IP:8080/eduCommons
    some how the first snapshot I posted
    is what is being served to the users on internet and that itself is taking time.
    So the website is slow.

    While on LAN where I can locally browse the website I am directly getting the response from Zope.
    So it is not noticeable on LAN.
     
  16. falko

    falko Super Moderator Howtoforge Staff

    I suggest you measure your site with http://gtmetrix.com/ and follow the advices given there to speed up your site.
     

Share This Page