Wordpress admin on nginx+php5-fpm on VPS incredibly slow. Other apps/WP front end OK

Discussion in 'General' started by talkingnews, Dec 13, 2011.

  1. talkingnews

    talkingnews New Member

    I've got extremely slow Wordpress under ISPConfig 3.0.4.1, but it's the back end only. I updated to WP 3.3 this morning, but still the same.

    The strange thing is, I'm running a personal WP install on a free "tiny" instance on Amazon S3 with a similar config and low memory, but not ispc and it's Amazon's own RHEL, and it flies along. Which is what makes it even more difficult to diagnose.

    Anyway, here's a whole load of details:

    Wordpress 3.3 on Ubuntu 11.10 + Nginx 1.0.10 + php5-fpm 5.3.8 + ISPconfig 3.0.4.1 + 256Mb VPS

    I've got a 256Mb VPS running a Zen Cart store and phpbb3, both in different php-fpm pools. There's hardly anything running except the essentials and both those sites absolutely rocket along. As does the front end of the Wordpress site, when W3TC accelerated.

    BUT.... the admin side takes 6-10 seconds to do anything.
    There's nothing in the mysql slow log, or the php-fpm error log, the load doesn't spike, and the memory usage doesn't shoot up (but see below about memory).

    The first time it loads, at wp-admin/options.php it shows a very long page that looks wrong, with line after line of stuff like...

    active_plugins SERIALIZED DATA

    Here's the main items from ps_mem.py

    Code:
    732.0 KiB +  87.5 KiB = 819.5 KiB       bash
      2.1 MiB + 369.0 KiB =   2.4 MiB       fail2ban-server
      1.8 MiB +   2.0 MiB =   3.9 MiB       nginx (5)
      5.1 MiB +  12.8 MiB =  17.9 MiB       php5-fpm (29)
     87.8 MiB + 149.0 KiB =  88.0 MiB       mysqld
    ---------------------------------
                            116.2 MiB
    =================================
    Here's the load average pretty much all the time:
    load average: 0.48, 0.53, 0.51

    And here's the output from free -m

    Code:
                 total       used       free     shared    buffers     cached
    Mem:           241        202         38          0          3         49
    -/+ buffers/cache:        149         92
    Swap:          511         29        482
    Here's the nginx.conf, including the cloudflare real_ip settings (tried without cloudflare too), and also the rewrite required to make permalinks work under nginx:

    Code:
    server {
            listen 31.172.x.x:80;
    		
            
            server_name mysite.co.uk www.mysite.co.uk www.my-site.co.uk my-site.co.uk;
    
            root   /var/www/mysite.co.uk/web;
    				
            index index.html index.htm index.php index.cgi index.pl index.xhtml;
    		
            error_page 400 /error/400.html;
            error_page 401 /error/401.html;
            error_page 403 /error/403.html;
            error_page 404 /error/404.html;
            error_page 405 /error/405.html;
            error_page 500 /error/500.html;
            error_page 502 /error/502.html;
            error_page 503 /error/503.html;
    		
            error_log /var/log/ispconfig/httpd/mysite.co.uk/error.log;
            access_log /var/log/ispconfig/httpd/mysite.co.uk/access.log combined;
    
            ## Disable .htaccess and other hidden files
            location ~ /\. {
                deny all;
                access_log off;
                log_not_found off;
            }
    		
            location = /favicon.ico {
                log_not_found off;
                access_log off;
            }
    
            location = /robots.txt {
                allow all;
                log_not_found off;
                access_log off;
            }
    		
            location /stats {
                index index.html index.php;
                auth_basic "Members Only";
                auth_basic_user_file /var/www/clients/client3/web9/.htpasswd_stats;
            }
    
            location ~ \.php$ {
                try_files $uri =404;
                include /etc/nginx/fastcgi_params;
                fastcgi_pass unix:/var/lib/php5-fpm/web9.sock;
                fastcgi_index index.php;
                fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
                fastcgi_param PATH_INFO $fastcgi_script_name;
                fastcgi_intercept_errors on;
            }
    	
             set_real_ip_from   204.93.240.0/24;
              set_real_ip_from   204.93.177.0/24;
              set_real_ip_from   199.27.128.0/21;
              set_real_ip_from   173.245.48.0/20;
              set_real_ip_from   103.22.200.0/22;
              set_real_ip_from   141.101.64.0/18;
              set_real_ip_from   108.162.192.0/18;
              real_ip_header     CF-Connecting-IP;
            client_max_body_size 28M;
            client_body_buffer_size 128k;
            
            if (!-e $request_filename) {
                    rewrite  ^(.*)$  /index.php?q=$1  last;
                    break;
                }
            #include /var/www/mysite.co.uk/web/nginx.conf;     	
    }

    Here's the php5-fpm pool conf for that site:

    Code:
    [web9]
    
    listen = /var/lib/php5-fpm/web9.sock
    listen.owner = web9
    listen.group = client3
    listen.mode = 0660
    
    user = web9
    group = client3
    
    pm = dynamic
    pm.max_children = 4
    pm.start_servers = 1
    pm.min_spare_servers = 1
    pm.max_spare_servers = 2
    
    chdir = /
    
    php_admin_value[open_basedir] = /var/www/clients/client3/web9/web:/var/www/clients/client3/web9/tmp:/var/www/mysite.co.uk/web:/srv/www/mysite.co.uk/web:/usr/share/php5:/tmp:/usr/share/phpmyadmin:/etc/phpmyadmin:/var/lib/phpmyadmin
    php_admin_value[session.save_path] = /var/www/clients/client3/web9/tmp
    php_admin_value[upload_tmp_dir] = /var/www/clients/client3/web9/tmp
    
    php_admin_value[date.timezone] = "UTC"
    php_admin_value[post_max_size] = 28M
    php_admin_value[session.gc_maxlifetime] = 604800
    php_admin_value[upload_max_filesize] = 28M
    php_admin_flag[display_errors] = off
    php_admin_flag[display_startup_errors] = off
    php_admin_flag[log_errors] = off
    php_admin_flag[ignore_repeated_errors] = off
    php_admin_flag[ignore_repeated_source] = off
    php_admin_value[memory_limit] = 32M
    That last line about memory limit is new, but made no difference either.

    I had to make changes to /etc/php5/conf.d/suhosin.ini as advised by phpmyadmin, and also upped the memory limit for WP as I was getting
    "ALERT - script tried to increase memory_limit to 268435456 bytes which is above the allowed value".

    Code:
    ; configuration for php suhosin module
    extension=suhosin.so
    suhosin.executor.include.whitelist="phar"
    suhosin.request.max_vars=2048
    suhosin.post.max_vars=2048
    suhosin.request.max_array_index_length=256
    suhosin.post.max_array_index_length=256
    suhosin.request.max_totalname_length=8192
    suhosin.post.max_totalname_length=8192
    suhosin.get.max_value_length=1024
    suhosin.memory_limit=128M
    I reduced the memory limit in wp-config.php as shown below.

    Code:
    define('WP_MEMORY_LIMIT', '32M');
    define('WP_MAX_MEMORY_LIMIT', '32M');
    Although, I've changed these limits from 256 to 128 to 64 to 32 and it makes NO difference to the front or back end speeds.

    I changed the theme to default, turned off all the plugins, changed all the mysql tables to to innodb and followed the recommendations of mysqltuner (even though, as I said, there's nothing about slow mysql in the logs I can see).

    I've tried changing php-fpm from socket to port and back, and so on and so forth. Not really sure what else to do now - can anyone spot anything here or advise?

    And yes, I might be running a bit tight on memory, but then why does Zen Cart and phpbb running with a big DB load pages sub 200ms?
     
  2. till

    till Super Moderator Staff Member ISPConfig Developer

    I guess thats you have a problem with I/O Bandwidth or in other words, the harddisk speed is too slow. I've seen this problem several times with vps servers when too many vps are on the same host system. This can also affect mysql performance. I guess there is not much that you can do about this except of talking with your hosting provider.

    What you can try to optimize mysql;

    http://www.faqforge.com/linux/optimize-mysql-performance-with-mysqltuner/

    If your sites dont use innodb, then you should consider to switch innodb off in my.cnf file, so that mysql uses less memory.


    Most likely they cache the results better.
     
  3. talkingnews

    talkingnews New Member

    As per my post above:

    Possibly, but here's the output from Zen Cart's diagnostics:

    Code:
    Database Data Size: 25,936 kB	Database Index Size: 6,496 kB
    Parse Time: 0.119 - Number of Queries: 54 - Query Time: 0.008176764892578
    So why would an old, non-optimized app manage to query a 26mb database and parse a complex page in 0.119s, while Wordpress admin takes 6-9 seconds to load a 2mb db?
     
  4. till

    till Super Moderator Staff Member ISPConfig Developer

    Changing tables to innodb causes mysql to use more ram so that the performance can decrease on low memory systems.

    You compare apples with pies here. Not every mysql query is the same and the raw database size is only one of several parameters that affects mysql performance. If a query is slow or fast depends on a lot of things like joins, if it has good indexes set and if the temp table has to be created on the harddisk etc.

    One indicator for low harddisk performance is e.g. when you run:

    rkhunter -c

    and the results are slow, rkhunter searches trough a lot of files, so if its slow, then your harddisk performance is low.

    One other thing that you might want to check is if the nameservers in /etc/resolv.conf are rechable and responding fast.
     
  5. talkingnews

    talkingnews New Member

    I found the answer! Scroll down, but first, some answers...

    With Innodb being the standard for the current mysql, I thought I'd do some digging around here. Looking at speed/memory comparisons from 2008/2009, that might have been the case, but now, most tests put innodb as around 25% faster. So while it might use a little more memory, that memory is used for less time.

    .

    Well, I think we can deal with the DB side of things quite quickly - here's the output from wordpress in debug mode:

    [0] => SELECT wp_posts.* FROM wp_posts WHERE 1=1 AND wp_posts.ID = 4 AND wp_posts.post_type = 'page' ORDER BY wp_posts.post_date DESC
    [1] => 0.032161951065063
    [2] => require, wp, WP->main, WP->query_posts, WP_Query->query, WP_Query->get_posts, W3_Db->query

    [0] => SELECT p.* FROM wp_posts AS p WHERE p.post_date < '2011-09-14 09:45:51' AND p.post_type = 'page' AND p.post_status = 'publish' ORDER BY p.post_date DESC LIMIT 1
    [1] => 0.0060410499572754
    [2] => require, require_once, include, get_header, locate_template, load_template, require_once, wp_head, do_action, call_user_func_array, adjacent_posts_rel_link_wp_head, adjacent_posts_rel_link, get_adjacent_post_rel_link, get_adjacent_post, W3_Db->query

    [0] => SELECT p.* FROM wp_posts AS p WHERE p.post_date > '2011-09-14 09:45:51' AND p.post_type = 'page' AND p.post_status = 'publish' ORDER BY p.post_date ASC LIMIT 1
    [1] => 0.00027585029602051
    [2] => require, require_once, include, get_header, locate_template, load_template, require_once, wp_head, do_action, call_user_func_array, adjacent_posts_rel_link_wp_head, adjacent_posts_rel_link, get_adjacent_post_rel_link, get_adjacent_post, W3_Db->query

    [0] => SELECT comment_approved, COUNT( * ) AS num_comments FROM wp_comments GROUP BY comment_approved
    [1] => 0.00011110305786133
    [2] => require, require_once, include, get_footer, locate_template, load_template, require_once, wp_footer, do_action, call_user_func_array, wp_admin_bar_render, do_action_ref_array, call_user_func_array, wp_admin_bar_comments_menu, wp_count_comments, W3_Db->query

    So that doesn't even total 0.1 seconds for the db query part of things.

    DNS is totally turned off, and handled by either Cloudflare, Amazon Route53 or ZoneEdit. The server doesn't run sendmail, smpt, pop3, dns... it just runs mysql, nginx, php5-fpm and ftp (oh, and a couple of basics like rkhunter!) but I only need websites, not other things. The front end runs fine, it was only the back end of WP that was slow. How would DNS affect this?

    So, in fact, it was none of those things....

    Two days of my life and a helluva lot of learning about things like xdebug, I can safely say you CANNOT now run WordPress on a VPS with less than 512Mb RAM. Which is a shame, because the 256Mb VPS had always been perfectly good before.

    Previously, it had been OK. You remember in my original post, I noted the line about "ALERT - script tried to increase memory_limit to 268435456 bytes which is above the allowed value"?

    Well, I also noted that I didn't see any memory spiking in the command line utility "top". But clearly, that doesn't refresh fast enough. With a bit of help from the VPS host who has graphical tools, we were seeing bursts of 228Mb usage when the ADMIN side of WP 3.3 was trying to load. But the front end? Well, as I said before, that zipped along and barely even registered on their graph.

    I also mentioned that I tried limiting memory use in both php.ini and the wordpress config. Well, I stepped down through 256 to 128 to 64 to 32, although from 128Mb, it would only serve blank pages.

    EDIT: Turns out this is confirmed by another blogger:
    http://www.dev4press.com/2011/blog/benchmark/wordpress-benchmark-3-0-vs-3-1-vs-3-2-vs-3-3/
    I've temporarily moved to a 512Mb RAM VPS while I decide what to do, and, of course, WP is fine again. But it's not an affordable long-term solution. So will either have to try and revert, or look at another CMS. Which is a shame, after 5 years of happy WordPressing.

    Final note to pre-empt the inevitable "but you can get 1Gb RAM VPS very cheap these days" comment, yes, you can. And I've been badly burnt that way. Amazon EC2 looks incredibly tempting - but you try not being able to stop, recover, relaunch or save a backup or even a snapshot from a failed instance for the 6 days it takes support to reply. And as for HostEurope/Webfusion, hell will freeze over and I'll start listening to Justin Bieber before I even wish them on my worst enemy.

    So I'm happy that I can just pick up the phone and immediately speak to someone, in London, in English.
     
    Last edited: Dec 15, 2011
  6. talkingnews

    talkingnews New Member

    Finally cracked it!

    Finally, I've solved it!

    The quick version:
    Running php-fpm on ubuntu:

    Code:
    sudo nano /etc/php5/conf.d/apc.ini
    You should see:

    Code:
    extension=apc.so
    paste in the below lines:
    Code:
    apc.shm_size="196"
    apc.ttl=0
    apc.max_file_size="10M"
    save and close.

    If you've got limited memory, you could try:

    Code:
    apc.shm_size="96"
    apc.ttl=0
    apc.max_file_size="2M"
    Do
    Code:
    service php5-fpm restart.
    Now go to your Wordpress admin and watch that sucker fly with near instant response.

    The long version
    I was so damn sick of this slow admin I just decided to keep going, trying stuff.

    Finally, I came across this post:
    http://stackoverflow.com/questions/...-require-once-unable-to-allocate-memory-for-p
    then this
    http://groups.drupal.org/node/75583
    which led me to this post:
    http://www.litespeedtech.com/support/forum/showthread.php?t=4366
    and finally this:
    http://2bits.com/articles/high-php-...l-and-tuning-apc-includeonce-performance.html

    The key to this was "download http://pecl.php.net/get/APC extract and run the apc.php, there you have a nice diagram how your cache usage look like"

    It showed APC was completely using its tiny default 32Mb, and was 100% fragmented. I've been running it for an hour, and see
    Hits: 70874 (98.6%) Misses: 991 (1.4%) Used: 124.4 MBytes (63.5%) Fragmentation: 0.00%

    Before this tweak it was something like 70% misses and 100% fragmentation. No wonder Wordpress was running slow!
     
  7. falko

    falko Super Moderator Howtoforge Staff

    I've switched to Xcache which seems to be faster for me (haven't done any benchmarking, but that's how it feels).
     

Share This Page