HAProxy keeps switching between servers

Discussion in 'HOWTO-Related Questions' started by wxman, May 5, 2009.

  1. wxman

    wxman New Member

    I have HAProxy and Heartbeat running as my load balancer/fail over. I used the how-to here to set it up. I noticed that when I refresh a page, it reloads using the opposite server's file. I could tell because I temporarily shut off my rsync to test what was happening. I thought HAProxy would set a cookie (yes the browser allows them) to keep you on the server you started on as long as it was still running? I tested this in more than one browser, and it still happens.

    On a similar note; why would I keep needing to restart heartbeat on the load balancer so I can see the HAProxy stats page? frequently, if I go to the stats page, it's not there. I check, and Heartbeat is running, and if I restart it, the the stats return. I don't see anything in the logs that would answer this except every time it happens, I see that it's looking for it on the web server, not the load balancer:
    Code:
    File does not exist: /var/www/apache2-default/haproxy
     
    Last edited: May 6, 2009
  2. falko

    falko Super Moderator Howtoforge Staff

    I've never had such problems with HAProxy and heartbeat. Please double-check your setup - maybe you made a typo somewhere?
     
  3. wxman

    wxman New Member

    I've checked them many times now. I even went through and completely redid them. I've also done a lot of Googling the the two problems, and I haven't found anyone else talking about it.
     
  4. wtarreau

    wtarreau New Member

    The fact that the stats page is forwarded to the servers happens because you forgot to set "option httpclose" in haproxy's config, so a connection to your web site is maintained between the client and the server and haproxy does not see subsequent requests for the stats in it. I'm not sure why restarting heartbeat fixes the issue, that may simply be a side effect of breaking the connection to the browser.
     
  5. wxman

    wxman New Member

    I just checked, and option httpclose is set in my cfg:
    Code:
    global
            log 127.0.0.1   local0
            log 127.0.0.1   local1 notice
            #log loghost    local0 info
            maxconn 4096
            #debug
            #quiet
            user haproxy
            group haproxy
    
    defaults
            log     global
            mode    http
            option  httplog
            option  dontlognull
            retries 3
            redispatch
            maxconn 2000
            contimeout      5000
            clitimeout      50000
            srvtimeout      50000
    
    listen webfarm 192.168.31.100:80
           mode http
           stats enable
           stats auth netadmin:5bgr+bdd1WbA
           balance roundrobin
           cookie JSESSIONID prefix
           option httpclose
           option forwardfor
           option httpchk HEAD /check.txt HTTP/1.0
           server web1 192.168.31.202:80 cookie w01 check inter 2000 rise 2 fall 2
           server web2 192.168.31.212:80 cookie w02 check inter 2000 rise 2 fall 2
           option persist
           redispatch
           contimeout 5000
    
    Maybe that will help. I thought I went over it enough, and didn't see any problems.
     
  6. wxman

    wxman New Member

    I've been doing more testing and research on what's going on here.
    I use http://192.168.31.100/haproxy?stats to get to that stats page. The .100 is the shared address between the load balancers. If I use .201, which is LB1, I get the browser's 404 notice. If I use .100, it shows my apache generic 404 page. So somehow it stops seeing LB1, and goes to port 80 on my web server on the web1 node. That's where I see the apache error saying it can't find the HAProxy stats page.
    I've never used tcpdump before, so I had to do some quick research first. I reset the node again to make things work. When I used the "tcpdump -q -i eth0 tcp port 80 and src host 192.168.31.100" command, it showed me looking at the stats, and the test web page:
    Code:
    tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on eth0, link-type EN10MB (Ethernet), capture size 96 bytes
    11:23:16.106664 IP 192.168.31.100.www > 192.168.30.64.2289: tcp 0
    11:23:16.254209 IP 192.168.31.100.www > 192.168.30.64.2289: tcp 0
    11:23:16.254409 IP 192.168.31.100.www > 192.168.30.64.2289: tcp 262
    11:23:16.254501 IP 192.168.31.100.www > 192.168.30.64.2289: tcp 0
    11:23:17.460534 IP 192.168.31.100.www > 192.168.30.64.2290: tcp 0
    11:23:17.628385 IP 192.168.31.100.www > 192.168.30.64.2290: tcp 0 11:23:17.628590 IP 192.168.31.100.www > 192.168.30.64.2290: tcp 2712
    11:23:17.839448 IP 192.168.31.100.www > 192.168.30.64.2290: tcp 2712 11:23:17.839460 IP 192.168.31.100.www > 192.168.30.64.2290: tcp 524
    Once I couldn't see the stats page again, the tcpdump output stopped. I watched it on LB2 as well. It seems like it stops listing to the .100 IP address. If I use "tcpdump -q -i eth0 tcp port 80" I see LB1 checking web1 and web2, but nothing on the .100 address. If I was running a name server, I'd say that's the problem, but I'm not.
     
  7. astroman1

    astroman1 New Member

    Did you ever figure out why this was happening? I am seeing this. I have a page which loads 8 files (html, images, javascript and css), and the HTTP GET requests are split alternately between the two web servers?
     
  8. astroman1

    astroman1 New Member

    I found that mine was not working because of the line
    I had to change it to
    cookie JSESSIONID insert

    as the JSESSIONID cookie was not previously sent from my site. The haproxy server data could not be prefixed to the cookie and the server data was not sent to the client, All traffic from the client did not have the server tag and every request went to alternate servers.

    Note that the cookie name above (JSESSIONID) should be unique and not be the same as any cookie emitted by your server since haproxy will replace it with its own if cookie insert is used.
     
  9. sufehmi

    sufehmi New Member

    Thank You !

    I logged on just to say THANKS - this is spot on :)

    Reading all over HAproxy documentation at 3 am sure doesn't help me finding the information I needed.
    Thankfully, Google never felt sleepy - and correctly pointed me to this post of yours.


    Another BULLSEYE, I was using PHPSESSID and wondered why it didn't work.

    So I changed it to a unique cookie name, and voila -- session-aware load-balancer !

    Thanks again !!


    Cheers, HS
     
  10. deepakpalmca

    deepakpalmca New Member

    Not serving request in round robin fashion.

    Hey,

    I have confiured haproxy to work in a roundrobin fashion with two backend servers and enabled session persistence. But it's serving most of the requests from appserver1 only rarely it's going to appserver2 like once in a dozen.


    my configuration is.

    haproxy.cfg:

    global
    maxconn 50000
    user haproxy
    group haproxy
    daemon
    #number of processing core use 3
    nbproc 3
    pidfile /tmp/haproxy.pid
    defaults
    mode http
    clitimeout 10000
    srvtimeout 10000
    contimeout 10000
    #Disable keepalive
    option httpclose
    log global
    log 127.0.0.1 local3
    option httplog
    # option http-server-close

    listen http_proxy 127.0.0.1:80
    #Balancing Algorithm
    balance roundrobin
    #Use this to chek the status of servers on stats
    # option httpcheck HEAD /check.txt HTTP/1.0
    #This option will set x-forward-for
    option forwardfor
    stats enable
    #check this once again
    stats uri /ha_stats
    stats realm Global\ Haproxy
    stats auth analytix:analytix
    stats refresh 30s
    # stats hide-version
    # option httpclose
    capture cookie JSESSIONID len 52
    #check this
    # stick-table type string len 52 size 2g expire 5h
    cookie HTTPSERVERID insert nocache indirect
    # appsession JSESSIONID len 52 timeout 5h mode query-string
    appsession JSESSIONID len 52 timeout 5h

    #Our Backend servers entry by command line arguments

    server appServer1 204.232.206.201:80 weight 1 maxconn 15000 check inter 2000 rise 2 fall 3 cookie A

    server appServer2 204.232.206.221:80 weight 1 maxconn 15000 check inter 2000 rise 2 fall 3 cookie A




    Regards,
    Deepak
     

Share This Page