Thursday, September 3, 2015

How to use HAProxy as Load Balancer in RHEL

In general, the load balance is highly required for the cloud environment, since all the requests should go through the proper resource allocator. And the balancer should be able to scale out via adding more resource to support heavier loading in the coming future.

 Recently, I have played around the open-source HAProxy as load balancer for my Openstack Swift Proxy nodes. I found couple useful links in reference but none of them collect what's exactly I need which is setup HAProxy in REHL and combine rsyslogd properly. 

 In this post, I will list the steps how you setup/configure HAProxy , Rsyslogd and Unit Test the whole load balance feature. Here is the highlight for whole process.
  1. Setup/Configure HAProxy
  2. Start/Enable HAProxy
  3. Add HAPRoxy log Configure in Rsyslog
  4. Restart Rsyslog
  5. Unit Test for load balance feature.

Before we start, here is the assumption, you have one server as HAProxy and three servers which needs to be balanced the loading via round robin strategy. The servers list can be as blow.

  • HAProxy: 10.0.0.1
  • Server-001: 10.0.0.2
  • Server-002: 10.0.0.3
  • Server-003: 10.0.0.4
  • Swift Cluster: swift

1. Setup/Configure HAPRoxy

Install haproxy for redhat
#yum install haproxy

eg:
[root@xxx-001 ~]# yum install haproxy
Loaded plugins: product-id, subscription-manager
This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register.
ccs-secure-repo                                                                                                                            | 2.9 kB  00:00:00     
ice-1.2.2-ceph                                                                                                                             | 2.9 kB  00:00:00     
ice-1.2.2-ceph-deploy                                                                                                                      | 2.9 kB  00:00:00     
mongodb                                                                                                                                    | 2.9 kB  00:00:00     
osp5-hotfix                                                                                                                                | 2.9 kB  00:00:00     
percona-centos6                                                                                                                            | 2.9 kB  00:00:00     
rhel-7-server-ansible-rpms                                                                                                                 | 2.9 kB  00:00:00     
rhel-7-server-cisco-rpms                                                                                                                   | 2.9 kB  00:00:00     
rhel-7-server-extras-rpms                                                                                                                  | 2.9 kB  00:00:00     
rhel-7-server-hybrid-rpms                                                                                                                  | 2.9 kB  00:00:00     
rhel-7-server-openstack-5.0-rpms                                                                                                           | 2.9 kB  00:00:00     
rhel-7-server-optional-rpms                                                                                                                | 2.9 kB  00:00:00     
rhel-7-server-rabbitmq-rpms                                                                                                                | 2.9 kB  00:00:00     
rhel-7-server-rhn-tools-rpms                                                                                                               | 2.9 kB  00:00:00     
rhel-7-server-rpms                                                                                                                         | 2.9 kB  00:00:00     
rhel-7-server-supplementary-rpms                                                                                                           | 2.9 kB  00:00:00     
svl-pod-3                                                                                                                                  | 2.9 kB  00:00:00     
(1/2): ccs-secure-repo/primary_db                                                                                                          | 1.0 MB  00:00:00     
(2/2): svl-pod-3/primary_db                                                                                                                | 7.2 MB  00:00:00     
Package haproxy-1.5.4-2.el7.x86_64 already installed and latest version
Nothing to do

Edit haproxy configuration file.
#vi /etc/haproxy/haproxy.cfg

eg:
#cat /etc/haproxy/haproxy.cfg
global
#    local2.*                       /var/log/haproxy.log
log 127.0.0.1 local2
maxconn 4096
user haproxy
group haproxy
daemon

defaults
        log     global
        mode    http
        option  httplog
        option  dontlognull
        retries 3
        option  redispatch
        maxconn 2000
    timeout queue           1m
    timeout connect         10s
    timeout client          10m
    timeout server          10m


listen swift 10.0.0.1:80
        mode    http
        stats   enable
        stats   auth username:password
        balance roundrobin
        option  httpchk HEAD /healthcheck HTTP/1.0
        option  forwardfor
        option  httpclose
        server  server-001 10.0.0.2:80 weight 5 check inter 2000
        server  server-002 10.0.0.3:80 weight 5 check inter 2000
        server  server-003 10.0.0.4:80 weight 5 check inter 2000


2. Start/Enable HAProxy

[root@xxx-001 haproxy]# systemctl start haproxy.service
[root@xxx-001 haproxy]# systemctl enable haproxy.service
ln -s '/usr/lib/systemd/system/haproxy.service' '/etc/systemd/system/multi-user.target.wants/haproxy.service'
[root@xxx-001 haproxy]# systemctl status haproxy.service

PS: In REHL, start is start the daemon and enable can make sure after server reboot the daemon will be stared automatically.

3. Add HAPRoxy log Configure in rsyslog.conf

#vi /etc/rsyslog.conf 


Remove comment: find the line as below and remove the '#' before '$'.The final result should looks like as below.

 # Provides UDP syslog reception
$ModLoad imudp
$UDPServerRun 514

# Provides TCP syslog reception
$ModLoad imtcp
$InputTCPServerRun 514

Add this line
local2.*                                                /var/log/haproxy.log

PS: you can run #yum install rsyslog to install rsyslog if you don't have in RHEL, but in my RHEL 7/7.1. The rsyslog is default.


4. Restart Rsyslog 

Restart the rsyslog service
#systemctl restart rsyslog.service


Check rsyslogd status
#systemctl status rsyslog.service

5. Unit Test

Unit test for loadbalance via checking the log, 
eg: just refresh the web portal
or 
You can use the swift command to get the account status for triggering the request against the Load Balancer.
#swift -A http://HAProxy/auth/v1.0 -U swift -K swift stat
                                Account: AUTH_swift
                             Containers: 4
                                Objects: 5
                                  Bytes: 868960818
Containers in policy "standard-replica": 4
   Objects in policy "standard-replica": 5
     Bytes in policy "standard-replica": 868960818
                      Meta Temp-Url-Key: d4cfaa78-034d-42ac-a197-9f9d431f7e60
                          Accept-Ranges: bytes
                             Connection: close
                            X-Timestamp: 1440183945.29423
                             X-Trans-Id: tx8733adb891334aedb929f-0055e9340f

                           Content-Type: text/plain; charset=utf-8
#tail -f /var/log/haproxy.log

If you have any issue, you might try to check message directlry, then troubleshooting from there.
#tail -f /var/log/message

Both log's content, you can see the it most like follow round robin strategy.
PS: both network log in message and haproxy.log should be the same.

[root@xxx-001 ~]# tail -f /var/log/haproxy.log
Sep  4 04:33:22 localhost haproxy[5015]: 10.24.104.76:57582 [04/Sep/2015:04:33:22.846] swift swift/server-003 0/0/0/106/107 200 168 - - ---- 0/0/0/0/0 0/0 "GET /console/css/extra-styles.css HTTP/1.1"
Sep  4 04:33:49 localhost haproxy[5015]: 10.24.104.76:57589 [04/Sep/2015:04:33:49.186] swift swift/server-001 5/0/0/2/7 200 817 - - ---- 1/1/0/0/0 0/0 "GET /console/ HTTP/1.1"
Sep  4 04:33:49 localhost haproxy[5015]: 10.24.104.76:57591 [04/Sep/2015:04:33:49.195] swift swift/server-003 26/0/0/106/132 200 14696 - - ---- 5/5/5/1/0 0/0 "GET /console/css/lib/bootstrap-responsive.css HTTP/1.1"
Sep  4 04:33:49 localhost haproxy[5015]: 10.24.104.76:57593 [04/Sep/2015:04:33:49.195] swift swift/server-001 26/0/0/110/153 200 24192 - - ---- 4/4/4/1/0 0/0 "GET /console/css/lib/glyphicons.css HTTP/1.1"
Sep  4 04:33:49 localhost haproxy[5015]: 10.24.104.76:57592 [04/Sep/2015:04:33:49.195] swift swift/server-003 31/0/0/198/229 200 168 - - ---- 3/3/3/0/0 0/0 "GET /console/css/extra-styles.css HTTP/1.1"
Sep  4 04:33:49 localhost haproxy[5015]: 10.24.104.76:57594 [04/Sep/2015:04:33:49.195] swift swift/server-002 26/0/0/212/239 200 4556 - - ---- 2/2/2/1/0 0/0 "GET /console/css/app/app.css HTTP/1.1"
Sep  4 04:33:49 localhost haproxy[5015]: 10.24.104.76:57595 [04/Sep/2015:04:33:49.253] swift swift/server-001 0/0/0/181/181 200 14999 - - ---- 1/1/1/0/0 0/0 "GET /console/js/lib/require.js HTTP/1.1"
Sep  4 04:33:49 localhost haproxy[5015]: 10.24.104.76:57590 [04/Sep/2015:04:33:49.186] swift swift/server-002 36/0/0/211/304 200 100347 - - ---- 0/0/0/0/0 0/0 "GET /console/css/lib/bootstrap.css HTTP/1.1"
Sep  4 04:33:49 localhost haproxy[5015]: 10.24.104.76:57596 [04/Sep/2015:04:33:49.563] swift swift/server-002 0/0/0/2/2 200 1245 - - ---- 0/0/0/0/0 0/0 "GET /console/js/main.js HTTP/1.1"
Sep  4 04:33:49 localhost haproxy[5015]: 10.24.104.76:57597 [04/Sep/2015:04:33:49.616] swift swift/server-003 0/0/0/2/2 200 3574 - - ---- 0/0/0/0/0 0/0 "GET /console/js/app/app.js HTTP/1.1"
Sep  4 04:33:49 localhost haproxy[5015]: 10.24.104.76:57601 [04/Sep/2015:04:33:49.673] swift swift/server-003 6/0/0/4/10 200 2147 - - ---- 5/5/5/1/0 0/0 "GET /console/js/app/router.js HTTP/1.1"
Sep  4 04:33:49 localhost haproxy[5015]: 10.24.104.76:57600 [04/Sep/2015:04:33:49.673] swift swift/server-002 6/0/0/4/10 200 6257 - - ---- 4/4/4/1/0 0/0 "GET /console/js/app/collections/containers.js HTTP/1.1"
Sep  4 04:33:49 localhost haproxy[5015]: 10.24.104.76:57602 [04/Sep/2015:04:33:49.673] swift swift/server-001 6/0/0/4/10 200 1620 - - ---- 3/3/3/1/0 0/0 "GET /console/js/app/models/user.js HTTP/1.1"
Sep  4 04:33:49 localhost haproxy[5015]: 10.24.104.76:57603 [04/Sep/2015:04:33:49.673] swift swift/server-003 6/0/0/4/11 200 6122 - - ---- 2/2/2/0/0 0/0 "GET /console/js/app/collections/accounts.js HTTP/1.1"
Sep  4 04:33:49 localhost haproxy[5015]: 10.24.104.76:57599 [04/Sep/2015:04:33:49.673] swift swift/server-001 6/0/0/3/24 200 42791 - - ---- 1/1/1/0/0 0/0 "GET /console/js/lib/underscore.js HTTP/1.1"
Sep  4 04:33:49 localhost haproxy[5015]: 10.24.104.76:57605 [04/Sep/2015:04:33:49.745] swift swift/server-002 0/0/0/2/2 200 4296 - - ---- 4/4/4/1/0 0/0 "GET /console/js/app/collections/queue.js HTTP/1.1"
Sep  4 04:33:49 localhost haproxy[5015]: 10.24.104.76:57606 [04/Sep/2015:04:33:49.745] swift swift/server-003 0/0/0/2/2 200 1987 - - ---- 3/3/3/0/0 0/0 "GET /console/js/app/views/authentication.js HTTP/1.1"
Sep  4 04:33:49 localhost haproxy[5015]: 10.24.104.76:57607 [04/Sep/2015:04:33:49.745] swift swift/server-001 0/0/0/3/3 200 4349 - - ---- 2/2/2/1/0 0/0 "GET /console/js/app/models/account.js HTTP/1.1"
Sep  4 04:33:49 localhost haproxy[5015]: 10.24.104.76:57604 [04/Sep/2015:04:33:49.745] swift swift/server-001 0/0/0/4/4 200 4355 - - ---- 1/1/1/0/0 0/0 "GET /console/js/app/views/root.js HTTP/1.1"
Sep  4 04:33:49 localhost haproxy[5015]: 10.24.104.76:57608 [04/Sep/2015:04:33:49.785] swift swift/server-002 0/0/0/2/2 200 487 - - ---- 1/1/1/1/0 0/0 "GET /console/js/lib/swift-jquery.js HTTP/1.1"
Sep  4 04:33:49 localhost haproxy[5015]: 10.24.104.76:57598 [04/Sep/2015:04:33:49.673] swift swift/server-002 6/0/0/3/147 200 253694 - - ---- 0/0/0/0/0 0/0 "GET /console/js/lib/jquery.js HTTP/1.1"
Sep  4 04:33:49 localhost haproxy[5015]: 10.24.104.76:57610 [04/Sep/2015:04:33:49.840] swift swift/server-003 0/0/0/2/2 200 2008 - - ---- 2/2/1/0/0 0/0 "GET /console/js/app/models/container.js HTTP/1.1"
Sep  4 04:33:49 localhost haproxy[5015]: 10.24.104.76:57609 [04/Sep/2015:04:33:49.840] swift swift/server-001 0/0/0/3/3 200 1904 - - ---- 1/1/0/0/0 0/0 "GET /console/js/app/collections/pagination.js HTTP/1.1"
Sep  4 04:33:49 localhost haproxy[5015]: 10.24.104.76:57611 [04/Sep/2015:04:33:49.840] swift swift/server-002 4/0/0/2/6 200 8477 - - ---- 0/0/0/0/0 0/0 "GET /console/js/app/collections/objects.js HTTP/1.1"
Sep  4 04:33:49 localhost haproxy[5015]: 10.24.104.76:57612 [04/Sep/2015:04:33:49.861] swift swift/server-003 0/0/0/1/1 200 2847 - - ---- 1/1/1/0/0 0/0 "GET /console/js/app/views/modal.js HTTP/1.1"
Sep  4 04:33:49 localhost haproxy[5015]: 10.24.104.76:57613 [04/Sep/2015:04:33:49.861] swift swift/server-001 0/0/0/2/2 200 12602 - - ---- 0/0/0/0/0 0/0 "GET /console/js/lib/text.js HTTP/1.1"
Sep  4 04:33:49 localhost haproxy[5015]: 10.24.104.76:57615 [04/Sep/2015:04:33:49.903] swift swift/server-003 0/0/0/1/1 200 1971 - - ---- 3/3/1/0/0 0/0 "GET /console/js/app/templates/root.html HTTP/1.1"
Sep  4 04:33:49 localhost haproxy[5015]: 10.24.104.76:57614 [04/Sep/2015:04:33:49.903] swift swift/server-002 0/0/0/2/2 200 1250 - - ---- 2/2/0/0/0 0/0 "GET /console/js/app/templates/authentication.html HTTP/1.1"
Sep  4 04:33:49 localhost haproxy[5015]: 10.24.104.76:57617 [04/Sep/2015:04:33:49.903] swift swift/server-002 5/0/0/1/6 200 676 - - ---- 1/1/1/0/0 0/0 "GET /console/js/lib/uuid.js HTTP/1.1"
Sep  4 04:33:49 localhost haproxy[5015]: 10.24.104.76:57616 [04/Sep/2015:04:33:49.903] swift swift/server-001 5/0/0/2/7 200 1668 - - ---- 0/0/0/0/0 0/0 "GET /console/js/app/models/queue.js HTTP/1.1"
Sep  4 04:33:49 localhost haproxy[5015]: 10.24.104.76:57618 [04/Sep/2015:04:33:49.925] swift swift/server-003 0/0/0/1/1 200 1058 - - ---- 1/1/1/0/0 0/0 "GET /console/js/app/views/alert.js HTTP/1.1"
Sep  4 04:33:49 localhost haproxy[5015]: 10.24.104.76:57623 [04/Sep/2015:04:33:49.960] swift swift/server-001 0/0/0/1/1 200 5914 - - ---- 4/4/4/1/0 0/0 "GET /console/js/app/views/container.js HTTP/1.1"
Sep  4 04:33:49 localhost haproxy[5015]: 10.24.104.76:57621 [04/Sep/2015:04:33:49.960] swift swift/server-003 0/0/0/1/1 200 1811 - - ---- 3/3/3/0/0 0/0 "GET /console/js/app/core/swift-search.js HTTP/1.1"
Sep  4 04:33:49 localhost haproxy[5015]: 10.24.104.76:57620 [04/Sep/2015:04:33:49.960] swift swift/server-002 0/0/0/2/2 200 571 - - ---- 2/2/2/1/0 0/0 "GET /console/js/app/templates/alert.html HTTP/1.1"
Sep  4 04:33:49 localhost haproxy[5015]: 10.24.104.76:57619 [04/Sep/2015:04:33:49.925] swift swift/server-001 0/0/0/1/39 200 73141 - - ---- 1/1/1/0/0 0/0 "GET /console/js/lib/handlebars.js HTTP/1.1"
Sep  4 04:33:49 localhost haproxy[5015]: 10.24.104.76:57622 [04/Sep/2015:04:33:49.960] swift swift/server-002 0/0/0/2/17 200 22777 - - ---- 0/0/0/0/0 0/0 "GET /console/js/app/views/object.js HTTP/1.1"
Sep  4 04:33:49 localhost haproxy[5015]: 10.24.104.76:57624 [04/Sep/2015:04:33:49.984] swift swift/server-003 3/0/0/1/4 200 401 - - ---- 0/0/0/0/0 0/0 "GET /console/js/app/templates/container.html HTTP/1.1"
Sep  4 04:33:50 localhost haproxy[5015]: 10.24.104.76:57627 [04/Sep/2015:04:33:50.026] swift swift/server-001 0/0/0/1/1 200 1358 - - ---- 2/2/0/0/0 0/0 "GET /console/js/app/templates/container/list.html HTTP/1.1"
Sep  4 04:33:50 localhost haproxy[5015]: 10.24.104.76:57626 [04/Sep/2015:04:33:50.026] swift swift/server-002 5/0/0/2/7 200 1475 - - ---- 1/1/1/0/0 0/0 "GET /console/js/app/templates/object.html HTTP/1.1"
Sep  4 04:33:50 localhost haproxy[5015]: 10.24.104.76:57625 [04/Sep/2015:04:33:50.026] swift swift/server-003 5/0/0/2/7 200 2346 - - ---- 0/0/0/0/0 0/0 "GET /console/js/app/templates/object/list.html HTTP/1.1"
Sep  4 04:33:50 localhost haproxy[5015]: 10.24.104.76:57630 [04/Sep/2015:04:33:50.049] swift swift/server-002 0/0/0/1/1 200 1136 - - ---- 2/2/2/0/0 0/0 "GET /console/js/app/views/search-result.js HTTP/1.1"
Sep  4 04:33:50 localhost haproxy[5015]: 10.24.104.76:57628 [04/Sep/2015:04:33:50.049] swift swift/server-001 0/0/0/1/1 200 4613 - - ---- 1/1/1/0/0 0/0 "GET /console/js/app/views/queue.js HTTP/1.1"
Sep  4 04:33:50 localhost haproxy[5015]: 10.24.104.76:57629 [04/Sep/2015:04:33:50.049] swift swift/server-003 0/0/0/1/1 200 4876 - - ---- 0/0/0/0/0 0/0 "GET /console/js/app/views/account.js HTTP/1.1"
Sep  4 04:33:50 localhost haproxy[5015]: 10.24.104.76:57631 [04/Sep/2015:04:33:50.078] swift swift/server-001 0/0/0/1/1 200 375 - - ---- 2/2/0/0/0 0/0 "GET /console/js/app/templates/queue.html HTTP/1.1"
Sep  4 04:33:50 localhost haproxy[5015]: 10.24.104.76:57633 [04/Sep/2015:04:33:50.078] swift swift/server-002 5/0/0/2/7 200 445 - - ---- 1/1/1/0/0 0/0 "GET /console/js/app/templates/queue/item.html HTTP/1.1"
Sep  4 04:33:50 localhost haproxy[5015]: 10.24.104.76:57632 [04/Sep/2015:04:33:50.078] swift swift/server-003 5/0/0/2/7 200 264 - - ---- 0/0/0/0/0 0/0 "GET /console/js/app/templates/queue/list.html HTTP/1.1"
Sep  4 04:33:50 localhost haproxy[5015]: 10.24.104.76:57634 [04/Sep/2015:04:33:50.107] swift swift/server-001 0/0/0/1/1 200 1591 - - ---- 2/2/2/0/0 0/0 "GET /console/js/app/templates/search-result.html HTTP/1.1"
Sep  4 04:33:50 localhost haproxy[5015]: 10.24.104.76:57635 [04/Sep/2015:04:33:50.107] swift swift/server-002 0/0/0/1/1 200 1146 - - ---- 1/1/1/0/0 0/0 "GET /console/js/app/templates/account/list.html HTTP/1.1"
Sep  4 04:33:50 localhost haproxy[5015]: 10.24.104.76:57636 [04/Sep/2015:04:33:50.107] swift swift/server-003 0/0/0/2/2 200 553 - - ---- 0/0/0/0/0 0/0 "GET /console/js/app/templates/account.html HTTP/1.1"
Sep  4 04:33:50 localhost haproxy[5015]: 10.24.104.76:57637 [04/Sep/2015:04:33:50.131] swift swift/server-002 0/0/0/1/1 200 4910 - - ---- 1/1/1/0/0 0/0 "GET /console/js/lib/swift-jquery/auth.js HTTP/1.1"
Sep  4 04:33:50 localhost haproxy[5015]: 10.24.104.76:57638 [04/Sep/2015:04:33:50.131] swift swift/server-001 0/0/0/1/1 200 5924 - - ---- 0/0/0/0/0 0/0 "GET /console/js/lib/swift-jquery/core.js HTTP/1.1"
Sep  4 04:33:50 localhost haproxy[5015]: 10.24.104.76:57639 [04/Sep/2015:04:33:50.136] swift swift/server-003 0/0/0/1/1 200 5698 - - ---- 0/0/0/0/0 0/0 "GET /console/js/lib/swift-jquery/container.js HTTP/1.1"
Sep  4 04:33:50 localhost haproxy[5015]: 10.24.104.76:57641 [04/Sep/2015:04:33:50.158] swift swift/server-002 0/0/0/1/1 200 4201 - - ---- 2/2/2/0/0 0/0 "GET /console/js/app/models/object.js HTTP/1.1"
Sep  4 04:33:50 localhost haproxy[5015]: 10.24.104.76:57640 [04/Sep/2015:04:33:50.158] swift swift/server-001 0/0/0/1/1 200 9412 - - ---- 1/1/1/0/0 0/0 "GET /console/js/app/core/paginator.js HTTP/1.1"
Sep  4 04:33:50 localhost haproxy[5015]: 10.24.104.76:57645 [04/Sep/2015:04:33:50.185] swift swift/server-002 0/0/0/1/1 200 1687 - - ---- 3/3/3/0/0 0/0 "GET /console/js/app/core/number-formatter.js HTTP/1.1"
Sep  4 04:33:50 localhost haproxy[5015]: 10.24.104.76:57644 [04/Sep/2015:04:33:50.185] swift swift/server-003 0/0/0/1/1 200 2432 - - ---- 2/2/2/0/0 0/0 "GET /console/js/app/views/container/create.js HTTP/1.1"

Configure HAProxy to Load Balance Site with SSL PassThrough


Another method of load balancing SSL is to just pass through the traffic. With this approach since everything is encrypted, you won’t be able to monitor and tweak HTTP headers/traffic. Here are a couple of sample setups:

$ cat /etc/haproxy/haproxy.cfg.bypassssl
global
    log 127.0.0.1 local0
    maxconn 4000
    daemon
    uid 99
    gid 99

defaults
    log     global
    timeout server 5s
    timeout connect 5s
    timeout client 5s

frontend https_frontend
    bind *:443
    mode tcp
    default_backend varnish_cluster

backend varnish_cluster
    mode tcp
    balance roundrobin
    stick-table type ip size 200k expire 30m
    stick on src
    server s1 10.32.0.6:443
    server s2 10.32.0.8:443

 If you want load-balancing of HTTPS sessions handle cert at haproxy server, you can use this configuration, you might try 

 frontend https_frontend

    bind *:443 ssl crt /home/ubuntu/varnish_ssl/stunnel.pem

 PS: 
  • $ cat wild-elatov-local-cert.pem wild-elatov-local-priv-key.pem > elatov-local-cert-key.pem

 Reference:

 http://virtuallyhyper.com/2013/05/configure-haproxy-to-load-balance-sites-with-ssl/


on
Labels: , , , , , , ,

1 comment:

Subscribe to: Post Comments (Atom)