High Availability Load Balancer for Weblogic Cluster

Oracle Weblogic Application Server has a lot of features to make your Web Applications or Web Services Scalable and High Available. To achieve Scalability you can add managed servers to a WLS cluster and High Availability can be achieved by Server and Service migration. The only thing Weblogic does not provide for you is a one shared IP address for the outside HTTP world. You don't want the user or application connect to one specific managed server of the cluster. For this Oracle made a mod plugin for Apache which can listen on one address and redirect the http request to one of the managed server in a cluster. Ok this is fine but this can be your new single point of failure. So you need some software which can monitor for example the apache process, when it fails this software needs to switch the ip address to the other server so that apache server can do the work. Off course this is also possible in hardware but this in this blog I will show you how this can be done with opensource Linux software.When you have Windows Server then Microsoft NLB can also do this for you.

Required Software
I got this working with Oracle Enterprise Linux 5.5 ( get it from edelivery ) and I installed with all the OEL options  ( Development , Cluster , Web etc ). When you do this you will have all the required libraries and tools to compile the required software. This also provides the Apache Web Servers.
HAProxy is a free very fast and reliable solution offering high availability, load balancing, and proxying for TCP and HTTP-based applications. It is particularly suited for web sites crawling under very high loads while needing persistence or Layer7 processing.
Keepalived this linux program can monitor HAProxy and can switch the shared ip address so that the requests are handled by the second server.
This is a picture of my Weblogic configuration.

Before you begin you need to install Weblogic software ( on both servers ) and configure a Weblogic Domain with a Admin server and a cluster with two managed servers. And off course the Weblogic nodemanagers.

First step is to configure apache on both machines.
You should be root to do this.
Copy the weblogic apache mod to the apache module folder.
cp /WLSHOME/wlserver_10.3/server/plugin/linux/i686/*.so /etc/httpd/modules

Change the httpd.conf
vi /etc/httpd/conf/httpd.conf
Add the module in the module section.
LoadModule weblogic_module modules/mod_wl_22.so

provide the ip addresses and port number of the managed servers in the cluster
<IfModule mod_weblogic.c>
  WebLogicCluster 10.10.10.150:7001,10.10.10.151:7001
</IfModule>

this redirect everything else use /weblogic instead of /

<Location / >
  SetHandler weblogic-handler
</Location>

Start apache on both servers.
cd /usr/sbin
./apachectl start

test it on both servers, check if you can invoke a Web Service or open a Web application on the Apache port number.

The next step is to install and configure HAProxy
Download the latest HAProxy source from http://haproxy.1wt.eu/#down
You should be root to do this.

unzip the source
gunzip  haproxy-1.4.4.tar.gz
tar xvf  haproxy-1.4.4.tar
cd haproxy-1.4.4

Compile HAProxy
make TARGET=linux26 CPU=generic


Copy the haproxy executable to both OEL servers ( /usr/sbin )
cp haproxy /usr/sbin/

Check haproxy by retrieving the version
./haproxy -v

Create the haproxy user and group, you can do this in OEL ( on both servers )

Make the haproxy config file. ( on both servers )
vi /etc/haproxy.cnf
##### begin #####
global
        log     127.0.0.1   local0
        log     127.0.0.1   local1 notice
        maxconn 4096
        user      haproxy
        group   haproxy

defaults
    log              global
    mode          http
    option         httplog
    option         dontlognull
    option         redispatch
    retries         3
    maxconn      2000
    contimeout   5000
    clitimeout     50000
    srvtimeout    50000

listen wlsproxy 10.10.10.40:80
       mode http
       balance roundrobin
       stats enable
       stats auth weblogic:weblogic1
       cookie  JSESSIONID prefix
       option  httpclose
       option  forwardfor
       server  wls1 10.10.10.50:81 cookie wls1 check
       server  wls2 10.10.10.51:81 cookie wsl2 check
##### end #####

10.10.10.40:80 is the shared VIP ip address and 10.10.10.50:81 is one of the Apache server.
stats enable and stats auth weblogic:weblogic1 enables the HAProxy status application with weblogic as username and weblogic1 as password.


Install and configure Keepalived
Download the latest source from http://www.keepalived.org/download.html
You should be root to do this.

Unzip the source
gunzip keepalived-1.1.19.tar.gz
tar xvf keepalived-1.1.19.tar
cd keepalived-1.1.19


Compile and install Keepalived
./configure
make
make install

Because Keepalived uses a shared ip address you need to add a kernel parameter ( on both servers )
vi /etc/sysctl.conf
Add this line to sysctl.conf
net.ipv4.ip_nonlocal_bind=1

Reload the kernel parameters
sysctl -p

Copy these files to both OEL servers
cp /usr/local/sbin/keepalived /usr/sbin
cp /usr/local/etc/rc.d/init.d/keepalived /etc/init.d/
cp /usr/local/etc/sysconfig/keepalived /etc/sysconfig/

Add the keepalived configuration files and do this on both servers
mkdir /etc/keepalived
vi /etc/keepalived/keepalived.conf
##### begin ######
vrrp_script chk_haproxy {          # Requires keepalived-1.1.13
        script "killall -0 haproxy"     # cheaper than pidof
        interval 2                            # check every 2 seconds
        weight 2                             # add 2 points of prio if OK
    }

    vrrp_instance VI_1 {
        interface eth0
        state MASTER
        virtual_router_id 51
        priority 101                    # 101 on master, 100 on backup
        virtual_ipaddress {
            10.10.10.40
        }
        track_script {
            chk_haproxy
        }
    }
##### end #####




You have to decide which server is your primary http server, this one need to have priority 101 and the other priority 100.





Starting and testing load balancing and failover

Start on both servers haproxy
cd /usr/sbin
./haproxy -f /etc/haproxy.cnf

Start on both servers keepalived
cd /etc/init.d
./keepalived start

Now you can check on both servers if the shared ip address is only mapped on the primary server.
ip addr sh eth0

on the primary server
[root@wls1 init.d]# ip addr sh eth0
2: eth0: mtu 1500 qdisc pfifo_fast qlen 1000
    link/ether 08:00:27:3f:68:d6 brd ff:ff:ff:ff:ff:ff
    inet 10.10.10.50/24 brd 10.10.10.255 scope global eth0
    inet 10.10.10.40/32 scope global eth0
    inet6 fe80::a00:27ff:fe3f:68d6/64 scope link
       valid_lft forever preferred_lft forever

on the slave
 [root@wls2 init.d]# ip addr sh eth0
2: eth0: mtu 1500 qdisc pfifo_fast qlen 1000
    link/ether 08:00:27:d1:f2:c0 brd ff:ff:ff:ff:ff:ff
    inet 10.10.10.51/24 brd 10.10.10.255 scope global eth0
    inet6 fe80::a00:27ff:fed1:f2c0/64 scope link
       valid_lft forever preferred_lft forever

On the primary server you can kill the haproxy process and this will failover the shared ip address to the slave. When you start haproxy on the master then the ip address will be moved back to the master.

The last thing you can check is the haproxy status application. Go to http://10.10.10.40/haproxy?stats and log in as weblogic with password weblogic1.
Here a picture of the status application.