Pages

Wednesday, February 18, 2009

High Available WebLogic Cluster

For one of my customer I am building a WebLogic cluster to provide high availible JMS services and web applications. In this blog I will give you some hints how to achieve this.
First a picture of the WebLogic Cluster configuration.

On the left site I have an normal Apache http server version 2.2 with a weblogic plugin. This plugin does the http forwarding to one of WLS server instances. Then I have two servers where the WLS are running on. On the first WLS server I am running the admin server with one of the cluster server instances and on the second WLS server is the other Cluster server instance running. Every server instance has its own JMS server and the WLS cluster has one JMS module (in here you define the distributed queues).
The JMS messages of the JMS servers, HTTP session data and WLS security realm configuration are all stored in the MySQL cluster database. This memory database ( a bit like coherence then with tables instead of java objects) provides me a high available WLS data. Off course you can use Oracle RAC or a single database for this data, but I prefer the MySQL cluster because I just need a few tables and no PLSQL + the MySQL cluster is up and running in a few hours. Or you can use file persistence but then you need to do something with scripts or cluster file systems when there is a failover.
This cluster configuration has the following failover features.
  1. The Apache redirects the http session to one of the servers. Session data is stored in the database so each server can handle the http request.
  2. WLS Server instance failover. Each server has a dynamic ip address when a server instance fails, the nodemanager will add the ip address to the other server and start the WLS server instance so you always have 2 WLS server instances.
  3. WLS Service failover, when for example a JMS server fails then this JMS server will be started in the other WLS server instance so you always have two JMS servers.
  4. MySQL cluster node failure. WebLogic uses for all their configuration an multi jdbc datasource so when one MySQL cluster node fails the other datasource is used to connect to the other MySQL cluster node
To setup a WLS domain see my previous blogpost. or see this blogpost for how to setup a MySQL cluster .

Here some pics how to configure high availability in WebLogic. First we need to configure the cluster for failover. In my case I select all the servers as migratable servers and using Consensus as migration basis.

Every WLS server instance need to have its own migratable target. On this migratable target we add the JMS servers

Set automatic migrate recovery services on these migratable targets.

After this we need to set an dynamic ip address to every Weblogic Server instance. Just select an server and set the ip address. Now we need to configure the WLS nodemanager on every server and put in the netmask and interface parameter in nodemanager.properties.
NetMask=255.255.255.0
Interface=eth0
Now the wlsifconfig script can add the ip address to the network device and start the WLS server instance.

The server configuration is ready we can go on with the JMS part.

Step 1 configure the JDBC persistence stores and target these to the migratable targets.

Step 2, create the JMS servers on a migratable target and every JMS server has its own jdbc persistence store

Create a JMS systemmodule

Create an connection factory and a distributed queue both targeted on the cluster.

That's all for the JMS part.

The last part is to configure Apache and the enterprise applications for High Availability.
Install apache 2.2 and download the weblogic module for apache here.
edit the httpd.conf add the following lines

LoadModule weblogic_module modules/mod_wl_22.so

<ifmodule>
WebLogicCluster node1_ip:port,node2_ip:port
</ifmodule>

<location>
SetHandler weblogic-handler
</location>


Step 2 is to configure every enterprise application ( ADF app or a WS) with a weblogic deployment descriptor so it stores its session data in the database.

<?xml version="1.0" encoding="ISO-8859-1"?>
<weblogic-application xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.bea.com/ns/weblogic/weblogic-application.xsd" xmlns="http://www.bea.com/ns/weblogic/weblogic-application">
<session-descriptor>
<persistent-store-type>jdbc</persistent-store-type>
<persistent-store-pool>JDBCMultiDataSourceMHS</persistent-store-pool>
<sharing-enabled>true</sharing-enabled>
<persistent-store-table>WL_SERVLET_SESSIONS</persistent-store-table>
<jdbc-connection-timeout-secs>120</jdbc-connection-timeout-secs>
</session-descriptor>
</weblogic-application>


that's all

42 comments:

  1. Edwin, any particular reason you plan on using database replication instead of in-memory for session replication between cluster members? It is my understanding that in-memory has better performance.

    ReplyDelete
  2. Hi Jambay,

    I read somewhere in the documentation that for JMS etc. the database gives you the highest availabilty. But I don't know if this still the case with http sessions.

    Off course you can use coherence too but we already had an mysql cluster which is a memory database.

    thanks Edwin

    ReplyDelete
  3. Hi Edwin,

    What happens if JMS server goes down when message is getting processed, like say one of the managed server which is handling the JMS fails? I have heard there is no feature (yet) in weblogic which provide us complete JMS failover -Only writing of messages in queues is handled by failover but processing has to be done by the managed server which initiated it -is this true?

    ReplyDelete
  4. Hi , I don't know exactly but I hope that the jms server which uses a file or database persistence store should delete the message from it's store when it successfull is delivered. The app which supplies the message to wls. should use XA so there is no lost messages.

    ReplyDelete
  5. Hi Edwin,
    Would you, please, explain:
    1. " ...we need to set an dynamic ip address to every Weblogic Server instance. Just select an server and set the ip address" - Does it mean to set the managed servers Listen Address ?
    2.in nodemanager.properties:
    Interface=eth0
    What is eth0 - the name of the NetWork adapter ? How to see it?
    3. "...Now the wlsifconfig script can add the ip address to the network device" - Should we use floating IPs? I mean if I have a static IP on my machine, should it be "Obtain an IP automatically" ?

    Thank you very much !
    George

    ReplyDelete
  6. Hi

    1. " ...we need to set an dynamic ip address to every Weblogic Server instance. Just select an server and set the ip address" - Does it mean to set the managed servers Listen Address ?

    Yes wls will use the nodemanager to set this ip on the network device. it will use ifconfig.

    2.in nodemanager.properties:
    Interface=eth0
    What is eth0 - the name of the NetWork adapter ? How to see it?

    eth0 is the network adapter name of your OS. and in windows this is your network name

    3. "...Now the wlsifconfig script can add the ip address to the network device" - Should we use floating IPs? I mean if I have a static IP on my machine, should it be "Obtain an IP automatically" ?

    yes floating ip is the best , when a server fails , wls will failover this server instance and use this ip on a other server OS.

    thanks Edwin

    ReplyDelete
  7. James,
    I am trying to do a whole server migration with only one active node in a cluster. All we need is a simple failover with no load balancing or replication. I have seen on the oracle docs that weblogic does support a cold failover (active - passive failover) with this 'whole server migration' concept.

    I am trying to do this but confused with some of the steps.

    Please help me, its an urgent issue for us to achieve the agreed SLA's and we are got past the deadlines with no progress.

    1. for the floating ips to work, do I need 2 NICS and two cables on each server?
    2. Do I have to configure 2 different ips on one interface(eth0), which will be used as node manager's listen address's for each of the 2 machines?
    3. What will be the listen address of each of these servers(primary server & failover server)? Is it the floating ip address? or empty (all local addresses)?
    4. can we start both the primary server & failover server so when primary goes down, the failover will be quick?

    Your help is much appreciated. my email id is poluruc@gmail.com

    Thanks,
    Chandra

    ReplyDelete
  8. Hi


    1. for the floating ips to work, do I need 2 NICS and two cables on each server?

    No you can add more ip's on 1 nic, in unix you can for example use ifconfig to add a second ip on eth0

    2. Do I have to configure 2 different ips on one interface(eth0), which will be used as node manager's listen address's for each of the 2 machines?

    you need one fixed ip for the nodemanager / server. And the wls server has a dynamic ip

    3. What will be the listen address of each of these servers(primary server & failover server)? Is it the floating ip address? or empty (all local addresses)?
    servers has a floating ip and this server runs on the primary server and when it fails it starts on the failover. admin server can have a fixed ip.

    4. can we start both the primary server & failover server so when primary goes down, the failover will be quick?
    No a migrateable server can only run on 1 hardware server

    thanks Edwin

    ReplyDelete
  9. Hi Edwin,
    I am sorry for the name confusion in my earlier post.

    Thanks a lot for your reply, it clears many things.

    So we dont need 2 inerfaces to configure this. We can just use one interface, say eth0 on both machines configured with a fixed ip 10.0.0.1 on machine A and 10.0.0.2 on machine B.

    We can use 10.0.0.200 as the floating ip, with interface name eth0 and same net mask value on both machines.

    I am trying this now, I will update you the progress.

    Thanks a lot for you help.

    ReplyDelete
  10. Hi Edwin, in your setup are you sharing any part of the file system between server1 and server2? The reason why I ask is that we are trying to set up whole server migration and are sharing the complete Domain directory between servers and are getting errors like "unable to obtain lock ...." on ldap or .lok files when migrating from server1 to server2. In the tested situation server1 gets powered off.

    Regards, John

    ReplyDelete
  11. Hi

    I don't use the file system because this is too expensive with sun so I use a cluster database.

    Why are sharing the domain directory, You only have to copy this once on every server.

    thanks Edwin

    ReplyDelete
  12. I accidentally rejected this message

    Hi Edwin,

    i have some query
    1-while creating a managed server what ip shall be given,the physical
    ip of the machine or the floating ip,

    floating when you want to have HA

    2-can node manager able to start a server using floating ip?

    yep, the nodemanager has a fixed ip and uses a script to add the floating ip to the nic and after that its start the server

    thanks edwin

    ReplyDelete
  13. Hi Edwin

    I need your help. We are intend to create weblogic 10.3 installed on windows o/s - cluster solution which has to support HP/Failover. ie., when primary server cluster node fails on machine a, then the request and current status should handled by secondary server cluster node on machine b. Please provide me the detail configuration steps to support this solution.

    ReplyDelete
  14. Hi,

    You dont need anything special just two servers and a database or shared file system. basically you need a common location where both wls servers can read and write. you can run both servers a wls server.when one fails then wls will migrate the ip address and start a second server with the ip addess of the server.

    you can also make a Operating cluster with one wls server. when one server fails it start the second server.

    thanks

    ReplyDelete
  15. Hello Edwin,

    great post!
    One comment: wouldn't the server migration be enough? It will automatically migrate the JMS instance hosted on that server as well, wouldn't it?
    Is it really advisable to use both at the same time?
    thanks, for your feedback,

    Frank

    ReplyDelete
  16. Hi,

    Server migration is possible from version 10.3. I Don't know if it possible that the server continues for the webapps and the jms server is migrated to a other managed server.

    I need to investigate this more.

    thanks

    ReplyDelete
  17. Hi Edwin,

    Great post.

    Could you please share your comments on the below design.

    I have used Glassfish App Server 9.2 to create Cluster for one of the Insurance Giant in US. Mentioned below are the details:

    1. Created one virtual server as a entry point of my cluster. Lets say 9001
    2. Created one load balancer on sun web server 7. Using Glassfish load balancer utility. Lets say 9002
    3. Created one cluster on two machines each with two instances. Lets say 9003 and 9004
    4. I have used Active-Active environment with session replication and High Availability Database(HADB with MySQL database) for failover.

    Everytime a request comes to virtual server, it redirect it to load balancer. Load balancer then distribute the load between two server in clustered environement. I have configured failover mechanism using glassfish inbuilt utility, so if during any request if one of the server(9003) goes down, the session will be replicated from 9003 to 9004. It will migrate all user data along with session to other server. The reason being two server kept Active-Active is to utlised the resource of the clustered servers.

    My application is live and giving great performance. But still if you think I could do some best practices on top of that, your thoughts are welcome.

    Cheers,
    Allwyn
    Solution Architect
    BA.com

    ReplyDelete
  18. Hi,

    great setup and it is cool that is working fine.

    is your virtual server & load balancer not a single point of failure.

    and your web applications deployed on this cluster do they have EJB / JPA as a model, then it probably need a JTA transaction manager. What happens with these transactions logs when there is a crash.

    and is your database a cluster db like Oracle RAC, MySQL Cluster ( has nodes ) or is there some failover mechanism, then probably in those failover seconds your connections will be dropped.

    Hope this helps,

    thanks

    ReplyDelete
  19. Hi,

    Thanks for reply.

    Yes, my virtual server and load balancer are both separate servers. It is because to hide the actual IP sets from Internet. And expose only the virtual server, which do not have any business running.

    Also I don’t have EJBs in my application (as there is no financial transaction).

    My database is 2-nodes cluster. And both nodes are Active-Active mode.
    Do you still think there would be loss of connection while transferring data from one db server to other even though they are Active-Active.

    Cheers,
    Allwyn Panchwatikar
    Solution Architect
    BA.com

    ReplyDelete
  20. Hi,

    no comments , you did well. this should work.
    Only the virtual server and load balancer are a single point of failure. but this does not mean the cluster is not HA. when the virtual server fail then the cluster is not accessible.

    thanks

    ReplyDelete
  21. Hi Edwin,

    Good blog. I have just a tricky question regarding JMS configuration update on a Weblogic cluster environment.

    We have setup a two-managed-server cluster, each one of which has a dedicated JMS server and persistent store.

    The tricky question is, in case we need additional managed servers added to the cluster, is there a way for the associated JMS server and persistent store to be created dynamically or do we have to do it manually?

    Our customer requires a horizontal scaling capable solution. They want to add managed servers at will, so we would like to know whether a new managed server means additional manual configuration work or is there a way to add them dynamically?

    Thanks a lot in advance, again congrats on your blog.

    Nasos.

    ReplyDelete
  22. Hi Nasos.

    No problem, you can do it all with scripting. And with the new OEPE you got even an IDE with some templates for it. See my blog http://biemond.blogspot.com/2010/08/wlst-scripting-with-oracle-enterprise.html

    The only thing you have to do , is to install wls and copy the domain with (un)pack and configure the nodemanager. see http://biemond.blogspot.com/2010/05/setup-secure-node-managers-and-enable.html

    After that you can make a script where you create a new managed server on the new machine a filestore with a jms server.

    thanks

    ReplyDelete
  23. Hello Edwin,

    Nice post, I learned a lot for it.

    But I still have a small question regarding the Weblogic appache plugin to load balance the requests.

    This is working perfect for webservices, but I have a problem for the jms service.

    I recieve the following error :
    Tunneling result unspecified - is the HTTP server at host: 'xxx.xxx.xxx.xxx' and port: '55205' a WebLogic Server?; No available router to destination

    ReplyDelete
  24. Hi,

    the apache plugin can only be used for http,

    for jms you should use the t3://x.x.x.x,x.x.x.x:7001 url and use the weblogic remote client.
    Jms spec does not support clustering but in wls you can create distributed queue and failover messages when they are not picked up on 1 managed server.

    dont know if you can do jms over http with the apache plugin, never tried it.

    thanks

    ReplyDelete
  25. Edmond,
    You shud b able to do jms over http. She might want to search http tunnelling. I guess most app servers support it.

    Chandra

    ReplyDelete
  26. Hi Edwin,
    You said that for JMS we should use t3://ip1,ip2:port.
    I also read somehwere that there is something called cluster ip. so if the cluster ip is 'clip', the same can be achieved by using
    t3://clip:port
    Is this understaing correct.
    In the same article also read that best practice is to have a DNS name(dnsname) for the clip and then use
    t3://dnsname:port.
    Please provide some clarification.
    Also , how does t3s work. do we just need to have secure ports configured and then use
    t3s://ip1,ip2:sport

    Thanks for all the info you already have posted.

    Regards,
    Aniket

    ReplyDelete
  27. Hi,

    I never heard of a cluster ip. Where did you read this, I will check it out and test it.

    t3 and t3s is almost the same, it is like rmi, dont know why it should be encrypted, binary is hard to read. For 1 project we using this, it does some ssl handshaking and secure transport.
    So you need to setup your keystores

    thanks Edwin

    ReplyDelete
  28. Hello, Edwin.

    I'm new of Oracle Products.

    Now I'm installing Weblogic and OSB in my system.

    I have 2 application servers. What is the appropriate architecture?

    I need to install as clustered connection. Is it required proxy server? btw, i have hardware load balancer.

    Thank you
    Erdenebayar

    ReplyDelete
  29. Hello Edwin,

    Learnt a lot. Great post!!!

    I am new with using floating IP address in the cluster for sever migration, I request you to please provide me some inputs.

    I am trying to implement ‘server migration’ using WebLogic 10.3 on our application. We are not using any proxy plug-in for load balancing, we are concentrating mainly on HA & Failover.
    The planned cluster architecture is as follows:
    • 2 managed servers (MS1 & MS2)
    • 2 machines(Machine1 & Machine2 are two Linux machines with static IP each)
    • MS1 on Machine1 and MS2 on Machine2 (all are clustered)
    • 1 admin server outside the cluster running on Machine1
    • Using SAN storage for persistence.
    • We have a static IP for each machine.
    • war and ear deployed to the cluster
    • We don’t have an issue even if web app does not failover.

    Q1. Is server migration possible if proxy plug-in is not used?
    Q2. How to create and configure the floating IP?
    Q3. What IP should be used in the listen address section of MS1 and MS2. Should this be the floating IP?
    Q4. We have an external application which communicates with the WLS application(.ear deployed) using IP:Port. Will it be able to access using the same IP:port even after server migration.

    Please provide your inputs and suggestions, it would be of great help!

    Thank you,
    Kumar

    ReplyDelete
  30. Hi,

    Q1. Is server migration possible if proxy plug-in is not used?

    yep , you dont need this, only thing is that without proxy you connect with http to 1 managed server. with server migration you are then 1 minute offline. t3 has cluster aware stub, http not

    Q2. How to create and configure the floating IP?

    configure nodemanager , with a fixed ip, interface and subnet.
    put a not existing ip on a managed server , and enable server migration and start it up.
    http://biemond.blogspot.com/2010/05/setup-secure-node-managers-and-enable.html



    Q3. What IP should be used in the listen address section of MS1 and MS2. Should this be the floating IP?

    yep, and nodemanager and wlsifconfig.sh put this ip on your nic.

    Q4. We have an external application which communicates with the WLS application(.ear deployed) using IP:Port. Will it be able to access using the same IP:port even after server migration.

    yep , ip moves with the wls managed server to the other server. so your client only notice the migration time,

    good luck

    ReplyDelete
  31. Thank you so much for your quick response, it is really helping us.

    I have one more point o clarify, please suggest.
    Q) Should the floating IP be created for both the machine each or one floating IP for both the machines.

    Highly appreciated!!

    Thank you,
    Kumar

    ReplyDelete
  32. Hi,

    Should the floating IP be created for both the machine each or one floating IP for both the machines.

    you need 1 IP for every managed server you want to migrate. The floating IP travels with the wls managed server to a machine.

    with a cluster of 2 managed servers you need 2 IP. with 3 servers you need 3 IP's

    thanks

    ReplyDelete
  33. Hi Edwin,

    Thank so much for your quick education.

    I have some questions regarding your response on Q1.

    “”Q1. Is server migration possible if proxy plug-in is not used?
    yep , you dont need this, only thing is that without proxy you connect with http to 1 managed server. with server migration you are then 1 minute offline. t3 has cluster aware stub, http not””

    Q1) What do you mean by “t3 has cluster aware stub, http not”?
    From my readings, WebLogic provides the EJB client applications with cluster-aware stubs that transparently perform load balancing and failover. Failover and
    load-balancing of EJBs is done with replica-aware stubs.

    Q2) Is there a way to overcome down time? Is this downtime because of the migrateable server gets started at the time of server migration?

    Q3)Before server migration will the migrateable server be down? Will the migration (with floating IP) not work if the migrateable server is up and running?


    Regards,
    Kumar

    ReplyDelete
  34. Hi


    Q1) What do you mean by “t3 has cluster aware stub, http not”?
    From my readings, WebLogic provides the EJB client applications with cluster-aware stubs that transparently perform load balancing and failover. Failover and
    load-balancing of EJBs is done with replica-aware stubs.

    yep that is true , t3 and the weblogic.jar can do this for you. for http this is not possible. the client can only have 1 connection and you need a loadbalancer or proxy

    Q2) Is there a way to overcome down time? Is this downtime because of the migrateable server gets started at the time of server migration?

    no, not possible , only a proxy or loadbalancer can redirect connection to the other still active server

    Q3)Before server migration will the migrateable server be down? Will the migration (with floating IP) not work if the migrateable server is up and running?

    no, it is only down when you do migration. it is active - active situation not active - passive

    thanks

    ReplyDelete
  35. Edwin,

    You haven't mentioned how you achieve JTA failover. From my research this is important in the case of JMS since XA transactions are a must. We have seen production freezes as a result of using the default persistent store.

    Regards,
    Simon

    ReplyDelete
  36. Hi,

    Indeed, you need to put the JTA logs on a shared storage (which is expensive) so in case of a failover the other server can also read them.

    but don't know what happens when failover takes too long.

    We use for JMS a database persistence in a mysql cluster. it is really fast and HA. We get crazy fast result (every message size is 1mb) and can do failover without any problem.

    what are your problems with file persistence. Operating System or Disc related?

    thanks

    ReplyDelete
  37. Edwin,

    In honesty the problems are due to stupidity. The default store location was never set so the JTA logs are not available when the server failover occurs (since they are on the failed server).

    What do you use to provided a shared file store?

    The documentation talks about a SAN disk or dual ported SCSI drive. I assume that means that the drive is only actually mounted on one server at a time?

    Regards,
    Simon

    ReplyDelete
  38. Hi Simon,

    What do you use to provided a shared file store?

    We use jdbc persistence store and we use a MySQL Cluster, so in case of a failure, Weblogic always can retrieve the messages from the database. You can also use Oracle RAC.

    Our client app has a transaction manager and use two phase commit. So the consuming and the producing of a message is in one transcation. This ways we never a loose message. In our setup we can kill a mysql ndb node , weblogic / jms server and the client app without any problem and still achieve a high performance.

    The documentation talks about a SAN disk or dual ported SCSI drive. I assume that means that the drive is only actually mounted on one server at a time?

    yep you are right, WebLogic can start a script before and after the server migration which can do this.

    thanks

    ReplyDelete
  39. Hi Edwin,
    Is there any documentation to set up load balancer on WLS from scratch. I would prefer to use all oracle products for web server (OHS) and other components wherever possible. The requirement is that we can have multiple servers which are not in cluster. We need to load balance between these servers. They are in the same domain but not in a cluster. They would however have the same application deployed.

    Need stateless, horizontal active-active, single service on a node, load balancing among the severs using software load balancer.

    Can you please help?

    Thanks
    Abhishek

    ReplyDelete
  40. Hi,

    for setting up a webtier you casn follow this blog http://andrejusb.blogspot.com/2010/06/oracle-webtier-11g-configuration-for.html

    but Oracle also uses apache so called oracle http server.

    but I think they wont use this in the future , they move to the sun one products

    like this http://biemond.blogspot.com/2010/09/using-oracle-iplanet-web-proxy-server.html

    so if I was you I would use apache and set some forwarding rules. easy , lot of info / doc and lot of people can help you.

    thanks

    ReplyDelete
  41. Hello All,

    i am new to this weblogic administration, cluster environment,

    Can some body help me how to start the servers node manger, admin server, managed server, weblogic server in sequence?

    thanks in advance.

    ReplyDelete
  42. Dear Edwin,

    we need to configure failover horizontal clustering am newbie to clustering can please tell me the steps and prereqs it's very helpful us asap.

    thanks in advance

    ReplyDelete