Tuesday, March 23, 2010

High available Fusion Middleware Cluster

On a project I am building a High Available Soa Suite 11g Cluster. This is not so difficult thanks to Weblogic and when you follow the guides of Antony Reynolds or Marc Kelderman, they did a great job.
High Availability for FMW is not so different then for a normal Weblogic Cluster HA.The Soa Suite Documentation contains a great document called Enterprise Deployment Guide For Oracle SOA Suite , this document shows all  the best practices how to setup a High Available Weblogic Cluster.
When you just want a Weblogic Cluster or a Soa Suite Weblogic Cluster for the performance you can follow the guide of Marc Kelderman. This is a cluster without shared storage for the Domain, Transaction logs and JMS servers. This means you can't manual migrate the Admin Server to a other machine and no automatic Managed Server Migration by the Node managers. And more important, you can loose transactions and JMS messages when a Managed Server fails. When you also use resource adapters in the Soa Suite Cluster you need to make sure that the resource adapter deployment plan with all the EIS connections is available on every Managed SOA Server .

To solve this you need to have a shared storage. This shared storage needs to be available on every machine and because of the shared storage, every machine can start the Admin Server or Managed Servers. The migrated Managed Server can pick up the transaction logs and JMS messages of the failed Managed Server( No data loss)
In the Enterprise Deployment Guide Document, Oracle did every right to make a perfect cluster like
  • Using Virtual IP addresses
  • A Domain folder for the Admin Server and a separate one for the Managed Server
  • FMW installation on the Shared storage and two installs one for the Admin and one for the Managed Server
  • Oracle RAC for the leasing table
The only downside of this configuration is the NFS share, when you need to patch or configure this NFS share the cluster must go down. Else it would be possible to never shutdown all the Managed Servers in the Cluster. Maybe Oracle or Sun can solve this by some SAN hardware solution.
And maybe in the next version of this Solution, Oracle can give some more detail how to setup a Weblogic Cluster when every WLS machine has two network interfaces on different switches ( to solve network failures ).

Here are the important chapters you should read to make a HA Soa Cluster besides the document of Marc Kelderman.

Chapter 2.2.3 IPs and Virtual IPs

Chapter 2.3 Shared Storage and Recommended Directory Structure

Chapter 8  Server Migration
  • How to setup the Leasing table in Oracle RAC
  • Multi DataSource
  • Configure the Node Managers
  • Configure the Cluster for migration in the Weblogic Console
  • Test the Server Migration.

Chapter 11.5 Using Shared Storage for Deployment Plans And Soa Infrastructure Applications Updates


  1. Alot of applies as well and is well worth a read.

  2. Couple of more comments:
    You could probably use Network Channels which bind to VIPs allocated to separate NICs for redundancy.

    The only problem I've found with the design Oracle have given is that I can't find a way for Node Manager to start the Admin Server since the domain directory that gets registered with NM is the managed server directory. Any ideas how to solve this when one the Admin Server and one of the Managed Servers share a host?

  3. Hi,

    for the network we have a special server network and every server has 2 connections on the same lan ( 2 different core switches ) and we use Link based IPMP groups on sun solaris to make sure only one nic is active and we changed to find out which nic is active.

    I have one only domain folder and we are now looking at a shared iscsi volume.

    What are you using for shared storage.


  4. We are using SAN which while provides a highly available storage system, does not provide a HA file system. The file system needs to be provided by the OS which needs to run in a OS cluster to get HA on the file system.

    Unfortunately we can't run a our OS as a cluster (politics). Our solution is a separate SAN LUN for each server which can be migrated to the other server if one of the hosts fail.

    However this doesn't solve the problem where we can't start the admin server using node manager.

    I think what we will end up doing is installing the admin server as a windows service on both our OSB Hosts. The backup server will have its service disabled. That way we can have separate domain structures for the admin server and the managed server as described in Oracle Enterprise Deployment Guidance. If we need to migrate the admin server the SAN LUN will get moved to the backup server and then we can start the previously disabled admin server.

    It sounds complicated :-)

  5. Is it possible to have 2 SOA domains connecting to the same database and sharing the same SOA DB schemas? I know that with previous BEA products like WLI you needed a separate DB schema per WebLogic domain. Is this the case with SOA11g domain?

    The reason I am asking is that if you have 2 SOA domains, one in Data Centre A and one in Data Centre B connecting to the same DB Schema, can the one Domain see the other domain's BPM/BPEL long running processes? And when OSB routes messages to these domains, how will it know to which domain to route to in order to continue a conversation with a running process?

  6. Hi,

    I think it is possible, but not recommended, probably you need to make a sort of soa cluster on two domain. and configure coherence, resource adapters and EM wont work I think.

    you can better make 1 soa domain and add 2 managed server who works in a soa cluster. And make every ready for server migration. from site A to B and back.


  7. Hi Edwin,

    great article. I have a question regarding the creation of multiple SOA clusters within the same domain. Would that be possible. If not what is the reason behind it. Thanks.

  8. Hi,

    you can only make 1 soa cluster in a domain and this cluster can contains many managed servers.

    this is because of the soa repository, every soa cluster need its own repos and it expects particular datasource / jms names.


  9. - Thanks for this! I need to know how to setup the leasing table in Oracle RAC as soon as possible.