Saturday, May 8, 2010

Setup secure node managers and enable the server migration option

In Weblogic it is not so hard to make a WLS Cluster for your java applications or make a Fusion Middleware cluster for your ADF, Webcenter applications or SOA / OSB processes. But to remotely control this cluster from the console web application you need to setup nodemanagers. If you want to go a little step further and also wants High Availability you need to enable and setup Server Migration in Weblogic. Server Migration only works when you use nodemanagers on all your servers. You can choose for a script nodemager so called SSH nodemanager or for a Java nodemanager.
And the question is, which nodemanager should you choose. 

When you have a normal Weblogic domain ( Don't enable any of the oracle_common options in the Weblogic domain wizard) then you can follow the SSH node manager guide. In this guide you will make a ndmgr account where you copy the weblogic domain from the main domain  location into the home directory of ndmgr account. Enable SSH without password between the weblogic and nodemanager accounts. The Admin Server will fire remote scripts over SSH to control the Managed Servers. There is no agent running only scripts. I think the SSH nodemanager is also possible with a Fusion Middleware Cluster but you can't create or use a ndmgr account and copy the domain to a home directory or to an other path. In Fusion Middleware you can copy the domain to a new folder next to the orginal domain. ( the java jars /ears are loaded on startup with a relative path so in a other path it can't find the ears) and use this domain for your managed servers. So don't use the SSH nodemanager with FMW. For a normal weblogic domain I would choose for SSH , because you can copy domain to the home directory of an account. And the nodemanager does not take any resources.

The first part of this blogpost is about the SSH Nodemanager and the second part is about the Java Nodemanager

Before you can follow these steps you need to install Weblogic ( /oracle/Middleware ) and use oracle as user and group. I will show you the steps for Oracle Enterprise Linux ( RedHat )

SSH Nodemanager
  • Create a user ndmgr which belongs to the oracle group and a home directory /home/ndmgr
  • log on as ndmgr and in the home directory a hidden ssh folder. mkdir.ssh and cd .ssh
  • create a rsa key ssh-keygen -t rsa ( accept the location and use no password )
  • fix the ssh rights by making a keys file copy authorized_keys
  • change the permissions of the .ssh folder cd .. and chmod 700 .ssh
  • Log in as root and copy this .ssh folder to all the oracle and ndmgr accounts on all machines
  • Change the owner of this folder chown oracle:oracle .ssh or chown ndmgr:oracle .ssh
  • For Server migration and fire sudo commands over SSH you need to edit  vi /etc/sudoers this allows ifconfig command for the oracle and ndmgr account
  • Add oracle ALL=NOPASSWD: /sbin/ifconfig,/sbin/arping
  • Add ndmgr ALL=NOPASSWD: /sbin/ifconfig,/sbin/arping
  • Enable sudo over ssh by disabling this rule Defaults  requiretty
  • Restart ssh cd /etc/init.d and ./sshd stop and ./sshd start
Log in as oracle and test the ssh  connections to all the servers  ssh -l ndmgr server1_ip_address

Test adding a new ip address with the weblogic script.
cd /oracle/Middleware/wlserver_10.3/common/bin
./ -addif eth0
./ -removeif eth0

The next step is create a Weblogic domain and copy this to ndmgr account.
  • Start the domain wizard cd /oracle/Middleware/wlserver_10.3/common/bin and start ./
  • Don't enable the oracle_common options
  • log in as root and copy this domain from the user_project folder to the ndmgr home directory like cp -R /oracle/Middleware/user_projects/domains/wls_domain /home/ndmgr
  • Change the rights to the ndmgr user chown ndmgr:oracle /home/ndmgr/wls_domain
  • log in as ndmgr and create in the home directory a new file vi
  • Add the local domain path /home/ndmgr/wls_domain
  • Go to the bin folder of your local domain and change in each script the domain variable value to the home directory path.
  • Create a file for each managed server because else weblogic will ask for a password and this will not work remotely. cd /home/ndmgr/wls_domain  and  mkdir -p servers/Server_1/security Do this for every server
  • add username=weblogic and password=weblogic
You can test  it when you log in as oracle and try to ask the status of a managed server.
ssh -l ndmgr -o PasswordAuthentication=no  Server_ip_address /oracle/Middleware/wlserver_10.3/common/bin/ -d wls_domain -n /home/ndmgr -c -f -s \'Server_1\' STAT

The SSH nodemanager does not have a file so you need to change the settings in the oracle/Middleware/wlserver_10.3/common/bin/ file. In our case you need to add the network device name and subnet mask.

The last part of the SSH node manager is to configure these node manager in the weblogic console.
  • go to http://xxxx:7001/console
  • Environment / Machines  and select the right servers
  • In the Node Manager Tab use the following values
  • Type = SSH 
  • Listen Address=  your server ip address
  • Node manager home = /home/ndmgr
  • Shell Command = ssh -l ndmgr -o PasswordAuthentication=no -p %P %H /oracle/Middleware/wlserver_10.3/common/bin/ -d %D -n /home/ndmgr -c -f -s %S %C
for more information see the Oracle Weblogic documentation

Java Nodemanager
For you can configure the nodemanager you need to start the node manager once. This will create the file.
  • cd /oracle/Middleware/wlserver_10.3/server/bin
  • ./

Enable sudo for the oracle account
  • For Server migration you need to edit  vi /etc/sudoers this allows ifconfig command for the oracle  account
  • Add oracle ALL=NOPASSWD: /sbin/ifconfig,/sbin/arping
  • Restart ssh cd /etc/init.d and ./sshd stop and ./sshd start

Test adding a new ip address with the weblogic script
cd /oracle/Middleware/wlserver_10.3/common/bin

./ -addif eth0
./ -removeif eth0

For secure communication between your Admin Server and the Java Nodemanamger you need to create some certificates. wls1.local matches with my machine name. Important for hostname verification.
  • java utils.CertGen welcome wls1_cert wls1_key domestic wls1.local
  • java utils.ImportPrivateKey wls1KeyStore.jks welcome wls1 welcome /oracle/Middleware/base_domain/wls1/certs/wls1_cert.pem /oracle/Middleware/base_domain/wls1/certs/wls1_key.pem

Make a new truststore based on the weblogic truststore
  • cp /oracle/Middleware/wlserver_10.3/server/lib/cacerts /oracle/Middleware/base_domain/wls1/certs/wls1TrustKeyStore.jks
  • cd /oracle/Middleware/base_domain/wls1/certs/
  • keytool -storepasswd -new welcome -keystore wls1TrustKeyStore.jks -storepass changeit
  • keytool -import -v -noprompt -trustcacerts -alias clientCACert -file /oracle/Middleware/wlserver_10.3/server/lib/CertGenCA.der -keystore wls1TrustKeyStore.jks -storepass welcome
Configure the by adding the network interface / subnet mask and the right certificates
  • cd /oracle/Middleware/wlserver_10.3/common/nodemanager
  • vi
  • Add or change these lines 
  • NetMask=
  • Add the security parameters
  • KeyStores=CustomIdentityAndCustomTrust
you are ready with the nodemanager configuration and can go to the weblogic console application.
  • go to http://xxxx:7001/console
  • go to environment / machines and select the right machine and go the Node manager Tab
  • Change the node manager field
  • Type = SSL
  • Listen Address = your server ip address
  • Listen Port = 5556
Next step is to change the server keystore and ssl properties
  • go to environment / servers and select the right server
  • Go to the Keystores Tab of the configuration Tab and use these values
  • Keystores = Custom Identity and Custom Trust
  • Custom Identity Keystore = /oracle/Middleware/base_domain/wls1/certs/wls1KeyStore.jks
  • Provide the Custom Identiy Keystore Passphrase
  • Custom Trust Keystore = /oracle/Middleware/base_domain/wls1/certs/wls1TrustKeyStore.jks
  • Provide the  Custom Trust Keystore Passphrase
  • go the SSL Tab
  • Private Key Alias = wls1
  • Provide Private Key Passphrase 
That's all for the java node manager

Enable Server migration
To enable server migration you need to have a database with a table called ACTIVE. The scripts for all database are located at /oracle/Middleware/wlserver_10.3/server/db. for HA you can choose Oracle RAC or MySQL Cluster ( the community edition , this is very light and free ).

See this chapter how to configure WLS for this leasing table.

After that you can set a virtual ip address on your managed servers and configure the cluster and managed servers for migration. See this Oracle Chapter


  1. Hi ,

    Thanks a tonne for this post. Being a novice weblogic administrator I was really looking for a article with detailed steps to configure nodemanager along with truststore and certificate creation and this page catered to my expectations.


  2. Hi,

    This post is really informative, thanks! I do have some followup questions:

    - What if we want to use one database (Oracle) to host the leasing tables for several WebLogic clusters? I understand I can create several 'leasing' users and each have its own leasing schema. For example, I can create a LEASING tablespace on the database, and then a leasing_dom1, leasing_dom2 user accounts and schemas for each domain, and then create the two tables. Or can I just create the LEASING user schema to be shared by multiple domains?

    - Some of the WebLogic clusters may be separated from the database via a WAN. How much traffic to the database would be expected from database leasing to a remote database? Would the latency of the WAN be a problem?


  3. Hi,

    you can do it in 1 leasing table , the only thing you managed server must have a unique name.

    there not much traffic so it wont be a problem. only the wan needs to be reliable.


  4. Hi Edwin Biemond,

    I have configured the Script Based node manager as you mentioned mentioned.
    Through admin console i am able to start the remote managed servers psMS1 , psMS2. –> Adminserver , psMS1 –> psMS2

    We have create Two unix machine Unix1 , Unix1.

    Unix1 –> assign servers Adminserver , psMS1. -->
    Unix2 –> assing server psMS2. -->

    The two machine have same subnetmask , eth0

    The Two manager servers (psMS1,psMS2) is configured with Cluster (CLUST).

    But If i configure the Whole server migration option in the weblogic Admin console. I am unable to start Two managed servers. I mentioned the Two managed severs errors logs.

    psMS1 Error log :- ( ) –> Unix1 machine

    psMS2 Error log :- ( ) –> Unix2 machine

    How to resolve the problem during the Script based node manager during the Whole Server Migration in Clustered Environment(CLUST).

    In my knowledge the main reason is " "

    S.Vinoth Babu

    1. Greetings,

      I have configured 2 manged servers to running in cluster but when i try to configure automatic server migration the 2 managed servers failed the 2 servers each one has 2 ips one is meachine ip and second floating ip and i update serer listening for each one to listen to floating ip and i edited , grant oracle user sudo privalages

      could anyone help me ?

      All thank's

  5. Hi,

    can you encode the xml to text, then I can see the error.


  6. Hi Edwin Biemond,

    I resolved that issue.In my configuration file is not available because i am using the script base node manager. The is available in the Jave based node manager. The error main reason is i am not mentioned the ip address for cluster. The log error message shows the cluster ip address is Null. Then i configured the ip address to cluster then the problem is resolved.

    I have one doubt the Whole server migration is only available in the Script based nodemanager in Oracle weblogic document. But you have configure the Java Based Nodemanager for server migration. Kindly give me the valuable advise.

    S.Vinoth Babu

  7. Hi,

    good you solved it.

    I think server migration should also work with the script node manager. both will use the script for the virtual ip.

    did you get an error.


  8. Greetings, I just wanted to comment and say that I was really impressed with your blog. Keep up the good work! You are a really talented writer and it shows.