Setup secure node managers and enable the server migration option

In Weblogic it is not so hard to make a WLS Cluster for your java applications or make a Fusion Middleware cluster for your ADF, Webcenter applications or SOA / OSB processes. But to remotely control this cluster from the console web application you need to setup nodemanagers. If you want to go a little step further and also wants High Availability you need to enable and setup Server Migration in Weblogic. Server Migration only works when you use nodemanagers on all your servers. You can choose for a script nodemager so called SSH nodemanager or for a Java nodemanager.
And the question is, which nodemanager should you choose. 

When you have a normal Weblogic domain ( Don't enable any of the oracle_common options in the Weblogic domain wizard) then you can follow the SSH node manager guide. In this guide you will make a ndmgr account where you copy the weblogic domain from the main domain  location into the home directory of ndmgr account. Enable SSH without password between the weblogic and nodemanager accounts. The Admin Server will fire remote scripts over SSH to control the Managed Servers. There is no agent running only scripts. I think the SSH nodemanager is also possible with a Fusion Middleware Cluster but you can't create or use a ndmgr account and copy the domain to a home directory or to an other path. In Fusion Middleware you can copy the domain to a new folder next to the orginal domain. ( the java jars /ears are loaded on startup with a relative path so in a other path it can't find the ears) and use this domain for your managed servers. So don't use the SSH nodemanager with FMW. For a normal weblogic domain I would choose for SSH , because you can copy domain to the home directory of an account. And the nodemanager does not take any resources.

The first part of this blogpost is about the SSH Nodemanager and the second part is about the Java Nodemanager

Before you can follow these steps you need to install Weblogic ( /oracle/Middleware ) and use oracle as user and group. I will show you the steps for Oracle Enterprise Linux ( RedHat )

SSH Nodemanager

• Create a user ndmgr which belongs to the oracle group and a home directory /home/ndmgr
• log on as ndmgr and in the home directory a hidden ssh folder. mkdir.ssh and cd .ssh
• create a rsa key ssh-keygen -t rsa ( accept the location and use no password )
• fix the ssh rights by making a keys file copy id_rsa.pub authorized_keys
• change the permissions of the .ssh folder cd .. and chmod 700 .ssh
• Log in as root and copy this .ssh folder to all the oracle and ndmgr accounts on all machines
• Change the owner of this folder chown oracle:oracle .ssh or chown ndmgr:oracle .ssh
• For Server migration and fire sudo commands over SSH you need to edit  vi /etc/sudoers this allows ifconfig command for the oracle and ndmgr account
• Add oracle ALL=NOPASSWD: /sbin/ifconfig,/sbin/arping
• Add ndmgr ALL=NOPASSWD: /sbin/ifconfig,/sbin/arping
• Enable sudo over ssh by disabling this rule Defaults  requiretty
• Restart ssh cd /etc/init.d and ./sshd stop and ./sshd start

Log in as oracle and test the ssh  connections to all the servers  ssh -l ndmgr server1_ip_address

Test adding a new ip address with the weblogic script.
cd /oracle/Middleware/wlserver_10.3/common/bin
./wlsifconfig.sh -addif eth0 10.10.10.150 255.255.255.0
./wlsifconfig.sh -removeif eth0 10.10.10.150


The next step is create a Weblogic domain and copy this to ndmgr account.

• Start the domain wizard cd /oracle/Middleware/wlserver_10.3/common/bin and start ./config.sh
• Don't enable the oracle_common options
• log in as root and copy this domain from the user_project folder to the ndmgr home directory like cp -R /oracle/Middleware/user_projects/domains/wls_domain /home/ndmgr
• Change the rights to the ndmgr user chown ndmgr:oracle /home/ndmgr/wls_domain
• log in as ndmgr and create in the home directory a new file vi nodemanager.domains
• Add the local domain path /home/ndmgr/wls_domain
• Go to the bin folder of your local domain and change in each script the domain variable value to the home directory path.
• Create a boot.properties file for each managed server because else weblogic will ask for a password and this will not work remotely. cd /home/ndmgr/wls_domain  and  mkdir -p servers/Server_1/security Do this for every server
• add username=weblogic and password=weblogic
  

You can test  it when you log in as oracle and try to ask the status of a managed server.
ssh -l ndmgr -o PasswordAuthentication=no  Server_ip_address /oracle/Middleware/wlserver_10.3/common/bin/wlscontrol.sh -d wls_domain -n /home/ndmgr -c -f startManagedWebLogic.sh -s \'Server_1\' STAT

The SSH nodemanager does not have a nodemanager.properties file so you need to change the settings in the oracle/Middleware/wlserver_10.3/common/bin/wlscontrol.sh file. In our case you need to add the network device name and subnet mask.
Interface=${WLS_Interface:-"eth0"}
NetMask=${WLS_NetMask:-"255.255.255.0"}
 
The last part of the SSH node manager is to configure these node manager in the weblogic console.

• go to http://xxxx:7001/console
• Environment / Machines  and select the right servers
• In the Node Manager Tab use the following values
• Type = SSH 
• Listen Address=  your server ip address
• Node manager home = /home/ndmgr
• Shell Command = ssh -l ndmgr -o PasswordAuthentication=no -p %P %H /oracle/Middleware/wlserver_10.3/common/bin/wlscontrol.sh -d %D -n /home/ndmgr -c -f startManagedWebLogic.sh -s %S %C

for more information see the Oracle Weblogic documentation


Java Nodemanager
For you can configure the nodemanager you need to start the node manager once. This will create the nodemanager.properties file.

• cd /oracle/Middleware/wlserver_10.3/server/bin
• ./startNodeManager.sh

Enable sudo for the oracle account

• For Server migration you need to edit  vi /etc/sudoers this allows ifconfig command for the oracle  account
• Add oracle ALL=NOPASSWD: /sbin/ifconfig,/sbin/arping
• Restart ssh cd /etc/init.d and ./sshd stop and ./sshd start

Test adding a new ip address with the weblogic script
cd /oracle/Middleware/wlserver_10.3/common/bin

./wlsifconfig.sh -addif eth0 10.10.10.150 255.255.255.0
./wlsifconfig.sh -removeif eth0 10.10.10.150

For secure communication between your Admin Server and the Java Nodemanamger you need to create some certificates. wls1.local matches with my machine name. Important for hostname verification.

• java utils.CertGen welcome wls1_cert wls1_key domestic wls1.local
• java utils.ImportPrivateKey wls1KeyStore.jks welcome wls1 welcome /oracle/Middleware/base_domain/wls1/certs/wls1_cert.pem /oracle/Middleware/base_domain/wls1/certs/wls1_key.pem

Make a new truststore based on the weblogic truststore

• cp /oracle/Middleware/wlserver_10.3/server/lib/cacerts /oracle/Middleware/base_domain/wls1/certs/wls1TrustKeyStore.jks
• cd /oracle/Middleware/base_domain/wls1/certs/
• keytool -storepasswd -new welcome -keystore wls1TrustKeyStore.jks -storepass changeit
• keytool -import -v -noprompt -trustcacerts -alias clientCACert -file /oracle/Middleware/wlserver_10.3/server/lib/CertGenCA.der -keystore wls1TrustKeyStore.jks -storepass welcome

Configure the nodemanager.properties by adding the network interface / subnet mask and the right certificates

• cd /oracle/Middleware/wlserver_10.3/common/nodemanager
• vi nodemanager.properties
• Add or change these lines 
• NetMask=255.255.255.0
  Interface=eth0
  StartScriptEnabled=true
  StopScriptEnabled=true
• Add the security parameters
• KeyStores=CustomIdentityAndCustomTrust
  CustomIdentityKeyStoreFileName=/oracle/Middleware/base_domain/wls1/certs/wls1KeyStore.jks
  CustomIdentityKeyStorePassPhrase=welcome
  CustomIdentityAlias=wls1
  CustomIdentityPrivateKeyPassPhrase=welcome

you are ready with the nodemanager configuration and can go to the weblogic console application.

• go to http://xxxx:7001/console
• go to environment / machines and select the right machine and go the Node manager Tab
• Change the node manager field
• Type = SSL
• Listen Address = your server ip address
• Listen Port = 5556

Next step is to change the server keystore and ssl properties

• go to environment / servers and select the right server
• Go to the Keystores Tab of the configuration Tab and use these values
• Keystores = Custom Identity and Custom Trust
• Custom Identity Keystore = /oracle/Middleware/base_domain/wls1/certs/wls1KeyStore.jks
• Provide the Custom Identiy Keystore Passphrase
• Custom Trust Keystore = /oracle/Middleware/base_domain/wls1/certs/wls1TrustKeyStore.jks
• Provide the  Custom Trust Keystore Passphrase
• go the SSL Tab
• Private Key Alias = wls1
• Provide Private Key Passphrase 

That's all for the java node manager

Enable Server migration
To enable server migration you need to have a database with a table called ACTIVE. The scripts for all database are located at /oracle/Middleware/wlserver_10.3/server/db. for HA you can choose Oracle RAC or MySQL Cluster ( the community edition , this is very light and free ).

See this chapter how to configure WLS for this leasing table.

After that you can set a virtual ip address on your managed servers and configure the cluster and managed servers for migration. See this Oracle Chapter