new Puppet 3 Weblogic provisioning module

The last few weeks I was busy re-writing of my puppet WLS module so it fully supports the power of Puppet 3 (thanks for more than 4000 downloads on puppet forge and all the github downloads).

With Puppet 3 we now can use Hiera, Iterations and Lambdas expression. This does not sound like a big change but with Hiera and the new Puppet Language features, I can define big WebLogic Domains without losing maintainability ( Got one customer with 5 Clusters, 25 Managed + JMS servers, more than 100 queues & topics).

With Puppet version 2 and my WLS module you need to have a lot of files (site specific classes) with at least more than 2000 a 5000 lines, here is an example of this . In this ORAWLS module I use the default/minimal approach ( declare once in Hiera) so you won't need to declare the same parameters over and over again. Off course you can still define everything like you did in the WLS module.

When you want to know more about Hiera you can take a look at this documentation page. For the new Puppet 3 language features look at this slideshare presentation.

In this post I will show you the new features and how it works with Hiera. Here you can download or take a look at the new orawls github code, or download it from puppetlabs forge. I Also made a vagrant reference implementation ( which you can use to test it yourself ),  this reference will create a 3 node WebLogic 10.3.6.0.6 Domain with 2 Clusters together with some JMS modules and Queues.

The first big change is that the puppet install classes does not contain any site specific code, I have now only 2 classes which I can use everywhere

• One for the WebLogic Admin server which contain all the possible WebLogic options
• One for WebLogic nodes which only installs the software, copy domain and configure the nodemanager )

Let's begin with installing the WebLogic software.  For this I only need to add this to a class.

include orawls::weblogic

orawls::weblogic is a class and in Puppet 3 it will look in Hiera to find the parameter values ( it uses the following hiera lookup module_name::class_name:parameter_name )

Here is a snippet of my common.yaml file

# global WebLogic vars
wls_oracle_base_home_dir: &wls_oracle_base_home_dir "/opt/oracle"
wls_weblogic_user:        &wls_weblogic_user        "weblogic"
wls_weblogic_home_dir:    &wls_weblogic_home_dir    "/opt/oracle/middleware11g/wlserver_10.3"
wls_middleware_home_dir:  &wls_middleware_home_dir  "/opt/oracle/middleware11g"
wls_version:              &wls_version              1036

# global OS vars
wls_os_user:              &wls_os_user              "oracle"
wls_os_group:             &wls_os_group             "dba"
wls_download_dir:         &wls_download_dir         "/data/install"
wls_source:               &wls_source               "/vagrant"
wls_jdk_home_dir:         &wls_jdk_home_dir         "/usr/java/jdk1.7.0_45"
wls_log_dir:              &wls_log_dir              "/data/logs"

#WebLogic installation variables 
orawls::weblogic::version:              *wls_version
orawls::weblogic::filename:             "wls1036_generic.jar"
orawls::weblogic::middleware_home_dir:  *wls_middleware_home_dir
orawls::weblogic::log_output:           false

# hiera default anchors
orawls::weblogic::jdk_home_dir:         *wls_jdk_home_dir
orawls::weblogic::oracle_base_home_dir: *wls_oracle_base_home_dir
orawls::weblogic::os_user:              *wls_os_user
orawls::weblogic::os_group:             *wls_os_group
orawls::weblogic::download_dir:         *wls_download_dir
orawls::weblogic::source:               *wls_source

Next step is to add some WebLogic 10.3.6 patches (BSU)
Here is the code in the admin class which does a hiera lookup of bsu_instances and call create_resources for every BSU entry.

$default_params = {}
$bsu_instances = hiera('bsu_instances', [])
create_resources('orawls::bsu',$bsu_instances, $default_params)

and the matching yaml entry

# patches for WebLogic 10.3.6
bsu_instances:
  'BYJ1':
    patch_id:                "BYJ1"
    patch_file:              "p17071663_1036_Generic.zip"
    log_output:              true

  The same mechanism for all installing all the Fusion Middleware software

# FMW installation on top of WebLogic 10.3.6
fmw_installations:
  'osbPS6':
    fmw_product:             "osb"
    fmw_file1:               "ofm_osb_generic_11.1.1.7.0_disk1_1of1.zip"
    log_output:              true
  'soaPS6':
    fmw_product:             "soa"
    fmw_file1:               "ofm_soa_generic_11.1.1.7.0_disk1_1of2.zip"
    fmw_file2:               "ofm_soa_generic_11.1.1.7.0_disk1_2of2.zip"
    log_output:              true

The next step is create a WebLogic domain.

When you only have one domain then you can set the following hiera variables. These entries are also automatically picked up by the other WebLogic classes. With 2 domains or more you need to set the domain parameters everything you call a module function.

logoutput:                     &logoutput                     true

# when you have just one domain on a server
domain_name:                "Wls1036"
domain_adminserver:         "AdminServer"
domain_adminserver_address: "10.10.10.10"
domain_adminserver_port:    7001
domain_nodemanager_port:    5556
domain_wls_password:        "weblogic1"
domain_user_config_file:    "/home/oracle/oracle-Wls1036-WebLogicConfig.properties"
domain_user_key_file:       "/home/oracle/oracle-Wls1036-WebLogicKey.properties"

# create a standard domain
domain_instances:
  'wlsDomain':
     domain_template:          "standard"
     development_mode:         false
     log_output:               *logoutput

For more examples take a look at the readme page of het orawls module or look at the vagrant reference implementation.

The next part is used for bulk insert of WebLogic artifacts and do this with minimal configuration. this feature has some requirements

• puppet version > 3.2 ( make use of iterations and lambda expressions )
• set --parser future ( puppet agent config )

I also use hiera_array instead of hiera, this will search for this entry in all the hiera data files. This way I can separate application or cluster code in its own yaml file and created in one go.

example of how to call this wlstbulk define

$allHieraEntries = hiera_array('jms_module_jms_instances')

orawls::utils::wlstbulk{ 'jms_module_jms_instances':
  entries_array => $allHieraEntries,
}

And the matching hiera entry, this one supports global parameters so you won't need to add these params to every WLST entries.

This saves you a lot of extra configuration especially with many WLST entries and make your definitions much clearer

managed_servers_instances:
   - clusterOne:
      global_parameters:
         log_output:           *logoutput
         weblogic_type:        "server"
         script:               'createServer.py'
         params:
            - "listenAddress    = 8001"
            - "nodeMgrLogDir    = '/data/logs'"
      wlsServer1_node1:
         weblogic_object_name: "wlsServer1"
         params:
            - "javaArguments    = '-XX:PermSize=256m -Xms752m -Xmx752m'"
            - "wlsServerName    = 'wlsServer1'"
            - "machineName      = 'Node1'"
      wlsServer2_node2:
         weblogic_object_name: "wlsServer2"
         params:
            - "javaArguments    = '-XX:PermSize=256m -Xms752m -Xmx752m'"
            - "wlsServerName    = 'wlsServer2'"
            - "machineName      = 'Node2'"

So I hope this module will help you with all the WebLogic provisioning tasks.

Creating your own Virtualbox Development Images

For my Oracle Puppet provisioning development I can't do without these create image tools: Packer and Vagrant in combination with Oracle VirtualBox or VMware.  In this blogpost I will explain what these tools can do for you and how you can make your own images and use puppet as provisioning tool.

With Vagrant you can create your own virtual images and it can start puppet or chef to do all the server provisioning. Besides this, Vagrant can also manage all the network configuration or make a multi machine configuration. One of the great things of Vagrant is when you make a mistake you can start over by destroying the image ( vagrant destroy ) or start the provisioning again ( vagrant provision).

For Vagrant you need to download and import a small image box ( here is  an overview of the all public boxes which you can use right away )  like CentOS or Ubuntu. This image already contains the Virtualbox guest additions, which Vagrant will use to setup the shared folder etc.

Packer is a tool which can create this vagrant box for you. With this you can create a virtual image which also can be used for Amazon EC2, VirtualBox or VMware. In my case I need to have a particular image with the latest VBox guest additions and the latest Puppet RPM's. With packer I am now be able to create the same images as which my customer uses and test it on my own laptop.

How does this work?.  ( you can download a full working example at my Github page which will create a CentOS 6.4, 5.8 or Ubuntu images and this will also add Puppet )

First we need to define a json file and a Red Hat kickstart file.

here is an example of a CentOS 6.4 example.  This file contains the url of CentOS minimal or network iso, the Vagrant setup steps and post installation scripts, like the puppet install.

and the CentOS kickstart file


Next add the packer directory to your path variable , go to the packer github folder and use this command
packer build centos-6.4-x86_64.json

Wait some minutes and you will have a new box. Packer also shrinks this image to 600mb and is now ready to use it for Vagrant.


Next step is to download my vagrant github folder
When you go to this directory and use the vagrant up command, this will download the CentOS box, create a Virtualbox image and start the Puppet provisioning.

here is my example of a vagrant single node configuration, this will forward some ports and starts the Puppet site.pp class.

Here is an overview of my puppet folder

This is a shared folder ( virtual image and your machine ) and you can edit these Hiera and Puppet files in your fav editor, Add the needed puppet modules and start the provisioning again by using this command: vagrant provision or do it all over again by destroying it first: vagrant destroy -> vagrant up

Also vagrant supports multi machine configuration, just look at Vagrantfile_multinode.

Here is an working 3 node example, 3 VirtualBox Servers with a working WebLogic 10.3.6.0.6 Cluster which uses a private network.
https://github.com/biemond/biemond-orawls-vagrant 

Hope these will also help you.

The road ahead for WebLogic 12c

Before we can describe all the new features of WebLogic 12.1.3 & 12.1.4 and compare this to the 12.1.2 version we should first take a look at the 10.3.6 version. WebLogic 10.3.6 is still the latest 11g version but Oracle will support 10.3.6 till 2018 and extended support till 2021. So Oracle’s Fusion Apps and we have enough time to migrate to WebLogic 12.1.X. Oracle also promised that the upgrade should be easy.  That being said we can take look at the WebLogic 12.1.X features.

Last summer Oracle already released WebLogic 12.1.2 which has since WebLogic 12.1.1 been certified for Java EE 6 and it looks like the Java EE 7 certification is still far away, so Oracle updated the 12.1.2 version with some badly needed frameworks like WebSockets. To make the developer experience more complete Oracle added more support for Maven and it comes with a utility to synchronize a Maven repository with all the needed WebLogic libraries.

12.1.2 is also the first release, which comes with Fusion Middleware infrastructure components. For now FMW 12.1.2 contains ADF & OWSM and comes with Enterprise Manager & MDS.
WebLogic 12.1.2 replaced the BEA installer and the BSU patching utilities with the Oracle Universal Installer and the OPatch utilities for applying patches.  To make it even more easier it just comes in one taste ( no Java included or a specific Operating System installer ).  Just one big jar file for WebLogic with Coherence or one for WebLogic, Coherence and FMW.

WebLogic 12.1.2 introduced Dynamic Clusters, Elastic JMS and integrated Coherence configuration and management. Dynamic Clusters is a great feature to extend the WebLogic Cluster with Managed Servers based on a Server Template. For this we only need to change a parameter on the cluster and the new Managed Servers are distributed over the NodeManagers. To use this feature your blades need to have enough free resources to handle the extra Managed Servers.
Elastic JMS can be combined with Dynamic Clusters, this way we only need to create 1 JMS server, target this to the cluster and every Cluster node will have its own JMS server.

When we look at the announced features of WebLogic 12.1.3 and 12.1.4 we can see that Oracle continues on this road.

WebLogic 12.1.3 will be the first version for many FMW 12c products like Oracle SOA Suite 12c and probably come in one big jar. 12.1.3 & 12.1.4 will add extra features and improvements to Elastic JMS & Dynamic Clusters. Elastic JMS in 12.1.3 will support Server Migration so you can’t lose any JMS messages.
In 12.1.4, Dynamic Clusters will have support for auto-scaling based on thresholds based on user-defined metrics. WebLogic 12.1.4 will also have an API to control the Dynamic Clusters, this way we can easily program when to stop, start or remove nodes from a dynamic cluster.

WebLogic 12.1.3 will come with the JDBC 12c driver, this driver will gives us better integration and reliability between WebLogic 12c and Oracle Database 12c (Application Continuity).
When we take a look at the administration side of WebLogic we can see that the Enterprise Manager will be more important, in WebLogic 12.1.3 we can also do Application and JMS administration from the EM application. Also 12.1.3 will contain a Restful management APIs for additional monitoring, operations, Datasource and deployment support.  Plus a global OWSM policy to protect all your Web and Rest services with one server policy.

WebLogic 12.1.4 will have a new feature called Multi-Tenant Applications, this way you can define an WebLogic template for an application, so one or more customers of this application which will have its own Cluster, Managed Servers, Application and (not shared) resources plus it will support Oracle Database 12c pluggable databases.

The last part of this article will talk about the development side of WebLogic 12.1.3 and 12.1.4.  First WebLogic 12.1.3 has to be the Modern Development Platform and will focus on HTML5 development.

12.1.3 has some great new features like
• Updated version of JAX-RS and WebSockets.
• Server-Sent Events, this allows you to continuously sending updates to the browsers.
• JSON-P or JSON with Padding will solve your cross domain security problems.
• Maven improvements.
• Avatar allows you to develop and deploy server side Javascript Services without knowing Java.

WebLogic 12.1.4 will be certified for Java EE 7 and this means the WebLogic platform will get some framework updates and some new frameworks like
• Java API for JSON 1.0
• Java API for WebSocket 1.0
• Batch Applications 1.0
• Concurrency Utilities 1.0

The last question is off course when can we expect WebLogic 12.1.3 & 12.1.4.  The 12.1.3 version will probably be released in March or April and 12.1.4 will be much later.

Custom Jersey WADL generation

I had a situation where the auto generated WADL did not match with my Rest services.
The first difference was that the response is presented as an object instead of a collection of objects and the second one is that it could not handle JSONWithPadding as response.  Because I use this WADL in my Rest client generation, I need to fix these issues.

Lucky for me, Jersey JAX-RS allows us to provide the necessary WADL input files so the generated WADL matches with the Rest services.

To make this happen I did the following steps.

First I extended the WadlGeneratorConfig class in which I defined the following properties applicationDocsStream, grammarsStream,  resourceDocStream. These vars point to the xml files which will be used in the WADL generation.

We need to add the following init param com.sun.jersey.config.property.WadlGeneratorConfig to the Jersey servlet. This init param points to your WadlGeneratorConfig class

The first file is the application-doc.xml ( located in my src folder ),  this will be used for the Title and high level description of your Rest service.

<applicationdocs targetnamespace="http://research.sun.com/wadl/2006/10">
 <doc title="Oracle HR Demo API" xml:lang="en"> This is a paragraph that is added to the start of the generated application.wadl </doc>
</applicationdocs>

Next is the application-grammars.xml  ( located in my src folder )  this file contains the location of your own XML schema which contains all our XML Elements and Complextypes.

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<grammars xmlns="http://wadl.dev.java.net/2009/02"     xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xi="http://www.w3.org/1999/XML/xinclude">
   <include href="../xsd/schema.xsd" />
</grammars>

Add the schema.xsd with the right definitions to the xsd folder located in the web folder of the web application.

The last file is the resourcedoc.xml, this file will be generated by the Maven javadoc plugin.

To generate the content of this file we need to add some javadoc annotations to our Rest Operation methods
For example the response.representation.200.qname annotation points to the employees element (schema.xsd).

Also the response.representation.200.example annotation points to an example object.

Add the Maven JavaDoc plugin with com.sun.jersey.wadl.resourcedoc.ResourceDoclet to your project pom and generate the resourcedoc.xml

Example of a resourcedoc.xml

Startup your Webapp and now the WADL looks like this

Here is my github demo project.

Coherence 12.1.2 Rest application build with OEPE

With WebLogic 12.1.2 Oracle also released a new version of Coherence and OEPE. The 12.1.2 release contains many new Coherence features like WebLogic Managed Coherence Servers and Coherence Grid Archive ( GAR ) which can be included in an normal EAR. Coherence also has some nice new REST features like direct & named queries,  Custom Query engines and new Security options.
Plus with OEPE you can develop Coherence applications in Eclipse and it has Coherence editors for all the Coherence configuration files.

In this blogpost we will test these tools and features in a demo application which uses the HR Oracle demo schema, JPA and expose these entities as Coherence REST Services.

We start by downloading OEPE Eclipse runtime bundle with WebLogic 12.1.2, Coherence and ADF http://www.oracle.com/technetwork/developer-tools/eclipse/downloads/index.html

Start OEPE and define a new workspace.

Create an Oracle Coherence Application

Provide a project name and make sure you define a target runtime.  Plus enable Add project to EAR.

Use the default Coherence options.

The Coherence application will also add a dynamic Web project.

JPA Project
For this demo I will use JPA so we can use these entities in Coherence.

Also target this to WebLogic 12.1.2 and add this project to the already existing EAR project.

Define an connection to the database this will also add a persistence unit to the JPA project.

Next choose JPA entities from tables, where I select the Departments and Employees tables for this demo

Change the JPA mapping relations between the Department and Employee entities and add for REST the XML annotations.
Like @XmlRootElement(name="Department")  and @XmlTransient on the getters to break the loop of loading the Department and Employee objects.

Coherence Project

Next step is to configure the Coherence.
We can use the OEPE Coherence Editors or go directly to the source tab.

First we change coherence-cache-config.xml file where we will define the Department and Employee cache and connect this to EclipseLink.

HrJPA is the name of the Persistence Unit ( Resource Local )

Create a new file called coherence-rest-config.xml, this contains our REST entity definitions where we add some coherence named queries and enable the direct query option.

<key-class>java.lang.Integer</key-class> must match with the primary Java Data type of the entity


The last file we need to change is the pof-config.xml and add <include>coherence-rest-pof-config.xml</include> to the user-type-list

Also upload the coherence-rest.jar to the lib folder of the CoherenceJPA project.

Web project
Last step is to enable the Web project for Coherence REST. We need to enable the Oracle Coherence Facet on this Web Project.

Remove all the Coherence files located in the src folder, we don't need this.

Add the following Coherence REST Servlet.

Also we need to add the following Jersey and Jackson jars files to the WEB-INF lib folder. ( Located  in the module folder of the oracle_common )

Also create your own servlet, so we can fill the Department and Employee Coherence cache ( else the cache will be empty )

In the Eclipse Servers tab we need to add an WebLogic Domain with a Managed Coherence Server ( maybe use right click to select an WebLogic target other than the default AdminServer.

For we Coherence REST we need to change the default EclipseLink JAXB provider.

Add these parameters to Server startup arguments to the Managed Server
-Dcom.sun.xml.ws.spi.db.BindingContextFactory=com.sun.xml.ws.db.glassfish.JAXBRIContextFactory 
-Djavax.xml.bind.JAXBContext=com.sun.xml.bind.v2.ContextFactory

Publish the EAR from OEPE which also contains the Grid Archive (GAR) to the Coherence Managed Server

Finally we can test the Rest service
Start by invoking the servlet http://wls12:7201/CoherenceJPAWeb/CoherenceServlet

Next we can use a Rest Client to test all the Rest operations.

Get all the department entries
http://wls12:7201/CoherenceJPAWeb/rest/Department

Get Deparment 100
http://wls12:7201/CoherenceJPAWeb/rest/Department/100

Add a new Department

 Delete a department

Direct query ( enabled in the coherence rest config xml )
http://wls12:7201/CoherenceJPAWeb/rest/Department?q=departmentName='Finance'

Location1700 Named query also defined in the coherence rest config xml

Location Named query with a integer parameter

Here you can download or look at the github demo project.

JAX-WS SOAP over JMS

With WebLogic 12.1.2 Oracle now also supports JAX-WS SOAP over JMS. Before 12.1.2 we had to use JAX-RPC and without any JDeveloper support. We need to use ANT to generate all the web service code. See this blogpost for all the details.

In this blogpost I will show you all the necessary JDeveloper steps to create a SOAP over JMS JAX-WS Web Service  ( Bottom up approach) and generate a Web Service Proxy client to invoke this service, plus let you know what works and what not.

We start with a simple HelloService class with a sayHello method.

After this we can start the Create Java Web Service wizard.

Select our HelloService class

Select our sayHello method

This will generated the following code.  Besides @WebService , @WebMethod and @WebParam annotation we can change the name attribute of the WebParam annotation and optional add the WebResult annotation.

Next step is to add the @JMSTransportService annotations on this HelloService class.
This will import com.oracle.webservices.api.jms.JMSTransportService class.

When we click on this annotation we modify all the JMS transport properties in the JDeveloper property window

We can also change this Web Service activation properties.

With this as result.

I will use localhost as jndiURL, when you change this to the server url ( you can't deploy it to an other server)  but then you can invoke this JMS service by generating a JAX-WS proxy client without changing anything.

Before we can deploy this Web Service to a WebLogic Domain we need to extend this domain with the WebLogic JAX-WS SOAP/JMS Extension ( or add $WebLogicHome/common/templates/wls/wls_webservice_soapjms.jar template with WLST )

This will create the following Queues and Connection Factory in its own JMS module and JMS Server

We can deploy the Web Service to the JAX-WS WebLogic Domain. 

Also retrieve the WSDL with HTTP.

The WSDL show all our JMS properties in the SOAP Binding and as one string in the soap address location attribute.

Next step is generating a JAX-WS Client and invoke this JMS service. 

Select our Web Service,  from there we can generate a new Web Service client.

Do this in a new project called JmsClient

Enable Copy WSDL into Project.

Leave the default url as it is, even when it is not valid.

Next we can change the PortClient class so it won't use HTTP to invoke this service. 

The documentation says we can do this in 3 ways ( for me only one way really works )

We can JMSTransportClient annotation to our Service class

@WebServiceClient(name = "HelloService",
                  targetNamespace = "http://jms.ws.amis.nl/",
                  wsdlLocation = "http://localhost:7101/...")

@JMSTransportClient (
 targetService = "HelloService",
 destinationName ="com.oracle.webservices.api.jms.RequestQueue",
 replyToName = "com.oracle.webservices.api.jms.ResponseQueue",
 jndiURL ="t3://devagent30:7001",
 jndiInitialContextFactory="com.oracle.webservices.api.jms.ConnectionFactory" ,
 jndiConnectionFactoryName="weblogic.jms.ConnectionFactory" ,
 timeToLive=1000,
 priority=4,
 messageType=JMSMessageType.TEXT)

public class HelloService_Service extends Service {

or as JMSTransportClientFeature in our PortClient class.

JMSTransportClientFeature feature =
  JMSTransportClientFeature.builder()
    .targetService("HelloService")
    .jndiURL("t3://devagent30:7001")
    .destinationName("com.oracle.webservices.api.jms.RequestQueue")
    .replyToName("com.oracle.webservices.api.jms.ResponseQueue")
    .destinationType(JMSDestinationType.QUEUE)
    .messageType(JMSMessageType.TEXT)
    .jndiInitialContextFactory("com.oracle.webservices.api.jms.ConnectionFactory")
    .priority(4)
    .build();

HelloService_Service helloService_Service =
  new HelloService_Service(url,new WebServiceFeature[]{feature});

But the only thing what works for me is changing the SOAP service address location url

like this

HelloService helloService = helloService_Service.getHelloServicePort();

BindingProvider bp = (BindingProvider) helloService;
bp.getRequestContext().put(BindingProvider.ENDPOINT_ADDRESS_PROPERTY,
"jms:jndi:com.oracle.webservices.api.jms.RequestQueue"+
"?targetService=HelloService&"+
"replyToName=com.oracle.webservices.api.jms.ResponseQueue&" +
"jndiURL=t3://devagent30:7001&messageType=TEXT&"+
"deliveryMode=PERSISTENT&priority=4");

System.out.println(helloService.sayHello("hi"));

You can now invoke the JMS Service with proxy client and check the message count on the request and response queue in the WebLogic console.

Here you can look and download my test project.