Soa Suite 10.1.3 performance

For a customer I'm busy with a proof of concept where we try to build a new message handling system. This MHS can already handle many and big edifact smime messages in a few seconds. The MHS also transforms the edifact messages to a xml. The second part of the POC is to use Oracle soa suite 10.1.3 especially the ESB part to deliver edifact xml messages to the right backoffice database. The ESB /BPEL needs a little bit tuning because the MHS handles a 500kb DELFOR message in a few seconds and the ESB takes more then a minute. 500kb edifact message leads to a 3.5Mb xml message. The soa suite server is a new Sun server with two intel quad cores with a lot of memory.
First let me explain the esb processes. The esb /bpel process look like this. The esb reads the jms queue and gives the xml to a simple bpel process ( I did this to make sure that first the xml is inserted into the backoffice tables and after this process we can pass the incoming xml to the backoffice queue, so the backoffice database knows which messages are ready to process). This bpel process calls two other esb process. Maybe this can be solved in only one esb process ,let me know if you know a solution.
The first esb process reads the queue and pass the xml to the bpel process.

The bpel process calls the first esb process.( see my previous blog entry for details)

The last step of the bpel process is to call the last esb process which put the xml message in the backoffice queue.

The total process takes about 55 seconds and our goal is to do this in 10 seconds. The first performance step is to replace the jms oracle database queue with a oc4j file persistence memory queue. We have won 10 seconds with this new queue. AQ with big xml's are not so fast as a memory queue. You can also use openjms. This is about 20% faster then OC4J.
Let's look at the bpel process. This is a simple process so I only need to store the faulted instances. We add the following configuration part to the bpel.xml.

<?xml version = '1.0' encoding = 'UTF-8'?>
<BPELSuitcase>
   <BPELProcess id="mhs_jms_esb_handling" src="mhs_jms_esb_handling.bpel">
      <partnerLinkBindings>
    .....
      </partnerLinkBindings>
      <configurations>
        <property name="inMemoryOptimization">true</property>
        <property name="completionPersistLevel">instanceHeader</property>
        <property name="completionPersistPolicy">faulted</property>
        <property name="deliveryPersistPolicy">off</property>
        <property name="auditLevel">off</property>
      </configurations>
   </BPELProcess>
</BPELSuitcase>

We have won an another 8 seconds. So the total processing time is 38 seconds.
Because the db adapter takes about 7 seconds and passing the xml to the database queue takes also 7 seconds. We have a overhead of 24 seconds. This is propably due to by passing the 3.5Mb xml with web services from esb to bpel and back to the esb. The last step I did is to pass only the run identifier to the backoffice queue and not the whole xml messages. Now the total process takes 23 seconds and the AQ part is from 7 seconds reduced to 10ms. The overhead is still 15 seconds. So next week I will try to tweak the esb web services. If you know how to do this let me now.
This are the result with different delfor edifact message size.

If we look at the result table we see that small xml messages are handled very fast and that 3.5Mb or 7Mb xml messages have the same total processing time which is very strange. Maybe soa suite handles big messages differently.
The last problem I had is that the esb handles only one process and waits until this finish before starting the next process, because I have a server with 8 cores I want to have more parallel process else I only need to have two cores with a high clock frequency.
This problem is also solved by making a esb router (which dequeues) and calls with a synchronous routing rule the BPEL process. The BPEL is now a asynchronous process where the first step is to report back to the esb and after this I am calling the other ESB processes. Now I see parallel processing and a 700kb delfor xml is processed in a second, the 3.5 mb delfor in 16 seconds and the 7mb delfor in 12 seconds. I had to remove the deliveryPersistPolicy from the bpel.xml else It won't process the bpel instances.