Pages

Friday, November 7, 2008

Split-Join in Oracle Service Bus

One of the cool features of the OSB is Split-Join feature with this you can split up you service request message and process these parts parallel. This part can be processed in an other service. The results or errors of these parts will be later joined together and send back in the response of the proxy service. In this blog I will use a Web Service which has as input Orders and we will split these orders to one order. Because I don't know how many orders I can receive we need to use a Dynamic Split-Join.

This is the xsd I used

I created a WSDL with operation which has a request and a response based on the xsd above

Create a new Split-Join ( You have to do this from the workshop application) where we select the operation of the just created wsdl
The Split-Join automatically creates two variables.

We have to initialize the response variable by adding an assign action where I add the result element to the response variable. Put this Assign action right after Receive
Now we can add a For Each Flow Control which will handle every Order, Put this after the Assign Action.

Select the For Each component where we can define the parallel mode of the order processing and we have to define the counter variable name which start in my case with 1 and ends with the count of the order elements.
Add a new variable order which is based on the order element. We will copy later one order to this variable.
Add a copy action to the Scope window of the For Each component where we use the counter variable to get the right order and pass this value to the order variable. (
$request.parameters/def:Orders/def:Order[$counter] )
Let's change the status of a order by adding a new copy action. Normally you will call an external service which processes the order and returns the result.
Now we collect the result of an order and this to the right place in the responce variable.
A Split-Join can only be called from a business service so let's create one automatically.


This business can be called by a proxy service. Create a proxy service based on the same WSDL as the split-join.
Add some routing to this proxy so the split-join business service is called from this proxy service
Publish this project to the ESB where we can test the proxy service. OSB automatically generates a example orders xml where we will add an extra order.
And voila here is the response where the status element value of the orders are changed to processed.

49 comments:

  1. Hi Biemond I am getting this error when ever I try to run my proxy based on this example

    BEA-380001: [subLanguageExecutionFault] [BEA-2031509] Executing an expression caused an unhandled fault: Variable 'response' is being referenced before it has been initialized with a value.: Fault [{http://docs.oasis-open.org/wsbpel/2.0/process/executable}uninitializedVariable] [BEA-2031659] Variable 'response' is being referenced before it has been initialized with a value.
    Executing Process Reference: NAL-ESB/Common/ProxyServices/SplitJoin1; instance ID: uuid:0ba4a262d036ce8a:67488548:12249b0ad5d:-7fe5; thread: Thread 1 (RUNNING); frame ID: transaction,2; location: {http://www.bea.com/wli/sb/stages/transform/config}assign [37,8] - ENTER.

    This is something that was happening to me when I tried to create a split join using the OSB documentation as well.

    I guess i am not properly using the response.

    can You help

    ReplyDelete
  2. Hi,

    Can you export your project so I can take a look.

    biemond at gmail dot com

    thanks

    ReplyDelete
  3. got it working mate... I was not using proper initialization so that make it throw that error. I now need to see how I can really use it in my real life issue the one you advised using split join for. Will bother you again if I face more issues.

    ReplyDelete
  4. Hi,
    Is there a way to use WSSecurity through a SPlit Join? I´m getting an Error that whe WSDL that I´m importing has Security type that is not supported by the Split Join,Maybe because I can´t see a way to pass the header into the Split Join and WSSecurity goes into the Header, Hope Somebody has solved this issue, Kind Regards

    Gerardo J

    ReplyDelete
  5. Hi,

    are you certain that this is split join problem. Maybe in the wsdl there is not supported ws security porfile

    please check this http://biemond.blogspot.com/2009/06/ws-security-in-osb.html

    and maybe you can use the auth or sign policy

    thanks Edwin

    ReplyDelete
  6. Hi Edwin,
    excellent blog!
    I am trying to accomplish the following:
    A message is de-queued from a JMS queue (XA Transactional) and I want to split into three separate calls to DB Adapter business services, which should all belong to the same XA Transaction.
    I tried this with a "serial" Publish pipeline, and inserts went fine, but did not get rolled back when an error occurred.
    The I tried a split-join, which also works fine with correct data, but behaves weirdly when data has errors: The message stays in the queue with state "receive transaction", and nothing gets inserted, which I guess means the transaction is still open.
    Any suggestions?
    Cheers,
    Morten

    ReplyDelete
  7. Hi Morton,

    the xa transaction problems can be caused because jms is not a jca adapter like aq or db.

    maybe you must try it with aq.

    For the split join error , i don't have a clue.

    thanks Edwin

    ReplyDelete
  8. Hi,
    while using split-join, I got problem with SOAP header. Here is my case:
    Proxy Service -> Split Join -> External Services

    In two cases, testing call from PS and from Split Join in OSB Console, contains full SOAP message (......)
    But when Split Join calls to external services , no Soap-header was included in its request(...)

    The WSDL was configured with advise from http://download.oracle.com/docs/cd/E13159_01/osb/docs10gr3/eclipsehelp/tasks.html#wp1150628, but didn't work.

    Could you please advise in this case? The Soap-header is user-defined and must be transfered to external sevices. Split-Join is a must because of parallel execution.

    Thanks a lot for your help

    ReplyDelete
  9. Hi,

    Did you do this "you must declare the header parts along with the body parts in a single request/response message"

    else can you make an easy very basic test case which I can try out.

    in the about page is my email address

    thanks.

    ReplyDelete
  10. Can we not use xquery inbuilt function like distinct-values etc in in split join (if else condition branching) . If yes then how to use it coz in i see just XPATH 1.0 functions( on string, number , node and boolean ) are avaible

    ReplyDelete
  11. This comment has been removed by the author.

    ReplyDelete
  12. Hi,

    What do you mean, you don't understand the concept of split join.

    this way you can make smaller pieces of software which can be handled and maintained more easily and you can invoke them parallel.

    thanks

    ReplyDelete
  13. Hi Edwin,When testing the project in request How can u get the values of OrderId,Customer as 100

    ReplyDelete
  14. Hi, do you know any way to use an MFL activity, inside the split and join? thanks in advance

    ReplyDelete
  15. Hi, do you know any way to use an MFL activity, inside the split and join? thanks in advance

    ReplyDelete
  16. Quick question : Have BPEL process with FlowN activity, do you think using split and join can improve the performance then FlowN in 11g.

    Thanks.
    Shiv

    ReplyDelete
  17. This comment has been removed by the author.

    ReplyDelete
  18. Hi,

    using split and join can improve the performance then FlowN in 11g.

    It can but it is no garantee, OSB is stateless. less overhead and you should program for performance. Use for example /aa/bb instead of //bb etc. and maybe Streaming can help

    Don't know if your have a process which take minutes and is invoked many times . else stay at BPEL

    thanks

    thanks

    ReplyDelete
  19. Hi,

    For Performance test you can fill up a queue or invoke a proxy with many request.

    you can use soapUI to do a loadtest on the proxy or fill up a queue and look at the result time.

    or use queuebrowser to fill up a queue.

    thanks

    ReplyDelete
  20. Hi Edwin,can you please tell ForEach loop concept in message flow of proxy without split-join.

    Mani

    ReplyDelete
  21. Hi Edwin,
    Thanks for nice article.I follow it and I am also getting the result.
    But when I tried with my requirement like request as schema (request.xsd) it is converted into response (response.xsd) by using xQuery mapping.
    For single transaction (xQuery mapping i.e soure is request.xsd and target as response.xsd) I am able to get the transaction time.
    But for multiple transactions (request and response is same)I am unable to get the transaction time.
    For this I use ForEach loop in message flow of proxy service but I am unable to get the response.
    Can you please help me orr else I can send jar file.

    Hope I expect response from you

    Thanks
    Mani

    ReplyDelete
  22. Hi,

    here you got an example, this works

    Assign <CompanyCollection>
    <company>
    <name>name1</name>
    </company>
    <company>
    <name>name2</name>
    </company>
    </CompanyCollection> to companies

    For Each variable company in ./company of variable companies
    Indexed by variable companyIndex with total count in variable companyCount

    log $companies/company[xs:int($companyIndex)]
    with Annotation COMPANY=

    it does work:

    COMPANY=: <company>
    <name>name1</name>
    </company>

    COMPANY=: <company>
    <name>name2</name>
    </company>

    never forget to cast to an int when using a variable as an index: you never know if OSB is representing it as a string, if it does, you are royally screwed (I let it to your imagination where).

    got it from here

    https://forums.oracle.com/forums/thread.jspa?threadID=982608

    thanks

    ReplyDelete
  23. Hi

    for transactions times you maybe can set some debug flags in the alsb debug xml located in your osb domain root.

    thanks

    ReplyDelete
  24. Hi Edwin, I'm sure you've covered this before, however I can't find a conscise explanation/example on how to propagate headers from the original request from the proxy through the split-join to the end business service. can you kindly help?

    ReplyDelete
    Replies
    1. Hi,

      I think soap headers are ignored, you should retrieve them first and put them somewhere in the body.

      thanks

      Delete
    2. In fact, I read a claim somewhere in the documentation SOAP headers as passed, but it never worked for me. Probably, only the ones defined in WSDL are passed automatically?

      Anyway, I have added a code that passes all headers in my GenericParallel library - a wrapper over Split-Join.

      Delete
  25. Hi Edwin... Thks for the great post.. I was trying to check out the error handling in split -join. I was able to retrieve soap faults but could not catch other faults generated in split-joins. For example i stimulated an error by assigning a input element not expected by xquery template but I could not find the error in the fault variables(catch all block). Tracing was also enabled but the fault details were not available. Could you pls help me here . I think I may be missing something here.

    ReplyDelete
    Replies
    1. Hi,

      Indeed that can be tricky, don't think it is possible.

      Delete
    2. The $fault can be caught in the proxy, i.e. if Split-Join calls a proxy, which in turn can call Biz. Then somehow the system faults got visible in catchAll in Split-Join.

      Delete
  26. Hi Edwin,

    Can you please advise me on below query?

    I have a requirement to implement middleware solution which can handle huge traffic of about 1.5 Lac req/resp coming within 10-15 mins time during peak hours, what is a better option
    1. OSB would be able to handle it considering only few xqueries and few javacallouts are there in message flow.
    2. SOA suite would be a better option.
    3. A combination of both

    Hope to hear from you.

    ReplyDelete
    Replies
    1. Hi,

      Better use OSB cause with SOA Suite you always have some minimal logging level and this leads to data in the infra soa repository. OSB has less overhead and is stateless. When the payload is big then use streaming option and in the xquery use direct xpath queries like /a[1]/name and not /a/name etc.
      Don't know about the performance of java callouts.

      thanks

      Delete
  27. Hi,

    Can you please explain me how different and efficient is Split joins compared to BPEL OSB?
    Correct me if Iam wrong

    ReplyDelete
    Replies
    1. Hi,

      OSB is very efficient cause it is OSB with less overhead then BPEL. OSB Split joins works fine but with Global transaction or transactions it can be very complex when you want to rollback . With BPEL you can do compensation.

      Thanks

      Delete
  28. Hi,
    We have created parallel -splitjoin and created businessService out of splitjoin , when we tried invoking business service from proxy service. But its failing with error message : "Resource not available".
    Can any one please help us what endpoint url we need to mention in businessService..

    Thanks in Advance.
    Ravikiran

    ReplyDelete
    Replies
    1. Hi,

      Do you mean this

      Split-Join can only be called from a business service so let's create one automatically.
      http://3.bp.blogspot.com/_earSixbe3dw/SRR8ADIcnSI/AAAAAAAAB1o/OjGa38aevpQ/s1600-h/osb_splitjoin_13.png

      Thanks

      Delete
  29. Hi Edwin,

    With this split join I am trying to call a web service which is having 50 threads open. I am having 200 orders in my one payload which would execute in parallel , but I want execute to 50 orders in parallel at once. Could you please let me know how to control thread or number request in parallel.My aim is to execute exactly 50 request in parallel . So even if payload is containing 200 orders in 4 times all orders will be processed. Or in this case 200 threads would open at once if I don't specify any configuration.

    Thanks,
    Rishi

    ReplyDelete
  30. Rishi, you have to split the 200 requests into batches of 50 each, and then execute the parallel call for each batch. This is essentially a double-loop inside of Split-Join. You may download the code from http://genericparallel.com/download to see how it is done.

    ReplyDelete
    Replies
    1. Hi ,

      Thanks for your suggestion and its working very fine also. Before doing this I also tried to control threads with work manger and by business service -->operational setting --> (Max size thread, queue size and time to get expire). But those setting doesn't work. I am just wondering theoretically those setting looks good but practically things don't work actually. If possible could you please let me know if any special or different setting is expected to woks thing done with thread control.

      Delete
    2. Throttling queue should work too (given the queue TTL is long enough), but it has an impact on all traffic coming via the Biz, while batching the requests in Split-Join has a fine-grained control over these requests only.

      Delete
  31. Can you direct me to download this demo project for split and Join

    ReplyDelete
  32. I had same requirement as Rishi.I have implemented it successfully.But sometimes I am getting response for first batch only.Sometimes it gives response for all the batches.Weird!!! .Do I need to configure something here to get response all the time.

    ReplyDelete
    Replies
    1. Hi Namrata,

      Does your issue got resloved ?

      Delete
    2. Hi Namrata,

      have you implemented it using the double loop only ? or there is any other way i can implement this ?

      Delete
  33. Hi Biemond,

    it is possible to use a Service Callout inside a Split flow?

    I have a webservice which I want to call it 7 times in parallel, instead of making the calls in sequence as I actually doing.

    Will appreciate a feedback from you.

    Regards,
    Ermal

    ReplyDelete
  34. Hi Biemond,
    We are getting issue with split-join in 11g. We have two managed server in our environment.
    Services using split-join are working as expected in first managed server and failing with below error in second managed server.

    Exception on TransportManagerImpl.sendMessageAsync: unchecked exception:, java.lang.NullPointerException java.lang.NullPointerException at com.bea.alsb.flow.debug.FlowDebuggerEngine.eventManager(FlowDebuggerEngine.java:47) at com.bea.alsb.flow.debug.FlowDebuggerEngine.onFlowStart(FlowDebuggerEngine.java:56) at com.bea.alsb.flow.debug.FlowDebuggerDispatcher.onFlowStart(FlowDebuggerDispatcher.java:125) at com.bea.alsb.flow.debug.FlowDebuggerDispatcher.receive(FlowDebuggerDispatcher.java:59) at com.bea.alsb.flow.transport.RequestResponseMessageDispatcher.dispatchMessage(RequestResponseMessageDispatcher.java:38) at com.bea.alsb.flow.transport.MessageDispatcher.dispatch(MessageDispatcher.java:87) at com.bea.alsb.flow.transport.FlowEndpoint.sendMessageAsync(FlowEndpoint.java:84) at com.bea.alsb.flow.transport.FlowTransportProvider.sendMessageAsync(FlowTransportProvider.java:203) at sun.reflect.GeneratedMethodAccessor452.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at com.bea.wli.sb.transports.Util$1.invoke(Util.java:83) at com.sun.proxy.$Proxy88.sendMessageAsync(Unknown Source) at com.bea.wli.sb.transports.LoadBalanceFailoverListener.sendMessageAsync(LoadBalanceFailoverListener.java:148) at com.bea.wli.sb.transports.LoadBalanceFailoverListener.sendMessageToServiceAsync(LoadBalanceFailoverListener.java:510) at com.bea.wli.sb.transports.LoadBalanceFailoverListener.sendMessageToService(LoadBalanceFailoverListener.java:445) at com.bea.wli.sb.transports.TransportManagerImpl.sendMessageToService(TransportManagerImpl.java:582) at com.bea.wli.sb.transports.TransportManagerImpl.sendMessageAsync(TransportManagerImpl.java:449) at com.bea.wli.sb.pipeline.PipelineContextImpl.doDispatch(PipelineContextImpl.java:691) at com.bea.wli.sb.pipeline.PipelineContextImpl.dispatch(PipelineContextImpl.java:594) at stages.routing.runtime.RouteRuntimeStep.processMessage(RouteRuntimeStep.java:128) at com.bea.wli.sb.stages.StageMetadataImpl$WrapperRuntimeStep.processMessage(StageMetadataImpl.java:346) at com.bea.wli.sb.pipeline.RouteNode.doRequest(RouteNode.java:106) at com.bea.wli.sb.pipeline.Node.processMessage(Node.java:67) at com.bea.wli.sb.pipeline.PipelineContextImpl.execute(PipelineContextImpl.java:1095) at com.bea.wli.sb.pipeline.Router.processMessage(Router.java:214) at com.bea.wli.sb.pipeline.MessageProcessor.processRequest(MessageProcessor.java:101) at com.bea.wli.sb.pipeline.RouterManager$1.run(RouterManager.java:597) at com.bea.wli.sb.pipeline.RouterManager$1.run(RouterManager.java:595) at

    ReplyDelete
  35. HI Biemond stuck since a week
    Need your Help
    i have an input XMl as
    123
    a
    data 1 /data
    /a
    a
    data 2 /data
    /a
    a
    data3/data
    /a
    /123
    tellme how to use the Copy node in the for each scope

    i have only request and response in my schema

    my $Order variable should point to request or response

    i can see here it is pointing to Order/status, in my case if the data is the repetitive tag
    1. what will be my type for the $order variable to create and
    2. what should be the Query

    ReplyDelete
  36. Hi Edwin,
    I have a requirement to process an incoming message and publish it to different destinations which includes web services and MQ after transforming the message according to the destination.
    I believe this could be achieved with split-join in OSB.
    I am not very well versed with OSB, thus looking for guidance on this.

    Regards,
    Ritesh

    ReplyDelete
  37. Interemitely my splitjoin consolidated results is getting duplicated...say if I process 5 threads..consolidated result set some times gives me results like 11345,12335.But most of the times it works fine
    Anyone faced similar issue with dynamic split join on OSB 11g?

    ReplyDelete