Pages

Saturday, November 13, 2010

Global Transactions and Quality of Service in OSB

Last week I had a great OSB BlackBelt training of Samrat Ray and organized by the SOA Community  In this training we talked a lot about Global Transactions ( XA), QoS ( Quality of Service )  in combination with Transports and JCA Adapters. Because this is so important ( you dont want to lose a Message ) I decide to make a blog about this subject and I will explain the different scenarios options you have in the Oracle Service Bus.

Before I can start I need to create and configure some Queues. In this blogpost I will use JMS for the Proxy and the Business service but you can also use EJB, MQ or the Direct Binding ( SOA Suite 11g ) for the Business Service, they all support XA, HTTP transport does not support XA ).

First create the RequestQ Queue combined with an ErrorQ Queue ( Now failed messages will be redirected to this Queue and won't be lost in Space), a ReponseQ Queue and a DestinationQ Queue ( used in the Business service )

Here is a picture how you can define an Error Queue on the RequestQ.  Go to the Queue, select the Delivery Failure Tab in the Configuration Tab. Now you can set the Redirect Expiration Policy and select the Error destinations.


I start with a simple scenario.  Add a message in the RequestQ and we will expect the message in the ResponseQ.

Add a Proxy Service ( any XML) and select JMS as protocol. OSB generates a default url with an internal XA connection factory and add the Queue name to this URL.

Add the destination Queue in the Proxy Service by enabling the Is Response Required option. Use text as Response Message Type ( else it will be hard to read the message ) and provide the Response URI.

Leave the Message Flow empty and a message in the RequestQ ( Go to the Queue, monitoring tab and a create a new message).  Always change the Redelivery Limit  ( Don't use the value -1 )
You will this message in the ResponseQ.

So far so good, now let's raise an Error on the Response Channel and take a look at the results. In the message flow I added a PipelinePair with a stage and an Error activity.
What you will see the message is removed from the RequestQ and the ResponseQ only gets a message with a SERVER_ERROR header and the content is gone.
We can solve this by making a change in the Proxy Service. Enable the Same Transaction For Response option, the response Thread will take over the transaction so when now something happens the transaction will be roll backed.



Add a message to the RequestQ, ( set the Redelivery to 2 else you will get a loop ) what you will see that the message now is added to the ErrorQ and not to the ResponseQ. This is good behaviour. Now you can re-process this message again by moving it to the RequestQ or let an ErrorHospital handle it.

The next step is adding a business service ( also use JMS and a XA Connection Factory )  and make this part of the Global Transaction. Add a routing to this Business Service in the message flow of the Proxy Service. This will also add the message to the DestinationQ. Make sure  that the Message Type = text ( for this test only )
When you now put a message on the RequestQ, the message is also roll backed on the DestinationQ and only exists in the Error Queue.
When you disable the option Same Transaction for Response ( Default behavior ) then this message will be removed even when there is an error in the Response Channel of the Proxy Service.

The last part of this blog is taking a look at Quality of Service ( QoS ). QoS requires a transaction ( The Proxy Service need to start it ) and you can control, if the Business Service will take part on the Global Transaction. You need to add a Routing options activity to the request part of the Routing Node. First set QoS on Best Effort. Exactly Once is the default value when you use JMS and XA in the Proxy Service.

When you add a message on the RequestQ with a retry delivery of 5, You will see 6 messages in the DestinationQ. With Best Effort you will force the Business Service not to use the Global Transaction. Don't think you will like this,so let's change this to Exactly Once or remove this Routing Options action.



Conclusion: You need to think about your process and ask yourself is it a problem when your message is lost and what about when you deliver the same message again and again. Be aware one transaction for the request and response in the Proxy Service is not enabled by default and you need to configure it and test all the situations. Off course you need to use XA connection factories, Datasources and XA configured Resource Adapters.
HTTP Proxy or Business Services won't be part of XA. When you call a BPEL service you can use the Direct Binding transport for this.

33 comments:

  1. Hello Biemond,

    Excuse me for the spam...

    I have a written a stateless session bean, which is compiled with out any errors, and is deployed on the weblogic server with out issues. But when Ii run my client program, i get this exceptions.

    I need some input on the issue, I am faced with Severe deadlines:


    Here is the full stack trace:

    Caught an unexpected exception!
    java.rmi.RemoteException: EJB Exception: ; nested exception is:
    java.lang.NoClassDefFoundError: Could not initialize class sample.EJB.check.GenericServicesEJB_fd1cts_Impl
    at weblogic.rjvm.ResponseImpl.unmarshalReturn(ResponseImpl.java:234)
    at weblogic.rmi.cluster.ClusterableRemoteRef.invoke(ClusterableRemoteRef.java:348)
    at weblogic.rmi.cluster.ClusterableRemoteRef.invoke(ClusterableRemoteRef.java:259)
    at sample.EJB.check.GenericServicesEJB_fd1cts_EOImpl_1033_WLStub.getGrantsUpdatedOrgData(Unknown Source)
    at sample.EJB.check.GenericServicesClient.main(GenericServicesClient.java:47)
    Caused by: java.lang.NoClassDefFoundError: Could not initialize class sample.EJB.check.GenericServicesEJB_fd1cts_Impl
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
    at java.lang.Class.newInstance0(Class.java:355)
    at java.lang.Class.newInstance(Class.java:308)
    at weblogic.ejb.container.manager.BaseEJBManager.createNewBeanInstance(BaseEJBManager.java:222)
    at weblogic.ejb.container.manager.BaseEJBManager.allocateBean(BaseEJBManager.java:231)
    at weblogic.ejb.container.manager.StatelessManager.createBean(StatelessManager.java:303)
    at weblogic.ejb.container.pool.StatelessSessionPool.createBean(StatelessSessionPool.java:201)
    at weblogic.ejb.container.pool.StatelessSessionPool.getBean(StatelessSessionPool.java:127)
    at weblogic.ejb.container.manager.StatelessManager.preInvoke(StatelessManager.java:148)
    at weblogic.ejb.container.internal.BaseRemoteObject.preInvoke(BaseRemoteObject.java:230)
    at weblogic.ejb.container.internal.StatelessRemoteObject.__WL_preInvoke(StatelessRemoteObject.java:41)
    at sample.EJB.check.GenericServicesEJB_fd1cts_EOImpl.getGrantsUpdatedOrgData(GenericServicesEJB_fd1cts_EOImpl.java:1985)
    at sample.EJB.check.GenericServicesEJB_fd1cts_EOImpl_WLSkel.invoke(Unknown Source)
    at weblogic.rmi.internal.BasicServerRef.invoke(BasicServerRef.java:589)
    at weblogic.rmi.cluster.ClusterableServerRef.invoke(ClusterableServerRef.java:230)
    at weblogic.rmi.internal.BasicServerRef$1.run(BasicServerRef.java:477)
    at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:363)
    at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:147)
    at weblogic.rmi.internal.BasicServerRef.handleRequest(BasicServerRef.java:473)
    at weblogic.rmi.internal.wls.WLSExecuteRequest.run(WLSExecuteRequest.java:118)
    at weblogic.work.ExecuteThread.execute(ExecuteThread.java:201)
    at weblogic.work.ExecuteThread.run(ExecuteThread.java:173)

    I do understand that a NoClassDefFoundError occurs, when a class is present in the classloader, but there was some problem loading the class. I am not able to figure out why I am getting an error on GenericServicesEJB_fd1cts_Impl which is present in the EJB ear file. I have started the whole project in new work spaces and installed on new servers as well, but nothing seemed to help in resolving the issue.

    I have been struggling with this error for quite sometime now and wanted some external help in figuring out why the issue is happening and what can be done to trouble shoot the issue properly.

    Any insights or advice will be greatly appreciated.

    Regards,

    ReplyDelete
  2. Bikram Bhusan SinhaOctober 6, 2011 at 5:26 AM

    Hi Biemond,

    Can you tell me how to handle this scenario ?

    1. OSB Proxy from AQ Topic
    2. Message transformation and sending multiple messages to another AQ through business services
    3. Now one of the messages failed to publish to the AQ business service

    In this case, can we rollback all the previous messages sent to the AQ business service ?

    ReplyDelete
  3. Hi,

    that is possible.

    Do you use a split join / for each or something.

    Important use xa jca adapters. In aq you should use a XA datasource and use transacted and set the QoS to exactly once and use same transaction for the response.

    thanks

    ReplyDelete
  4. Trying to understand the difference b/w "Transaction Required" and "XA Enabled". For example, when would you have XA enabled but not have "Transaction Required"?

    ReplyDelete
    Replies
    1. Hi,

      Transaction required , means the proxy should already have a open transaction , so it can join this transaction. Else it will start a new one.
      Thanks

      Delete
  5. Hi Edwin, can you help me out with one of the scenario.
    I receive a JMS message from an ERP system (say JMS Queue in ERPQueue on WLS. that has redelivery limit 3 and retry interval 3 minutes. it moves message to error queue as discard policy). I have a proxy service in OSB to consume messages.

    Now I need to redirect this message to 3 different system. For me First 2 systems are JMS queues. while 3rd system is a "One way" BPEL process that connects to OSB with direct binding. So, I have created 3 Business Services being called in a sequence from ERPQueue Proxy. (note sequence is not mandatory. I can use split join also).

    problem is with BPEL process. It communicates with other external systems. and if any communication fails it throws "Rollback" fault. I want to do retries from OSB proxy but only for BPEL process not the other 2 system already containing messages.
    however it doesn't work for me. is there anyway to keep retry configuration in OSB rather than at ERPQueue?

    Thanks for wonderful posts btw.

    ReplyDelete
    Replies
    1. Hi,

      can you change for the first 2 the quality of service property to not exactly once, and not take part in the xa transaction. i think you can do this in a routing options

      thanks

      Delete
    2. or publish to a topic and have multiple consuming proxies

      Delete
  6. Hi Edwin, I tried following approach with routing options -

    1. Proxy (XA Transaction - NO)
    Routing options QOS: first two BS (Best Effort). 3rd (BPEL) BS (Exactly once. retry 2 at 5 seconds).
    result - no message to first 2 queues. BPEL access retried as configured on ERPQueue.

    2. Proxy (XA Transaction - Yes)
    Routing options QOS: same as above
    Result : same as above

    3. Proxy (XA Transaction - Yes)
    Routing options QOS: same as above for first 2 queue BS. for BPEL BS "Best effort"
    Result : I get "9" messages in first 2 queues. (Not sure why its 9). BPEL retired only twice as per QOS. ERPQueue retry configuration didn't work. message removed from ERPQueue.

    4. Proxy (XA Transaction - No)
    Routing options QOS : same as # 3
    Result : No messages in first 2 queues. BPEL retired twice as per QOS. ERPQueue retry configuration didn't work. message removed from ERPQueue.

    So based on test #3 and #4 I guess if I set BPEL BS QOS to Best effort I can override retry attempt and interval. That is what I want. but still not sure how to get message in first 2 queues. Any clue on this ? Also when should we "Best Effort" QOS. can I end up receiving message more than once if I set QOS to "Best Effort" ?

    In another option are you suggesting : I can send message to a Topic using current proxy of ERPQueue. then create 3 proxy for each system as subscriber and then again redirect to relevant BS of each system ?

    ReplyDelete
    Replies
    1. With a topic , you can make some simple proxies which are easier to maintain and 2 or more service callouts or publish actions.

      thanks

      Delete
  7. Hi Edwin,
    Is there anyway to use one proxy consuming message from different queues available on same server. Reason I need this is because I am receiving messages from a ERP queue. this ERP system has multiple test environments and there is a JMS queue for each environment.
    but for me I just have one OSB test environment. I create foreign queue on weblogic for each JMS queue of ERP.
    I am planning to use same proxy to consume all messages and later based on some value in content redirect to right resource. is it feasible? or will I have to setup OSB for each ERP environment? Eagerly waiting for your response.

    ReplyDelete
    Replies
    1. Hi,

      Not possible you can't change endpoint on the consuming side. Just create more jms consuming OSB proxies and deliver it all to the same Proxy service.

      thanks

      Delete
  8. Hi Edwin,
    We met a stuck thread issue where it is being caused due to Synchronous receiver which is a OSB proxy in customer's environment.
    Suggested is to turn off "Same Transaction for Response" by Support Team.
    Our concern is to lose the message.
    So we are wondering if use XA connection factories, Datasources and XA configured Resource Adapters and turn off Same Transaction for Response, could still ensure that no message won't be lost?
    Thanks a lot in advance.
    Cheers,

    ReplyDelete
    Replies
    1. Hi,

      you can loose return message ( unless you can handle the error ) and no rollback on the consumer proxy. When something goes wrong on the way back the consuming message and BS is not rollback and the business service commits.

      with xa you can rollback everything and make sure the BS can get the message only once.

      thanks

      Delete
  9. I'm trying to test EJB3 transaction rollback in OSB. I created an EJB3 session bean with two methods persisting data in two different database tables. Following btw one of your posts-) For example the first method creates a region and the second one a country from that region in the Oracle XE HR schema. The session bean is marked @TransactionAttribute(TransactionAttributeType.REQUIRED).

    Of course it would be easier to use JPA with a 1-many relationship and save both records in one persist operation. This is just to test TX rollback.

    Suppose that if country name equals "bad", the persistCountry method throws a checked exception with @ApplicationException (rollback=true). The EJB method rollback itself works fine. Now I created an OSB project with a business service generated from the EJB client. The EJB transport "Supports TX" option is checked. Then I added a proxy service calling my two methods sequentially using two service callouts (not routing). I checked "TX required" on the proxy service message handling config page so I suppose both EJB methods should be invoked within a global TX. What happens though is that if country name="bad" a new region record (created by the 1st EJB callout) does not get deleted.

    Sorry for a long question but I think the scenario is pretty straightforward.

    ReplyDelete
    Replies
    1. Hi,

      What did you use for the invoke of the second ejb method. did you do use a publish or a service callout. These are default in an other transaction ( you can set QoS , in transport or routing options ) and also what is your starting point, do you use JMS as proxy transport( this enables XA )

      Thanks

      Delete
    2. Hi,

      My proxy service is a SOAP web service created from a WSDL defining its input parameters such as are region name, country name etc. The biz service is defined from the EJB client, as you described in one of your previous posts-) In the flow I use two service callouts to call my EJB methods. The second one throws an exception but the first one does not roll back.

      Thanks for your help

      Delete
    3. So how can I make sure that both EJB service callouts participate in the global TX started by the proxy srv?

      Delete
    4. Hi,

      Can you add routing options in the service callout and set QoS on exactly once, (default is best effort )
      and enable same transaction for response

      thanks

      Delete
  10. Sorry for the delay, and thanks for the hint, I will try that

    ReplyDelete
  11. Thanks, it works! If you add a Routing Options element after service callout and set QoS to exactly-once, both EJB calls are executed in the same PS transaction context. It's a little strange that there is no check box on the service callout properties tab itself but anyway.

    It caused another problem in my case-) Both EJB classes run within the same global TX but there must two different database TXs. So the second EJB call does not see uncommitted results of the first one. Which are required in my scenario.. But it's a separate problem

    ReplyDelete
  12. Hi Biemond,

    I tried to implement the mentioned case. I marked the JMS connection factory as XA. but still the a is not getting rotlled back to error queue but to ResponseQ.
    I am clueless. please suggest
    Thanks
    Soumitra

    ReplyDelete
    Replies
    1. Hi,

      Did you add a redirect policy on this ResponseQ , here you can provide the queue and set the maximum retries like 2.

      Are you now having a loop.

      Thanks

      Delete
  13. Hi Biemond,

    It worked after I marked "Is XA Required" in Advanced Settings of JMS Transport tab.

    Thanks for the great post ...

    ReplyDelete
  14. Hi Biemond,

    I Would request for your suggestion for this business scenario. Currently we have application design where OSB receives requests from other applications. OSB does service orchestration by forwarding to more than One BPEL process in any OSB Flow. Each BPEL in turn calls multiple Mediators passing on the request. These mediators calls Oracle EBS using APIs through DB Adaptors.

    Now the problem is, many Oracle EBS APIs have Global Temporary tables in them. All these APIs are standard ones provided by 11i or R12.

    Now when OSB initiates Global transaction, it spans across BPEL, Mediators, DB Adaptors and underlying Oracle EBS database (its Oracle 11.2 database). As per the limitations of Oracle database, Global Temporary tables are NOT supported for Global XA transactions as per Oracle database reference. But as per the current design, we need have data integrity for the whole flow. Also Oracle EBS does not do reverse/compensation transactions as per business guidelines.

    Please suggest how to address this issue and any alternative approaches if any for this integration.

    Regards,

    ReplyDelete
    Replies
    1. Hi,

      this is hard, you can skip the EBS api and do it directly ( not really a great plan ) on the EBS tables
      or give up Global XA try to do some compensation in the BPEL. The BPEL gives back a soap fault or a fault response in Async. The OSB flow should call the other invoked BPEL and ask for compensation or start a new BPEL compensation proces.
      or keep the adapters outside the Global XA and do a rollback in case of a error, try to handle the rollback exceptions in the other BPELs and do some compensation.

      thanks Edwin

      Delete
    2. Thanks Edwin for providing the solution. But we have a difficulty here - as per the business decision - we are NOT suppose to undo/delete any data outside the transaction which did the changes. This means either we have to commit or rollback changes in the SAME transaction itself. We can't undo changes through BPEL Compensation.

      Just looking for any solutions with these technical constraints.

      Regards,

      Delete
    3. Hi,

      there is no solution for this unless you develop your own API on EBS which is also not supported.
      Compensation is the only way and business talking about transactions :-). Maybe you need to do some extra monitoring, async and an errorhospital to compensate for not using XA.

      Thanks

      Delete
  15. Hi Edwin,

    I have made a proxy service using MQ transport and depending on the message content I'm routing the message to three different business services. Business services are of MQ, JMS and HTTP respectively. How I can ensure at least once QoS in this case. If At least Once is not possible then Can I at least gurantee message delivery.

    How to check whether the MQ, I'm connecting to supports, XA or not. If the Request MQ doesn't support XA then how to implement the global transaction. Because the message potentially can get lost after consumption if error has been raised in the proxy service message flow.

    Thanks & Regards
    Sagnik Sarkar

    ReplyDelete
    Replies
    1. Hi,

      I don't know MQ but you can check the MQ adapter or the IBM documentation.

      By the way HTTP does not support XA.

      But what can happen is that the OSB response message flow can fail -> OSB does rollback on a request queue and reads again the message ( without redirect you will have a loop ), with Exactly Once and XA it tries also to rollback the BS message, when not using XA and QoS you will get duplicates.

      So when you use XA like jms on the request, you won't loose your message. When you have a sync HTTP request you should return a soap fault and the caller should do a retry.

      Thanks

      Delete
  16. Hi Biemond,
    My requirement is, PS1 polls a database table for status with 'Ready'. markreadvalue='Processing'.-> publish to JMS Q

    PS2 polls the JMS Q and call a API. when the API returns success message, the record in the database tablw will be updated to 'Processed' if there is any error occoured in the services, the table will be updated to 'ERROR'.

    DB adapter proxy is XA.

    We will check the response from the API and use RAISEERROR action to call the error handler.
    Inside the error handler we have added a publish action which will publish the status as error.
    In the end of the error handler, we have added a RWS(Reply with success) to ensure the error is not rethrown.

    The issue is, when we run a load. like for 100 records, few records are getting struck in Processing status.
    please suggest, if i have to set any configuration for this scenario.

    ReplyDelete
  17. Hello All,
    I am using split join with MQ.Suppose I have 2 request then sometimes I am getting response for all 2 request but sometimes I am getting only 1 response.MQ is returning the resposne.But Invoke service is not picking up that response sometimes.

    Any suggession.

    ReplyDelete
  18. Hi,I have a service which is atomic transaction, So when we are invoking atomic transactional service over proxy service, we are getting the error "Transaction context is required", As for same we added policy file of AT in business and proxy service but again we are getting same error.
    please help me

    ReplyDelete