HI,

Ive a scenario where my Sources system can send multiple records but my destination system(WCF Service) can accept only one record per call.

For example if I my sources system sends 20 records, Ive to make 20 WCF service calls in parallel not one after one and then send the aggregated response back to caller.

 

Please give me some directions and the best possible way to design this kind of system.

 

 

Ive done this much(below steps) so far and stuck while sending the response back to caller

 

Main Orchestration:

Created an envelope schema

De-batched this envelope schema inside the orchestration by executing the receive pipeline

Prompted 2 custom property fields Count of Incoming records and the GUID for the incoming request

Published the de-batched single message into message box by initializing a co-relation set  on  Count of Incoming records and GUID of the original incoming request

 

WCF Service Call Orchestration:

This Orchestration subscribes to the messages published by the main Orchestration and makes the WCF service call , then the service response is published into message box after transforming into the required format

A co-relation set is again initialized on Count of Incoming records and the GUID of the original incoming request while publishing the message to message box

 

 

Aggregator Orchestration:

Ive an active receive shape followed by a listen shape which is subscribing to the messages published by WCF Service Call Orchestration. the Co-relation is initialized on the Active receive shape and its followed on the listens receive shape.

Here Im aggregating the message and sending it to folder now trying to figure out how to pass the aggregated message back to Main Orchestration. If this is not working out Im planning to convert the WCF Service Call Orchestration into Start Orchestration and then use the port parameters to pass the control back to Main orchestration.

 

Please suggest different ways of implementing the same and your help is greatly appreciated

I would suggest you to use the Messaging-Database Aggregator Pattern. As complete requirement is not clear I am trying to give you an idea of how it can be implemented:

-          Store the information that you are promoting in Main Orchestration i.e.  Count of Incoming records and the GUID for the incoming request [I guess response must be having this information] into a table in SQL.

-          Publish the de-batched messages to MsgBox using direct bound port use another send port (subscribing to the response messages) to store the response into SQL.

-          Now trick is to implement the Store procedure which would determine if all responses have been received and if yes then returned them into a single batch on next polling.

The main advantage of this implementation is that you would be having only single orchestration instance for each batch of messages.

Rohit,

Thanks for the response, I've a limitations here, storing the information in the Database is not an option for me:( .However, my source system expects the final response in the same thread(Request/Response manner),so I don't think we can achieve this by polling the response table thru stored proc, correct if I'm wrong, below is the flow, if you have any questions please let me know.

2-Way service              -----Rq-----> De-batch      ----Rq-->   WCF Service Request(One Record per request/call)
(Multiple records)        <----Rs------- Aggregate    <---Rs---   WCF Service Response

You have already mentioned the other alternative to do it i.e. by using the Start Orchestration shape and passing self-correlating port as parameter but in this case if you receive 1000 message in batch then 1000 orchestration instances would be created and this alternative is not scalable so I would recommend Database aggregator.

I would recommend asynchronous request-response here because processing batch might take longer time but if needed then response to request-response port can be sent in database aggregator. All you need is to preserve some properties from the message received by 2 way receive port and then copy these to the context of the response message. You can see the detail in this very well explained blog of Paolo Salvatori

Thanks for the inputs Rohith, I went through the link and also through this http://blogs.msdn.com/b/appfabriccat/archive/2010/09/29/biztalk-patterns-part-1.aspx

I've a small correction here my destination system is not a WCF service, it's a REST service for which I've to pass individual record/request as a query string. And there are couple of things which I forgot to mention before.
-We have to implement some timer concept i.e source system sends the "waiting time" for each request they make, if we are not able complete processing within specified waiting time, We've to send an Acknowledgment and then continue processing, later post the aggregated final response to call back end point or to some message queue.

-We also want to control the number of orchestrations(REST Service callers) as you mentioned " if you receive 1000 message in batch then 1000 orchestration instances would be created and this alternative is not scalable", how do we design this system keeping all these things in mind:). In my case max 50 records can come in per request!

-Last but not least the data we get is very sensitive/confidential and we will never be allowed to store it in DB.

I know my story is bit long and thanks for reading!

Just check the Ordered Delivery option on the send WCF port.  That's it.

Your complicating factor is the statement "20 WCF service calls in parallel not one after one".

I've dealt with many similar scenarios, incoming 'batch' to 'single operation' where the caller expected a response.  However, I've never had and 'in parallel' requirement.  Normally, the easy solution is to simple loop in the Main Orchestration, aggregate the responses and send one back.

Assuming there is a parallel requirement, I would vote for the Start Orchestration with a Self-Correlating Port.  That way, you don't have to mess around with correlating back yourself.  The calls will run mostly parallel, but not isolated from other calls, in case that makes a difference.  You'll also have only two Orchestrations to deal with and a single Main instance per incoming call.

Once caveat though.  With the parallel requirement, you risk flooding the target system.  The common solution to that is checking Ordered Delivery, but that would then be the same as looping in the Main orchestration.

Thanks for your response, I was out of site for a while so couldn't respond, I'm working on this and will post the update here ASAP.

All:

I've updated apprapch, please check.

Main Orchestration (Generic De-batcher) Part A:

Created an envelope schema with Any Element as the repeatable record (lets name this message as Message X), this is exposed as WCF Service.

De-batched this envelope schema inside the orchestration by executing the receive pipeline

Initialized 2 custom property fields Count of Incoming records and the GUID for the incoming request, these two properties will be written into message context as not promoted.

Published the de-batched single xml document message(Lets say Message A) into message box by initializing a Correlation set on  BTS.MessageType (Message Type will not be promoted since we are publishing XMl document so the Correlation set is initialized), here I can control the number of worker Orchestrations

Worker Orchestration (Application specific):

This Orchestration subscribes to the Message A based on the message type and does its work/processing and then it publishes a message into message box after transforming into the required format (Lets say Message E)

A Correlation set is initialized here on Count of Incoming records and the GUID of the original incoming request while publishing the message to message box, now these two properties will be written into message context as promoted.

Worker Orchestration always  publishes Message E in case of exceptions, faults, timeouts etc.

Aggregator Orchestration(Generic Aggregator):

This is an aggregator orchestration which is listening to Message E always

Here Ive an active receive shape followed by a listen shape which is subscribing to the Message E. the Correlation set is initialized on the Active receive shape and its followed on the listens receive shape(Co-relation set has Count of Incoming records and the GUID properties in it)

Based on the (Count of Incoming records+1 ) (will tell you why in the later section), Im aggregating all the messages, simple summation of messages ( lets say Message Y)

The aggregated Message Y is published to Message box by initializing a different co-relation set with same Correlation properties

       Here comes the interesting part, now read the Main Orchestration (Generic De-batcher) Part B

      

      Main Orchestration (Generic De-batcher) Part B:

      After publishing de-batched single messages (Message A) to message box, Ive to make this Generic de-      batcher orchestration to wait for Aggregator to complete his job and then collect the Message Y              from  Aggregator to send it back to caller/source system

      So to do this, Im publishing a dummy message of type Message E to message box by initializing the Correlation set on Count of Incoming records and the GUID of the original incoming request, this will be picked up by Aggregator and its designed to aggregate one extra Message E

      Then Ive a receive shape listening for Message Y from message box and its following the Correlation set initialized above.

      Aggregator publishes Message Y to message box whenever its done aggregating all the messages, this Message Y will be picked up by main Orchestration, the dummy message is then removed from it (Message Y) and then the final response is sent to caller

We have many application specific worker orchestrations to be developed in the future, were looking for kind of Generic De-batcher and Aggregator, any thoughts ?