Hi,

I have a large XML file retrieved via a ReST service once in a day. The data contained in the XML file needs to be referenced/ used throughout the day by an Orchestration.

How do I cache the data from the XML file, so that I don't need to call the ReST service again for all the Orchestration instances.

Regards,

Sharmishtha

If its a very large data then after you have downloaded it once, save it on ur disk and use the same from the location for all ur orch which would be using it. You can code the process in a streaming way.

I would not recommend to save this in memory for a day long recurrive process if the data is huge (Depends on ur hardware as well).

You can go with SQL save as well but based on the info u provided I can not weigh this option.

Hi Sharmishtha,

You can use any of the existing .NET caching patterns in your orchestration. For example, you can choose using a Distributed Cache Pattern or use a local cache instead. An solution  easy to implement could be using a In-Memory cache with the MemoryCache. 

https://msdn.microsoft.com/en-us/library/system.runtime.caching.memorycache%28v=vs.110%29.aspx?f=255&MSPPError=-2147217396

Regards

Hi Sharmishtha,

I would suggest to go with approach by Pi_xel_xar. If transaction is going with large XML file then save all the data on single location and once all data downloaded then process through BizTalk. 

Though you can look below useful article and thread that will help you to achieve your scenario,

Below article explores the pros and cons, implications, and common pitfalls of caching in BizTalk Server orchestrations.

Considerations for caching in BizTalk

Related thread,

Biztalk caching mechanisms

IMHO the better way would be to create a separate service set

1. Create a service which will download the large file and save it - something like an ATOM feed, you can design a service to publish updates into a database or other random accessible storage (yes XML is also random accessible but using the XML DOM would cause you to load up the whole XML into memory)
2. Create a set of services (exposed as WCF/ASMX) which you would then call through your orchestration as and when required.

by delinking these two and creating an operational control around the download, you would be better enabled to tackle the issue.

Regards.

IMHO the better way would be to create a separate service set

1. Create a service which will download the large file and save it - something like an ATOM feed, you can design a service to publish updates into a database or other random accessible storage (yes XML is also random accessible but using the XML DOM would cause you to load up the whole XML into memory)
2. Create a set of services (exposed as WCF/ASMX) which you would then call through your orchestration as and when required.

by delinking these two and creating an operational control around the download, you would be better enabled to tackle the issue.

Regards.

• Marked as answer by Sharmishtha 22 hours 36 minutes ago

IMHO the better way would be to create a separate service set

1. Create a service which will download the large file and save it - something like an ATOM feed, you can design a service to publish updates into a database or other random accessible storage (yes XML is also random accessible but using the XML DOM would cause you to load up the whole XML into memory)
2. Create a set of services (exposed as WCF/ASMX) which you would then call through your orchestration as and when required.

by delinking these two and creating an operational control around the download, you would be better enabled to tackle the issue.

Regards.

• Marked as answer by Sharmishtha Monday, August 31, 2015 8:29 AM

IMHO the better way would be to create a separate service set

1. Create a service which will download the large file and save it - something like an ATOM feed, you can design a service to publish updates into a database or other random accessible storage (yes XML is also random accessible but using the XML DOM would cause you to load up the whole XML into memory)
2. Create a set of services (exposed as WCF/ASMX) which you would then call through your orchestration as and when required.

by delinking these two and creating an operational control around the download, you would be better enabled to tackle the issue.

Regards.

• Marked as answer by Sharmishtha Monday, August 31, 2015 8:29 AM

IMHO the better way would be to create a separate service set

1. Create a service which will download the large file and save it - something like an ATOM feed, you can design a service to publish updates into a database or other random accessible storage (yes XML is also random accessible but using the XML DOM would cause you to load up the whole XML into memory)
2. Create a set of services (exposed as WCF/ASMX) which you would then call through your orchestration as and when required.

by delinking these two and creating an operational control around the download, you would be better enabled to tackle the issue.

Regards.

• Marked as answer by Sharmishtha Monday, August 31, 2015 8:29 AM

I would try to keep it as simple as possible.

1. Download the data daily, with the Scheduled Task Adapter of similar technique.
2. Load the data into a SQL Database your app owns.
3. Create Stored Procedures to access the data.
4. Write you app against that database.

I've done exactly this many times.  It's easy, understandable, and works.  Don't worry about 'performance'.

• Proposed as answer by Abhishek0127[Abhishek kumar]MVP 19 hours 51 minutes ago

I would try to keep it as simple as possible.

1. Download the data daily, with the Scheduled Task Adapter of similar technique.
2. Load the data into a SQL Database your app owns.
3. Create Stored Procedures to access the data.
4. Write you app against that database.

I've done exactly this many times.  It's easy, understandable, and works.  Don't worry about 'performance'.

• Proposed as answer by Abhishek0127[Abhishek kumar]MVP Monday, August 31, 2015 11:15 AM

I would try to keep it as simple as possible.

1. Download the data daily, with the Scheduled Task Adapter of similar technique.
2. Load the data into a SQL Database your app owns.
3. Create Stored Procedures to access the data.
4. Write you app against that database.

I've done exactly this many times.  It's easy, understandable, and works.  Don't worry about 'performance'.

• Proposed as answer by Abhishek0127[Abhishek kumar]MVP Monday, August 31, 2015 11:15 AM

I would try to keep it as simple as possible.

1. Download the data daily, with the Scheduled Task Adapter of similar technique.
2. Load the data into a SQL Database your app owns.
3. Create Stored Procedures to access the data.
4. Write you app against that database.

I've done exactly this many times.  It's easy, understandable, and works.  Don't worry about 'performance'.

• Proposed as answer by Abhishek0127[Abhishek kumar]MVP Monday, August 31, 2015 11:15 AM