Hello guys,

I have a requirement where I need to start storing all the processes information that goes thought the environment, including business data, technical data (AS2 properties, endpoints) the message bodies and message context.

There's a probability that in the future i'll need to marry some of this information with master data, like process names, department, etc... information that doesn't exist in BizTalk. Also track information from custom web services and SSIS processes is in the pipeline.

As such anyone has some experience to share about this topics:

• Storing messages bodies
• Storing the whole message context: there will be some properties that will be track as part of the activity, but i'm referring that besides that, I want to completely store it's context, to make sure that we have all information available.
• Using it with 3rd party .NET based applications, not only BizTalk.
• Linking it's data with another data source, like other database with custom master data.

Thank you

BAM adds a lot of headache in the infrastructure and hits performance. It makes sense if you need to see the aggregated data.

If you just need to audit data, there are many lightweight options.

• Use standard ports with filters, without any binding with orchestrations, to intercept data. It can be SQL of File ports. Huge Pro: You don't have to change existed code.
• Use the tracing utility like log4net or ETW or .... You do have to change your code, orchestrations, pipes... Pro: simple change and support, by adding and managing appenders for any transport.
• Use the NonSQL databases like DocumentDB, or MongoDB, or Redis... or even EventHub. Pro: funny and scalable. BTW log4net has appenders for many NonSQL DB.
• Use BizTalk Tracking Database. Pros: It is much faster than BAM. It is part of core BTS, so there is no additional headache. You don't have to change existed code. Cons: There is no good UI or API to extract tracking data. This part of BizTalk was not finished :) BTW Seems the BizTalk360 team is going to add something.

the following are the options (IMHO, FWIW and My 2 cents)

• Storing Message Bodies - a lot of requirement is there for purposes of audit, IT Rules/Regulation, etc. Use a component e.g.: BizTalk Archiving Component refer https://biztalkarchiving.codeplex.com/ When archiving to a database however remember to get the business to define an data retention and archiving policy. You may want to write a small web-based application over this to permit message retrieval.
• Storing Message context - there is no need really, the message context is promoted from the message and includes any disassembler component. So if you're archiving messages, you can replay them to generate the context properties - in other words you might want to revisit that requirement.
• Archiving to a database can be used by other components.
• Linking data - If you use BAM to store the processing data, workflow stages, execution times, etc. then you BAM (Completes Instances) can become an input into a data mart/warehouse where you can marry this with master data/common codes, etc. and generate more meaning full reports.

So Yes you should use BAM for the purpose of tracking process instances, message/business date, and such. BAM can be incorporated into BizTalk artefacts (pipelines, orchestration, helpers and such). As far as storing AS2 endpoint/properties are required, BizTalk Parties provides a very extensive storage of the same. The same information permits you to control the EDI message processing through pipelines, etc. so I would not recommend creating a parallel store.

Regards.

• Edited by ShankycheilMicrosoft community contributor 16 hours 17 minutes ago edit

the following are the options (IMHO, FWIW and My 2 cents)

• Storing Message Bodies - a lot of requirement is there for purposes of audit, IT Rules/Regulation, etc. Use a component e.g.: BizTalk Archiving Component refer https://biztalkarchiving.codeplex.com/ When archiving to a database however remember to get the business to define an data retention and archiving policy. You may want to write a small web-based application over this to permit message retrieval.
• Storing Message context - there is no need really, the message context is promoted from the message and includes any disassembler component. So if you're archiving messages, you can replay them to generate the context properties - in other words you might want to revisit that requirement.
• Archiving to a database can be used by other components.
• Linking data - If you use BAM to store the processing data, workflow stages, execution times, etc. then you BAM (Completes Instances) can become an input into a data mart/warehouse where you can marry this with master data/common codes, etc. and generate more meaning full reports.

So Yes you should use BAM for the purpose of tracking process instances, message/business date, and such. BAM can be incorporated into BizTalk artefacts (pipelines, orchestration, helpers and such). As far as storing AS2 endpoint/properties are required, BizTalk Parties provides a very extensive storage of the same. The same information permits you to control the EDI message processing through pipelines, etc. so I would not recommend creating a parallel store.

Regards.

• Edited by ShankycheilMicrosoft community contributor Saturday, May 23, 2015 2:47 PM edit

the following are the options (IMHO, FWIW and My 2 cents)

• Storing Message Bodies - a lot of requirement is there for purposes of audit, IT Rules/Regulation, etc. Use a component e.g.: BizTalk Archiving Component refer https://biztalkarchiving.codeplex.com/ When archiving to a database however remember to get the business to define an data retention and archiving policy. You may want to write a small web-based application over this to permit message retrieval.
• Storing Message context - there is no need really, the message context is promoted from the message and includes any disassembler component. So if you're archiving messages, you can replay them to generate the context properties - in other words you might want to revisit that requirement.
• Archiving to a database can be used by other components.
• Linking data - If you use BAM to store the processing data, workflow stages, execution times, etc. then you BAM (Completes Instances) can become an input into a data mart/warehouse where you can marry this with master data/common codes, etc. and generate more meaning full reports.

So Yes you should use BAM for the purpose of tracking process instances, message/business date, and such. BAM can be incorporated into BizTalk artefacts (pipelines, orchestration, helpers and such). As far as storing AS2 endpoint/properties are required, BizTalk Parties provides a very extensive storage of the same. The same information permits you to control the EDI message processing through pipelines, etc. so I would not recommend creating a parallel store.

Regards.

• Edited by ShankycheilMicrosoft community contributor Saturday, May 23, 2015 2:47 PM edit

the following are the options (IMHO, FWIW and My 2 cents)

• Storing Message Bodies - a lot of requirement is there for purposes of audit, IT Rules/Regulation, etc. Use a component e.g.: BizTalk Archiving Component refer https://biztalkarchiving.codeplex.com/ When archiving to a database however remember to get the business to define an data retention and archiving policy. You may want to write a small web-based application over this to permit message retrieval.
• Storing Message context - there is no need really, the message context is promoted from the message and includes any disassembler component. So if you're archiving messages, you can replay them to generate the context properties - in other words you might want to revisit that requirement.
• Archiving to a database can be used by other components.
• Linking data - If you use BAM to store the processing data, workflow stages, execution times, etc. then you BAM (Completes Instances) can become an input into a data mart/warehouse where you can marry this with master data/common codes, etc. and generate more meaning full reports.

So Yes you should use BAM for the purpose of tracking process instances, message/business date, and such. BAM can be incorporated into BizTalk artefacts (pipelines, orchestration, helpers and such). As far as storing AS2 endpoint/properties are required, BizTalk Parties provides a very extensive storage of the same. The same information permits you to control the EDI message processing through pipelines, etc. so I would not recommend creating a parallel store.

Regards.

• Edited by ShankycheilMicrosoft community contributor Saturday, May 23, 2015 2:47 PM edit
• Marked as answer by Ricardo JM 14 hours 50 minutes ago

the following are the options (IMHO, FWIW and My 2 cents)

• Storing Message Bodies - a lot of requirement is there for purposes of audit, IT Rules/Regulation, etc. Use a component e.g.: BizTalk Archiving Component refer https://biztalkarchiving.codeplex.com/ When archiving to a database however remember to get the business to define an data retention and archiving policy. You may want to write a small web-based application over this to permit message retrieval.
• Storing Message context - there is no need really, the message context is promoted from the message and includes any disassembler component. So if you're archiving messages, you can replay them to generate the context properties - in other words you might want to revisit that requirement.
• Archiving to a database can be used by other components.
• Linking data - If you use BAM to store the processing data, workflow stages, execution times, etc. then you BAM (Completes Instances) can become an input into a data mart/warehouse where you can marry this with master data/common codes, etc. and generate more meaning full reports.

So Yes you should use BAM for the purpose of tracking process instances, message/business date, and such. BAM can be incorporated into BizTalk artefacts (pipelines, orchestration, helpers and such). As far as storing AS2 endpoint/properties are required, BizTalk Parties provides a very extensive storage of the same. The same information permits you to control the EDI message processing through pipelines, etc. so I would not recommend creating a parallel store.

Regards.

• Edited by ShankycheilMicrosoft community contributor Saturday, May 23, 2015 2:47 PM edit
• Marked as answer by Ricardo JM Tuesday, June 02, 2015 4:14 PM

the following are the options (IMHO, FWIW and My 2 cents)

• Storing Message Bodies - a lot of requirement is there for purposes of audit, IT Rules/Regulation, etc. Use a component e.g.: BizTalk Archiving Component refer https://biztalkarchiving.codeplex.com/ When archiving to a database however remember to get the business to define an data retention and archiving policy. You may want to write a small web-based application over this to permit message retrieval.
• Storing Message context - there is no need really, the message context is promoted from the message and includes any disassembler component. So if you're archiving messages, you can replay them to generate the context properties - in other words you might want to revisit that requirement.
• Archiving to a database can be used by other components.
• Linking data - If you use BAM to store the processing data, workflow stages, execution times, etc. then you BAM (Completes Instances) can become an input into a data mart/warehouse where you can marry this with master data/common codes, etc. and generate more meaning full reports.

So Yes you should use BAM for the purpose of tracking process instances, message/business date, and such. BAM can be incorporated into BizTalk artefacts (pipelines, orchestration, helpers and such). As far as storing AS2 endpoint/properties are required, BizTalk Parties provides a very extensive storage of the same. The same information permits you to control the EDI message processing through pipelines, etc. so I would not recommend creating a parallel store.

Regards.

• Edited by ShankycheilMicrosoft community contributor Saturday, May 23, 2015 2:47 PM edit
• Marked as answer by Ricardo JM Tuesday, June 02, 2015 4:14 PM

Thanks for the feedback so far...

About storing the message body, although the archive component seems a good option, but assuming that I'm using BAM wouldn't be better to store it using BAM? I know that there is a 1mb limitation, but besides the size limitation, is there advised to go to a custom db for this?

IMHO, FWIW and My 2 cents - Yes (custom DB). This permits you to have a message archiving/retention that is outside of your strategy for BAM/BizTalk databases. Also BAM only permits a string storage and some messages may land up corrupted because of this. Custom DB permits you to explore other options such as BLOB/TEXT and also permits implementation of data security which would not affect or need not be necessarily supported by BizTalk Product.

Regards.

IMHO, FWIW and My 2 cents - Yes (custom DB). This permits you to have a message archiving/retention that is outside of your strategy for BAM/BizTalk databases. Also BAM only permits a string storage and some messages may land up corrupted because of this. Custom DB permits you to explore other options such as BLOB/TEXT and also permits implementation of data security which would not affect or need not be necessarily supported by BizTalk Product.

Regards.

• Marked as answer by Ricardo JM 14 hours 50 minutes ago

IMHO, FWIW and My 2 cents - Yes (custom DB). This permits you to have a message archiving/retention that is outside of your strategy for BAM/BizTalk databases. Also BAM only permits a string storage and some messages may land up corrupted because of this. Custom DB permits you to explore other options such as BLOB/TEXT and also permits implementation of data security which would not affect or need not be necessarily supported by BizTalk Product.

Regards.

• Marked as answer by Ricardo JM Tuesday, June 02, 2015 4:14 PM

IMHO, FWIW and My 2 cents - Yes (custom DB). This permits you to have a message archiving/retention that is outside of your strategy for BAM/BizTalk databases. Also BAM only permits a string storage and some messages may land up corrupted because of this. Custom DB permits you to explore other options such as BLOB/TEXT and also permits implementation of data security which would not affect or need not be necessarily supported by BizTalk Product.

Regards.

• Marked as answer by Ricardo JM Tuesday, June 02, 2015 4:14 PM

Hi Ricardo,

Columns in the BAM tables can be manually altered along with 2 of the stored procedures to remove this limitation of 1 MB (by making them TEXT columns). We have done this previously and we didn't have any issues.

A lot of good points have been mentioned already. But, if I were implementing.. I would check the data that is going into the BAM tables has any use for the business (not for keeping the business running). Can the data inside these BAM tables be mined / aggregated now / future to generate KPIs. If the answer is NO, then you would need to rethink the strategy.

If the main challenge is that of 'logging', then this can be done by various methods which are already listed down by Shanky and Leonid with elaborate pros and cons.

Hi Guys,

Thank a lot for the feedback.