Hello!

I have a scenario where I need to FTP files from about 250 FTP users approx. using BizTalk Server 2006 R2. From my initial research, I found out various different ways, although I have certain concerns regarding the approaches:

1. Using Orchestrations and configuring Dynamic FTP Send Ports.

2. Using Scheduled Task Adapter and utilizing Orchestration.

My concern is that as I have about 250 users for which I need to keep the pipelines open or at least if I can schedule the transfer for them every half an hour. The aim is to have them grouped in order to keep minimal connections to the FTP server as this FTP already gets lot of pings from lot of other servers as well.   

Can anyone please help me out here to understand the best possible way to FTP though for about 250 users.

Thanks!

Can you clarify.  Are you Sending to 250 targets or Receiving from 250?

You can leverage Orchestration with Dynamic send port for the FTP location .

This way you have minimal number of static location which need to be created .

Please let us know what more information you require .

Thanks

Abhishek

I will be receiving from about 250 FTP users everyone with different authentication.

I'm just curious here and want to understand that:

1. Will these ports need to be open all the time to take files from FTP (i.e. around 250 connections open at all the times) or do I need to instruct my Orchestration to have a schedule transfer from these 250 FTP accounts(all with separate credentials).

2. Also would this require to have setup for about 250 separate Dynamic Send Ports for every FTP Account?

Thanks!

I'm just curious here and want to understand that:

1. Will these ports need to be open all the time to take files from FTP (i.e. around 250 connections open at all the times) or do I need to instruct my Orchestration to have a schedule transfer from these 250 FTP accounts(all with separate credentials).

2. Also would this require to have setup for about 250 separate Dynamic Send Ports for every FTP Account?

Thanks!

#1. I would suggest to open the ports all the times to take the files from FTP. If you have heavy transaction then schedule the transaction from FTP account.

#2. If every FTP account have separate credential then you need to create 250 separate locations, instead I would suggest to create one common location and map it to all FTP account. That would be easy to manage with BT server.

I will be receiving from about 250 FTP users everyone with different authentication.

Sorry, I should have been a bit more clear in the question.  Will they be sending the files to you or will you have to go and get them?

Either way, Dynamic Ports/Orchestrations will not help. Those are for sending, not receiving.

If the users are sending the files, it's not really an issue for you, they just upload them to you FTP site.

If you have to go get them, you basically have two choices:

1. 250 Receive Locations
2. A script run by the Windows Scheduler or SQL Agent that get the files by a out of band FTP client.

Hi Kamlesh!

For me to keep all the 250 connections open to this FTP server would be a potential risk as this FTP is not only for EDI purposes but open to a lot more integration applications, and we have seen our FTP's crashing due to such problems. So basically, these are just comma delimited, semi colon delimited and positional delimited files that would be picked up from these accounts, this gives me a liberty to group these accounts accordingly. Although in this case I guess it would become mandatory for me to have 250 different ports with individual credentials on to them[Thanks for updating me on that]. Should a schedule for these accounts with 3 different orchestration be a better option?

Thanks!

Nishant,

You can schedule these accounts with different orchestration, But still I want to give you some useful suggestion.

You have mentioned "these are just comma delimited, semi colon delimited and positional delimited files that would be picked up from these accounts", that mean those files picked from FTP RL and process through different orchestration. I would suggest to create one separate FTP location and save all those file there and then use single RL and single orchestration to process the files. That would be more easy and helpful. And one more thing, if you have huge transaction then you can set-up a script with windows task scheduler.

HOLD ON!

If you are Receiving the files, then Orchestrations have nothing to do with this.

Do you really have to use 250 different sets of credentials?  If the users are uploading to a single server, maybe the files can be consolidated there and you just download from a single location.

Nishant,

What is a role of orchestration in your scenario, If its simple receive flat files with different delimiters then I would suggest to go with Johns-305 advise. Consolidate the all files on one location and schedule the task to process the file from single location. That would be simple and easy. 

Hi,

If your users uploading the files to a FTP server, better create another single user to connect from BizTalk to FTP and download the files using this account.

Cheers

Hey John!

Yeah... the problem is that Customers are dropping files to one of my internal servers where they have their individual 250 accounts. Using those same credentials I have one application that pulls out the files and pushes them according to their delimiter types. This application was developed 14 years back and really was plain and simple application with not much scope of proper exception handling or later support.

My concern is that I wish to get that complete infrastructure replaced by a simple BizTalk Receive and Send Ports keep as minimum open connections to the FTP server. Even if I think of bringing the files to one single location I still would require to login to 250 separate accounts and pull those files out to a single location which is equivalent of pushing those files to separate designated folders.

Is it possible that Kamlesh suggested in the comment below, to schedule a transfer of these files as I can sort them out according to different format of delimiters??? I was just thinking if I could instruct the Receive and Send Ports through Orchestration to pickup the files using schedule task adapter???

I would really appreciate if you can guide me a  way out for this...

Thanks a lot!

Do you own the FTP Server?  What platform is it on?

As an aside, you really should not be sharing the same credentials as your customers.  Is that really the case?

Yeah we own the FTP server and its on AIX platform. and yeah that's really the case where the application that is utilizing these credentials are same as what customers have. 

I know its a technical flaw and a SOX issue but I guess Business set this process long back and wouldn't want to change it, although I would definitely go for a separate credential for all the accounts for a single BizTalk User when I go for this infrastructure change.

Well, realistically, it shouldn't be too difficult for an AIX admin to sweep (with a script of course) all of the customer homedirs and copy their files to a single location where your BizTalk app can pick them up.

That is a perfectly reasonable and likely simplest solution.  Then you can delete all those passwords!

Hey John!
Thanks for the suggestion, probably I'll get this proposed to my teammates as a potential solution.

Thanks a lot Everyone for guiding me... :)