Hi there, I am trying to run One data flow task which has 10 look up and source and destination adapters.it loads around 20 million rows in destination but its giving below error. I had ran successfully couple of time this data flow task but its giving me error now. DTS.Pipeline] Error: The Data Flow task failed to create a required thread and cannot begin running. The usually occurs when there is an out-of-memory state. [DTS.Pipeline] Error: The Data Flow task engine failed at startup because it cannot create one or more required threads. Need help immediate! Thanks,

I am running through Bids right now on my local. I am getting this error after couple of minuets. I have set up 100 Mb buffering size ,no cache option on ,threads values manually set up up to 50 still same error.. so are you saying that i should deploy to development machine and try to test? It used ran successfully on my local. Thanks,

Yes, do. But do not play around with the buffers or any other memory related settings at this stage. It is because SSIS has an algorithm to come with a more or less optimal setting on the package's behalf, so unless you are absolutely sure the out of the box setting are no good do not tweak it. Later, if needed, look at SSIS performance tuning techniques.Arthur My Blog

i don't think so we can do better performance then this because we are just using Look up , source and destination nothing else. but volume of data is more here nothing else. if any one having idea to do it same with different techniques please let me know. need help immediate. Thanks,

It seems like your issue is probably related to the memory required by the lookups - are they all fully cached? Can you alter some of them to use only partial caching? Talk to me now on

I am using all of them as no cache but still same error after couple of minutes. do you thing is it make different by running on dev vs your local? Thanks,

No caching? What else is in this package? Are any other tasks running concurrently? How much free RAM do you have on your machine before the package starts? Talk to me now on

hey how to check free ram on my machine? there is nothing else in package. Just source ,destination and 10 look ups. it loads 20 million records in destination. each look ups it will match up 10 million records.in all of the look up using no cache option. Thanks,

Well, in most scenarios SSIS is part/on the same box where the SQL Server is, and SQL Server is if not specially configured (in terms how much memory it can consume max) will just sooner or later eat up all the available RAM. So this approach of checking how much memory is consumed is not going to reveal the truth. I suggest you change the design of the package instead, how about you do the data processing in chunks?Arthur My Blog

I can change design of package but i don't have a lot more transformation in it . just source , destination and look ups...do know have any ideas to change way around? Thanks,

Either the limit the amount of data you are looking at (processing) or reduce the number of lookups, E.g. do not use cache, btw it works better in SSIS 2008Arthur My Blog

Right click on the taskbar and select Task Manager. Talk to me now on

guys got solution for that it wasn't any problem i just close BIDS and open up again and restart my computer and it works fine... i don't know there wasn't any reason and i still don't understand...why? but i got make it works. thank you very much for your quick response. Thanks,