Hi, I have a huge xml file which I need to parse and load into a database. The file is about 1.7 Gig! Fun! I initially tried to use an XML source but I get an OutOfMemoryException error. Is it possible to use an XML source and optimize the memory use so that it can run? If not, is there a good place to go to for learning how this could be done in a script task? I have many nodes that need to be joined together in order to flatten the data, but I have no experience with scripting xml. Thanks for any advice on this, mark

Is the error occurring because of just loading the file or it is a result of some transformation?Arthur My Blog

That was loading the file at the XML source task. In my testing I only pulled the top level node which has 4 columns and attached it to a union transformation. I put a data viewer on it, put it never makes it past the source task. Mark

AFAIK SSIS does not have any problem laoding even larger XML files. Your issue appears to be related to what you are doing with the XML, namely the Union tranformation. And more specifically it will have to eventually boil down to ways of using the memory more efficient. Skim through some performance tuning techniques: http://technet.microsoft.com/en-us/library/cc966529.aspx and this: http://www.sql-server-performance.com/articles/biz/SSIS_An_Inside_View_Part_2_p1.aspxArthur My Blog

you can use XML Task and use XSLT to flatten your data first , and then use it in data flow xml source. an example of using xslt in XML task is here: http://www.rad.pasfu.com/index.php?/archives/21-XML-Task-Changing-Style-of-Data-XSLT.htmlhttp://www.rad.pasfu.com

Reza, I will try your method out - I am totally not familar but the example in the link looks like what I need to do. I'll give feedback shortly if I still am getting loading errors, but I plan to only work on a small portion of the XML for testing Mark

You may want to read up on Simon Sabin's technique. Talk to me now on