Handling variable column inputs

I receive a data feed from a third party in a pipe delimited file.  From time to time, they add a column at the end.  I would like my ssis package to continue to process the data even if they add a column with out it breaking.

How best do I handle this situation?

Eric S. Fanwick

May 18th, 2015 8:21pm

You can import/load each row as one column then use Derived Column or script component component to map your data.

Also you can read this blog post: http://agilebi.com/jwelch/2007/05/08/handling-flat-files-with-varying-numbers-of-columns/


Free Windows Admin Tool Kit Click here and download it now
May 18th, 2015 8:28pm

I have thought of that in the past, but not thrilled with that solution,  have any other ideas?
May 20th, 2015 2:45pm

The last column absorbs the remaining fields until the row delimiter so if the last column in the current definition is not required, just ignore it all. If you do need the current last column, let it absorb the rest and in the derived column task, split it out and ignore the rest. (Use TOKEN if SSIS2012 up, else a substring to the delimiter - assuming TOKEN is not available pre-2012...)
Free Windows Admin Tool Kit Click here and download it now
May 20th, 2015 5:18pm

If you have your base requirement package developed ready & working ,

its best you take last column large enough [with data type DT_WSTR probably to avoid any data sanity failure in future, as any data could come in future you wouldn't want your package fail for some special character in file in the column that you do not need at all ;) ].

Rest as Ryan already mentioned use derived column : find pipe substring , cast/ convert if you need last column data. 

May 21st, 2015 3:36am

This topic is archived. No further replies will be accepted.

Other recent topics Other recent topics