SSIS - importing flat files - how to dynamically unpivot the data
Hi Team, i received a flat file with the structure like this product name and last couple of dates with quantity eg: Name | 22/05/2012 | 21/05/2012 ===================================== product1 | 30 |40 product2 | 0 |30 product3 | 25 |24 Now my destination table schema is Name,date,Qty eg: Name | Date | qty ============================ Product1 |22/05/2012 |30 Product1 |21/05/2012 |40 Product2 |22/05/2012 |0 Product2 |21/05/2012 |30 Product3 |22/05/2012 |25 Product3 |22/05/2012 |24 Now,how can i achive this in SSIS... Thanks Anil Anil Inampudi
May 22nd, 2012 5:55am

By using the Unpivot Transformation? http://www.bimonkey.com/2009/07/the-unpivot-transformation/MCTS, MCITP - Please mark posts as answered where appropriate.
Free Windows Admin Tool Kit Click here and download it now
May 22nd, 2012 6:01am

Hi Koen, Thanks for the fast reply..... Here i can't use unpivot because.. the dates will Change at every point of time.. and i cannot modify the package for the next loads...So Can i able to implement in any other way dynamically.. Thanks AnilAnil Inampudi
May 22nd, 2012 6:12am

Hi Anil, I think you could try to full load the file to a table and the data type of all columns should be NVARCHAR, and process the first row as a date-data source, the rest should be the product - qty data source; Then...sperate it to be two: I just test it as SQL way, it works; DECLARE @T1 TABLE ( PRODUCT VARCHAR(10), TM DATE, IDX INT ) DECLARE @T2 TABLE ( PRODUCT VARCHAR(10), QT INT, IDX INT ) INSERT INTO @T1 VALUES('product1','2012-05-22',1),('product1','2012-05-21',2), ('product2','2012-05-22',1),('product2','2012-05-21',2), ('product3','2012-05-22',1),('product3','2012-05-21',2) INSERT INTO @T2 VALUES('product1',30,1),('product1',40,2), ('product2',0,1),('product2',30,2), ('product3',25,1),('product3',24,2) /** SELECT * FROM @T1 product1 2012-05-22 1 product1 2012-05-21 2 product2 2012-05-22 1 product2 2012-05-21 2 product3 2012-05-22 1 product3 2012-05-21 2 **/ /** SELECT * FROM @T2 product1 30 1 product1 40 2 product2 0 1 product2 30 2 product3 25 1 product3 24 2 **/ SELECT t1.PRODUCT, t1.TM, t2.QT FROM @T1 AS t1 INNER JOIN @T2 AS t2 ON t1.PRODUCT = t2.PRODUCT AND t1.IDX = t2.IDX /** Result Shows: product1 2012-05-22 30 product1 2012-05-21 40 product2 2012-05-22 0 product2 2012-05-21 30 product3 2012-05-22 25 product3 2012-05-21 24 **/ I'm not sure if it's a good way, just hope it helps.
Free Windows Admin Tool Kit Click here and download it now
May 22nd, 2012 6:37am

A script component is also possible. Read the first row as data and not as a header. Store the two dates in two variables inside the script component and discard that row. Split every row into two seperate rows (product - qty1 - date1 and product - qty2 - date2) and add both rows to the output buffer.MCTS, MCITP - Please mark posts as answered where appropriate.
May 22nd, 2012 7:06am

Hi, Thanks for your answer for both koen and Simon.. @Simon: In table variable( @T1) i guess, you have hard coded some of the dates(which are like column headers)..but in my case..how can i transform the Column headers to Data rows (as you hard coded).Is there any way..that i can get the result set like table variable( @T1) ??.. if yea,plz share it for me. @koen:If possible, Can you share a sample package for me... Thanks in Advance Anil InampudiAnil Inampudi
Free Windows Admin Tool Kit Click here and download it now
May 22nd, 2012 7:19am

I don't have simular code lying around and I'm not going to write it for you, but this blog posts explains how to set-up a synchronous script component with multiple outputs: http://consultingblogs.emc.com/jamiethomson/archive/2005/09/05/SSIS-Nugget_3A00_-Multiple-outputs-from-a-synchronous-script-transform.aspxMCTS, MCITP - Please mark posts as answered where appropriate.
May 22nd, 2012 7:22am

Hi Anil, @T1 and @T2 just an example what your testing data look like. You can load all data into a table.. Column1 , Column2, Column3 Name | 22/05/2012 | 21/05/2012 product1 | 30 |40 product2 | 0 |30 product3 | 25 |24 Then just transfer them to @T1 and @T2 by SQL way... Or you can refer to Koen by using script.
Free Windows Admin Tool Kit Click here and download it now
May 22nd, 2012 9:47pm

I just made a test in SSIS and SQL...It works.. Everything you need to do is to load the data from the flat file and convert the date.. The SQL I have tested, the result is what you want, hope it helps. My Testing File.. Name |2012-05-22|2012-05-21 product1|30 |40 product2|0 |30 product3|25 |24 /** Load data from flat files to Table T. CREATE TABLE [dbo].[T]( [Column 0] [nvarchar](50) NULL, [Column 1] [nvarchar](50) NULL, [Column 2] [nvarchar](50) NULL ) ON [PRIMARY] Result should be.. Column 0 Column 1 Column 2 Name 2012-05-22 2012-05-21 product1 30 40 product2 0 30 product3 25 24 **/ DECLARE @DATE TABLE ( DATES DATE, IDX INT ) INSERT INTO @DATE SELECT CAST([COLUMN 1] AS DATE) , 1 FROM T WHERE [Column 0] = 'Name' UNION SELECT CAST([COLUMN 2] AS DATE), 2 FROM T WHERE [Column 0] = 'Name' SELECT * FROM @DATE /** DATES IDX 2012-05-21 2 2012-05-22 1 **/ DECLARE @Product TABLE ( Product NVARCHAR(50), QT INT, IDX INT ) INSERT INTO @Product SELECT [Column 0], [Column 1], 1 FROM T WHERE [Column 0] <> 'Name' UNION SELECT [Column 0], [Column 2], 2 FROM T WHERE [Column 0] <> 'Name' SELECT * FROM @Product /** Product QT IDX product1 30 1 product1 40 2 product2 0 1 product2 30 2 product3 24 2 product3 25 1 **/ SELECT P.Product, D.DATES, P.QT FROM @DATE AS D INNER JOIN @Product AS P ON d.IDX = P.IDX /** The result is what you want.. product1 2012-05-22 30 product1 2012-05-21 40 product2 2012-05-22 0 product2 2012-05-21 30 product3 2012-05-22 25 product3 2012-05-21 24 **/
May 22nd, 2012 10:26pm

I just made a test in SSIS and SQL...It works.. Everything you need to do is to load the data from the flat file and convert the date.. The SQL I have tested, the result is what you want, hope it helps. My Testing File.. Name |2012-05-22|2012-05-21 product1|30 |40 product2|0 |30 product3|25 |24 /** Load data from flat files to Table T. CREATE TABLE [dbo].[T]( [Column 0] [nvarchar](50) NULL, [Column 1] [nvarchar](50) NULL, [Column 2] [nvarchar](50) NULL ) ON [PRIMARY] Result should be.. Column 0 Column 1 Column 2 Name 2012-05-22 2012-05-21 product1 30 40 product2 0 30 product3 25 24 **/ DECLARE @DATE TABLE ( DATES DATE, IDX INT ) INSERT INTO @DATE SELECT CAST([COLUMN 1] AS DATE) , 1 FROM T WHERE [Column 0] = 'Name' UNION SELECT CAST([COLUMN 2] AS DATE), 2 FROM T WHERE [Column 0] = 'Name' SELECT * FROM @DATE /** DATES IDX 2012-05-21 2 2012-05-22 1 **/ DECLARE @Product TABLE ( Product NVARCHAR(50), QT INT, IDX INT ) INSERT INTO @Product SELECT [Column 0], [Column 1], 1 FROM T WHERE [Column 0] <> 'Name' UNION SELECT [Column 0], [Column 2], 2 FROM T WHERE [Column 0] <> 'Name' SELECT * FROM @Product /** Product QT IDX product1 30 1 product1 40 2 product2 0 1 product2 30 2 product3 24 2 product3 25 1 **/ SELECT P.Product, D.DATES, P.QT FROM @DATE AS D INNER JOIN @Product AS P ON d.IDX = P.IDX /** The result is what you want.. product1 2012-05-22 30 product1 2012-05-21 40 product2 2012-05-22 0 product2 2012-05-21 30 product3 2012-05-22 25 product3 2012-05-21 24 **/
Free Windows Admin Tool Kit Click here and download it now
May 22nd, 2012 10:27pm

Hi Anil, Please vote my answer as helpful if the solution helps you.
May 25th, 2012 3:29am

Hi Anil, Please vote my answer as helpful if the solution helps you. Don't worry, the forum moderators will do it if Anil doesn't.MCTS, MCITP - Please mark posts as answered where appropriate.
Free Windows Admin Tool Kit Click here and download it now
May 25th, 2012 3:35am

Hi Simon/Koen, Sorry for the belated reply... and thanks for your answers... i have tried a sample solution for this with dynamic sql. the tbale which i have created is a stagin table..it will be created at starting and dropped at last(based on the data).. create table Testa (productid int,[6/5/2012] int,[6/6/2012] int,) insert into Testa select 1,30,20 union all select 2,76,98 union all select 3,54,89 union all select 4,67,33 union all select 5,23,76 union all select 6,45,65 --Select * from Testa declare @Table_name varchar(100), @col varchar(max), @sql nvarchar(max),@col1 varchar(max) set @Table_name ='Testa' set @col =(select stuff((select +','+ quotename( ''+name+'') from (select [name] from syscolumns where id=object_id(@Table_name) and colid >1 ) as a for XML PATH('')),1,1,'') ) set @col1 =(select stuff((select +','+ quotename( ''+name+'') from (select [name] from syscolumns where id=object_id(@Table_name) ) as a for XML PATH('')),1,1,'') ) set @sql = 'select productid,[Date],Qt from (select '+ @col1+' from '+@Table_name+') p unpivot (Qt FOR [Date] IN ('+@col +') ) as unpvt' exec sp_executesql @sql I have tried creating a sproc and call it in control flow and the results are saved to the object variable.. from object variable.. i am reading and writng to destination table ...... any way thanks for your help...keep rocking.. Regards Anil Inampudi Anil Inampudi
June 6th, 2012 3:04am

This topic is archived. No further replies will be accepted.

Other recent topics Other recent topics