SQL Azure-based Data Warehousing

Please let me know if you have implemented large scale data warehouse on SQL Azure. We are looking at a size of 15-20 TB data. A significant part of our data originates from files (semi-structured, unstructured) for which we are planning to use HDInsight to mine/process the data and store the extracted + aggregated (for example minute-level data to be aggregated to day level) data on the DWH.

Has SQL Azure been used successfully as a DWH for high volumes? Please share your thoughts.

June 19th, 2013 4:55pm

just curious that azure db will have a 150gb limit per db, with your data size needs you will need a large number of databases, 

have you explored option of on premises db?

Free Windows Admin Tool Kit Click here and download it now
June 20th, 2013 12:38am

SQL Azure is not focussed on a Datawarehouse environment, particularly high volumes.  For a start, the maximum database size is 150GB, so you'd need a lot of them, and then for reporting you'd need to run fan-out queries then collate and aggregate the data in some way.  At present no reporting tools offer this functionality, so you'd probably have to write a custom datasource to do it, or pull the data into a central area first (thus negating the use of SQL Azure in the first place). 

Even a persistent VM would probably not support your requirements, as at present you are limited to 16TB of storage - that is disk space, so you need to take everything into account such as transaction logs, tempdb etc.

June 20th, 2013 11:15am

Thanks for the reply. I was thinking in terms of federations - however, I understand that managing/accessing the DWH may be a challenge in that case. Yes, on premise DB, PDW, HIVE (on HDInsight) are the other solutions we are thinking of.
Free Windows Admin Tool Kit Click here and download it now
June 24th, 2013 7:22am

This topic is archived. No further replies will be accepted.

Other recent topics Other recent topics