Cloud Data Warehousing Data Revolution

Global Snowflake: Loading Data into Snowflake from Azure Blob Storage
March 30, 2018
Cloud Data Warehousing Data Revolution
April 3, 2018

Cloud Data Warehousing Data Revolution

Cloud Data Warehousing Data Revolution

It’s the age of disruption. Your business must be agile and be data-driven or quite often it will be disrupted and replaced. It is scary and exciting all at the same time.

Data Problems and Inefficiencies are prevalent at most organizations.
While Data Warehousing and even Big Data now has been around awhile it traditionally runs into scaling issues and its costly, slow to implement, complex, and non-flexible. Most organizations we survey, assess, and work with have a ton of data and managing it is getting more and more complex. Businesses also want access to data faster and faster for both analysis and automation. Slow moving CIOs and technical divisions are getting side-stepped by marketing and operations to move data to their own cloud silos increasing data complexity and security issues. Data complexity is growing rapidly and these are the major data problems almost everyone has if they are not looking at new innovative solutions:

Major data problems:
• Data speed. Data loading for many businesses is still batch driven and often takes hours and sometimes even days. Modern businesses just cannot wait this long to analyze and drive automation.
• Data concurrency problems. Business users often cannot access the latest data fast enough or have to wait until loading is done.
• Data sources are more numerous and varied. (Not just traditional rows/columns but JSON, AVRO, Parquet, etc.)
• Data is almost always in silos and cannot be cross referenced.
• Data access is often complex.

***On top of all this data security is a huge issue as security breaches continue to grow.

Snowflake Cloud Data Warehousing to the Rescue.
We are seeing clients have tremendous wins with cost savings and business performance with Snowflake.
• 78% cost savings replacing on-prem data warehouse and Hadoop.
• Moving to new solutions in weeks versus months or years.
• ETLs going from days to minutes.

This is why I’m the most excited I’ve been in years with the data solutions we can provide now. In the past we scaled solutions to help clients out with the latest on-prem data warehouses or cloud big data solutions including Teradata, Neteeza, Vertica, Impala, Presto, Redshift, and Hadoop but it was not easy. The business side of me always thought this is just too damn complex.

Why Cloud Data Warehousing with Snowflake?
We kept thinking how can we do this better and now with Snowflake we can. For most of our use cases we finally have something that outperforms all on-prem and other cloud big data solutions from a cost and most importantly EASE of USE perspective.

Traditional Data Warehouses suffer from many of the issues we outlined already. Hadoop suffers from complexity and infrastructure and resource costs. Other cloud data warehouse technologies were not built for the cloud and either do not support SQL or have concurrency issues.

The big advantages we have now with Snowflake are:

Speed and Ease of Use are huge.
• SQL is the most common technical language used. It’s relatively easy for even business users to pick up.
• Being able to easily load, query, and relate JSON, XML, and other sources with relational data makes analysis much faster.
• Imagine how awesome it is to create an entire clone of production in seconds. This is amazingly efficient for QA and Data Quality.
• Security is now taken care of for you.

Lower initial costs and TCO are huge.
1. Separating Compute and Storage opens up major innovations not available before. No concurrency limitations and issues that are present on all other solutions. Since compute can be separated then you can have isolated workloads for data loading, marketing, operations, data scientists, etc. etc.
2. Paying only for what you use. Now you can effectively size your costs for your workload when you need it. No longer do you have to buy hardware to scale for the maximum use cases.
3. Going to 1/10th the cost of database administration is amazing for TCO. All that expertise you had with indexing, vacuuming, etc. is no longer needed to pay for. It comes as part of the service.

Overall business data management becomes easier and more cost effective which adds incredible value to business performance and operations.

Frank Bell is a Founder and Principal at IT Strategists.

Frank has worked on data systems for over 20 years. He has done everything from leading teams producing some of the largest air combat data driven systems to building data warehouses that analyze data for the largest event systems in the world.