Solutions & Explore your data
Snowflake Error Integration for snowpipe and tasks become generally available for AWS, Azure and GCP in February 2023. You can send error message to Azure Event Grid if data pipeline fails. Then message can trigger Azure Function sending teams channel notification. Let’s discribe it in this article.
Continue reading “Snowflake Task Error Notification (Teams Channel)”
Data Quality validation is very hot topic nowadays. Usually, Data Analysts and Data Scientists require to execute some data validation rules before data is consumed by reporting or ML models.
Although, there are many tools allowing data quality checks like Great Expectations and DBT, It is not always easy to integrate these tools with existing ETL processes, especially if data pipelines are orchestrated by Snowflake Tasks.
Continue reading “Snowflake Data Quality Validation”
Snowflake Tasks allow to orchestrate data pipelines in Snowflake. It is extremely cost effective while using STREAM_HAS_DATA as task trigger condition. In order to keep an eye on ETL processes, it is important to manage push notifications when data processing fails.
Continue reading “Snowflake Task Error Notification (Email)”
Currently there is no Excel file format in Snowflake. Although Excel file cannot be natively loaded to Snowflake from Azure Blob Storage using copy into or snowpipe, it can be ingested using Snowpark with Python using pandas library.
DBT stands for Data Build Tool. It is open source project (python library) that allows to easily orchestrate data processing in Snowflake, Databricks, Redshift etc.
Would you like to start using Databricks or Synapse Spark Pool? Let’s review useful Pyspark commands used in Apache Spark DataFrames and transform data …
Power BI is quite often a first choice at early stage of Business Intelligence projects. The more projects, the more customer’s questions, why do we need Data Warehouse if we have Power Query available in Power BI. Let’s discuss it!
Continue reading “Why do I need Data Warehouse if I have Power BI?”
While designing Data Warehouse, it is extremely important to confirm dimension strategy and the way of data analysis with customer. Let’s discuss the most common SCD methodologies (Type 1, 2).
Data Lake allows to store big data volume that can be used by data scientists and data analysts. Well-Designed Data Lake Architecture is extremely important because it can address many business requirements not met in Data Warehouse solutions.
Although, there are many options to move data to Azure, the most popular way is to use Azure Data Factory service.
Continue reading “Move data from on-premise servers to Azure”