Solutions & Explore your data
Snowflake Error Integration for snowpipe and tasks become generally available for AWS, Azure and GCP in February 2023. You can send error message to Azure Event Grid if data pipeline fails. Then message can trigger Azure Function sending teams channel notification. Let’s discribe it in this article.
Continue reading “Snowflake Task Error Notification (Teams Channel)”
Data Quality validation is very hot topic nowadays. Usually, Data Analysts and Data Scientists require to execute some data validation rules before data is consumed by reporting or ML models.
Although, there are many tools allowing data quality checks like Great Expectations and DBT, It is not always easy to integrate these tools with existing ETL processes, especially if data pipelines are orchestrated by Snowflake Tasks.
Continue reading “Snowflake Data Quality Validation”
Snowflake Tasks allow to orchestrate data pipelines in Snowflake. It is extremely cost effective while using STREAM_HAS_DATA as task trigger condition. In order to keep an eye on ETL processes, it is important to manage push notifications when data processing fails.
Continue reading “Snowflake Task Error Notification (Email)”
DBT stands for Data Build Tool. It is open source project (python library) that allows to easily orchestrate data processing in Snowflake, Databricks, Redshift etc.
Would you like to start using Databricks or Synapse Spark Pool? Let’s review useful Pyspark commands used in Apache Spark DataFrames and transform data …
Data Lake allows to store big data volume that can be used by data scientists and data analysts. Well-Designed Data Lake Architecture is extremely important because it can address many business requirements not met in Data Warehouse solutions.