Introduction to Data Build Tool (dbt)
📌 Introduction to DBT and what it stands for data science
Source: https://unsplash.com/
Who is this article for?
This article is for those who want to understand what is DBT and ELT. Also, we recommend it for those who are new to data science and big data processing.
What is Data Build Tool (dbt)
Dbt is a tool for transforming raw data into processed ones that are ready to use for business analysis and any other data-driven solutions.
Extract Load Transform (ELT) paradigm-shift
ETL (Extract Transform Load) flow has been a common practice in data science for a long time. In ETL, engineers transform/process/convert data to a structured one after extracting it from a data warehouse, and then they put it into the data warehouse again. After these steps, data analysts can finally use the data for their work.
We cannot say it is an efficient way to process data because engineers need to download and upload data each time they make some changes in the data, and it causes a delay in the workflow. That is why we can tell ETL is not the best choice from the viewpoint of efficiency.
Source: https://media.striim.com/
With the immersing of cloud technology, data warehouses become more and more powerful. That made it possible to process (transform) data within the warehouse with no computing resource-related problems.
As a result, ELT (Extract Load Transform) flow is getting more popular in data science. In contrast to ETL, we can transform the data within the cloud by attaching a proper computing resource with no downloading and uploading steps.
What does dbt stand for?
Converting data into useful ones is an essential part of both ELT and ETL flows. Dbt (Data Build Tool) is the data transformation tool in the ELT flow.
Data Build Tool (dbt) and its merits
Dbt is a development framework for transforming data within the data warehouse. Dbt makes it possible to manage data transformation logic in one place throughout different data warehouses, including Snowflake, BigQuery, and others. The official website of dbt is https://www.getdbt.com.
Source: https://getdbt.com/
Dbt has the following advantages:
- Allows to manage all source logic in one place, even using different warehouses
- Dbt support many warehouse types including BigQuery, Redshift, Postgres, and Snowflake
- It has automatic document generation. This makes it document always up to date.
- Dbt is open-source, meaning we can deploy it on our servers for free
- If we do not want to build servers, dbt provides Cloud Service with paid plans
- Anyone who has SQL knowledge can use dbt to develop data transformation logic
- Supports data validation and customized testing
Dbt has the following disadvantages:
- Engineer friendly, which means data analysts from the business departments may have some challenges with understanding documents
- It is an SQL query code based; which makes it less readable compared to other tools with GUI
- Basically, it is SQL, but sometimes we need to write the warehouse-specific query
Conclusion
With the paradigm shift of ELT in data science, dbt is becoming a popular service for data analysis. Dbt comes in handy for data analysis, because it supports various warehouses out of the box, and anyone can develop his or her specific logic with only SQL knowledge.
Documentation of your data transformation project can always be up-to-date because dbt supports a helpful feature to generate highly informational documents automatically. For example, the framework generates pictures (lineage graphs) that show model/table interactions and how they connect.
If you are interested in developing your data processing logic in dbt, its online course is a good place to start. The link is here.