Course Overview
Introduction to dbt

Incremental Views

Lesson #18

In this lesson we will:

  • Learn about dbt incremental update feature, which allows us to incrementally keep our destination tables up to date as new data is entered into our data sources.

What Are Incremental Updates

As we have seen in previous lessons, dbt will build new tables or views from existing tables and views, potentially in a chain of dependencies.

By default, each time dbt creates the new objects, it drops and then rebuilds the table from scratch.

This makes dbt very simple to reason about, and each time we can be sure we are getting fresh up-to-date data.

There are however a number of downsides of this approach:

  • It can be slow, as we are recomputing all of the queries and rebuilding all of the tables;
  • It can create significant load on the database;
  • This database load can in turn result in additional costs if you are paying for compute;

Incremental views enable us to update our views based only on deltas.

This is quite a simple ask, but does result in the data engineer taking on additional complexity in ensuring that their tables remain up to date and containing accurate data.

Next Lesson:

Incremental Views

In this lesson we will learn about dbts incremental updates and incremental views.

0h 15m

Work With The Experts In Real-Time Analytics & AI

we help enterprise organisations deploy powerful real-time Data, Analytics and AI solutions based on ClickHouse, the worlds fastest open-source database.

Join our mailing list for regular insights:

We help enterprise organisations deploy advanced data, analytics and AI enabled systems based on modern cloud-native technology.

© 2024 Ensemble. All Rights Reserved.