Our dbt runs are taking forever. How can I optimize our dbt project for speed?

When dbt runs start taking forever, the biggest wins usually come from finding where the time is going rather than blindly adding compute. A good tuning pass usually focuses on four areas: model materializations, SQL workload, DAG structure, and execution parallelism.

1. Find the slowest models first

Start with your run artifacts:

• Look at target/run_results.json
• Sort models by execution_time
• Identify: - slowest models
• models blocking many downstream models
• models that scan huge amounts of data repeatedly

A 30-minute model at the top of the DAG can hurt more than 20 smaller slow models because everything waits on it.

2. Convert expensive table rebuilds to incremental models

A common cause of slow dbt runs is rebuilding large tables every time:

{{ config(materialized='table') }}

select *
from raw_events

For large append-heavy datasets, switch to incremental:

{{ config(
    materialized='incremental',
    unique_key='event_id'
) }}

select *
from raw_events

{% if is_incremental() %}
where updated_at >= (
    select max(updated_at)
    from {{ this }}
)
{% endif %}

This avoids reprocessing historical data on every run. dbt incremental models are specifically designed to process only new or changed rows instead of rebuilding everything.

Common candidates:

• event tables
• transactions
• logs
• clickstream data
• daily snapshots

Be careful with:

• late-arriving data
• updates to old records
• incorrect unique_key choices

3. Optimize your incremental strategy

Incremental does not automatically mean fast.

Examples:

• Append-only data → append
• Mutable records → merge
• Partitioned time-series data → consider partition-aware strategies

For very large time-series tables, microbatch-style processing can reduce the amount processed per query by splitting work into time windows.

4. Reduce unnecessary model layers

A common pattern:

source
  ↓
staging view
  ↓
intermediate view
  ↓
another intermediate view
  ↓
fact table

Each view can force the warehouse to repeatedly execute upstream logic.

Consider:

• Keep lightweight staging models as views
• Make expensive reusable transformations tables/incrementals
• Use ephemeral models for tiny one-use transformations

Example:

{{ config(materialized='ephemeral') }}

select
    customer_id,
    lower(email) as email
from {{ ref('customers') }}

5. Increase parallelism carefully

dbt can execute independent models concurrently using threads.

Check your profiles.yml:

target: prod

outputs:
  prod:
    type: snowflake
    threads: 8

But more threads are not always faster.

Too many threads can:

• overload warehouse concurrency
• create slot contention
• slow every query down

Tune threads against your warehouse limits.

6. Fix DAG bottlenecks

Look for:

A
|
B
|
C
|
D
|
E

This runs mostly sequentially.

You want more shapes like:

        A
      / | \
     B  C  D
      \ | /
        E

Ways to improve:

• split huge monolithic models
• remove unnecessary dependencies
• avoid creating long chains of intermediate models
• make independent transformations independent

7. Optimize warehouse scans

Your SQL may be the real bottleneck.

Check:

• Are you doing select * everywhere?
• Are you joining billions of rows unnecessarily?
• Are filters applied before joins?
• Are partition columns being filtered?
• Are you scanning historical data repeatedly?

Example improvement:

Before:

select *
from events
join users using(user_id)

After:

select
    e.event_id,
    e.user_id,
    e.event_time,
    u.country
from (
    select *
    from events
    where event_date >= current_date - 7
) e
join users u using(user_id)

8. Use slim CI builds

If your CI pipeline runs the entire project for every PR, use state comparison:

dbt build --select state:modified+

This limits builds to changed models and their downstream dependencies.

9. Clean up unnecessary full refreshes

Audit your jobs:

Bad:

dbt run --full-refresh

every night.

Better:

• normal incremental runs daily
• scheduled full refreshes only when needed
• targeted rebuilds for affected models

10. Check your dbt version

Newer dbt versions include performance improvements in graph processing and incremental workflows. Keeping dbt Core and adapters current can help, especially for larger DAGs.

---

A practical tuning order I’d use:

1. Find top 10 slowest models.
2. Convert the biggest table rebuilds to incremental.
3. Fix the slowest SQL scans.
4. Increase threads until warehouse saturation.
5. Shorten the critical DAG path.
6. Add slim CI selection.

If you share:

• warehouse (Snowflake / BigQuery / Redshift / Databricks)
• current dbt runtime
• number of models
• slowest 3 model runtimes
• whether runs are full refreshes or incremental

I can suggest more targeted changes.

Before:

select *
from events
join users using(user_id)

After:

select
    e.event_id,
    e.user_id,
    e.event_time,
    u.country
from (
    select *
    from events
    where event_date >= current_date - 7
) e
join users u using(user_id)

8. Use slim CI builds

If your CI pipeline runs the entire project for every PR, use state comparison:

dbt build --select state:modified+

This limits builds to changed models and their downstream dependencies.

9. Clean up unnecessary full refreshes

Audit your jobs:

Bad:

dbt run --full-refresh

every night.

Better:

• normal incremental runs daily
• scheduled full refreshes only when needed
• targeted rebuilds for affected models

10. Check your dbt version

Newer dbt versions include performance improvements in graph processing and incremental workflows. Keeping dbt Core and adapters current can help, especially for larger DAGs.

---

A practical tuning order I’d use:

1. Find top 10 slowest models.
2. Convert the biggest table rebuilds to incremental.
3. Fix the slowest SQL scans.
4. Increase threads until warehouse saturation.
5. Shorten the critical DAG path.
6. Add slim CI selection.

If you share:

Optimizing a sluggish dbt project requires attacking bottlenecks at both the warehouse layer and the dbt configuration layer. Here is a structured approach to speeding up your runs:

• Materializations Strategy , Switch large, downstream, or infrequently updated models from view or table to incremental . This ensures you only process new or updated rows instead of full table scans on every run.
• Cluster and Partition , Implement partitioning and clustering (or sorting/distributing keys depending on your warehouse like Snowflake, BigQuery, or Redshift) on your large underlying tables to minimize data scanned during builds.
• Refactor CTEs and Avoid Select Star , Eliminate redundant Common Table Expressions (CTEs) that get materialized or processed repeatedly, and replace SELECT * with explicit column declarations to reduce memory overhead and data transfer.
• Leverage --select and State Comparison , Use --select state:modified+ via the --state flag in production CI/CD pipelines. This ensures you only build models that have actually changed or depend on changed upstream data, rather than rebuilding the entire DAG.
• Optimize Threads , Increase the number of threads in your profiles.yml file (matching your warehouse's concurrency limits) so independent models run in parallel rather than serially.
• Ephemerals Wisely , Use ephemeral materializations for small, single-use transformations that act as simple code clean-ups, but avoid them for complex or heavily-reused logic which can bloat the compiled SQL.
• Audit Heavy Macros and Hooks , Review pre- and post-hooks or heavy custom macros that execute expensive queries or information-schema lookups on every single model build.

If you want to dive deeper, let me know:

• Which data warehouse are you using (Snowflake, BigQuery, Databricks, Redshift, etc.)?
• Are the bottlenecks happening in local development or your production CI/CD pipeline?

Optimizing a dbt project for speed requires a mix of configuration tweaks, architectural adjustments, and database-level tuning. Here is a structured approach to speeding up your dbt runs:

• Use incremental models : Instead of rebuilding entire tables every run, configure heavy models as incremental using the is_incremental() macro so they only process new or updated rows.
• Leverage --select and state : Avoid running your entire DAG every time. Use state-based execution like dbt run --select state:modified+ to run only models that have changed and their downstream dependencies (compare against a previous manifest artifact).
• Refactor ephemeral and view strategies : Use materializations wisely. Heavy transformations shouldn't be ephemeral if they are referenced multiple times (which re-evaluates the CTE everywhere), and don't materialize everything as a table if a view or incremental model suffices.
• Optimize queries and joins : Audit your SQL code for slow operations. Avoid unnecessary SELECT * , reduce expensive cross-joins or heavy window functions on raw data, and pre-aggregate early in your DAG.
• Tune warehouse concurrency and cluster size : Check your data warehouse (e.g., Snowflake, BigQuery, Databricks). Scale up your warehouse size temporarily for heavy runs, or optimize cluster keys / partition strategies so queries scan less data.
• Disable docs generation on production runs : Running dbt docs generate adds overhead. Keep it out of your primary, high-frequency transformation pipelines unless explicitly needed.
• Use slim CI/CD : In your pull request pipelines, run only modified models (--select state:modified ) rather than a full refresh to drastically cut down CI/CD build times.

If you'd like to dive deeper, tell me:

• What data warehouse are you using (Snowflake, BigQuery, Databricks, Postgres, etc.)?
• Are your bottlenecks happening during full refreshes or incremental runs?

I can give you specific configuration snippets or warehouse-level tuning tips for your stack.

When dbt runs take forever, the biggest gains usually come from figuring out where the time is going first: model SQL execution, unnecessary model builds, warehouse contention, or dbt overhead. A good optimization pass usually looks like this:

1. Find the slowest models first

Start with run timing data:

dbt run

Look at the models with the longest execution times. In many projects, a handful of models account for most of the runtime. dbt recommends analyzing model timing and focusing optimization effort on the biggest bottlenecks rather than tuning everything.

Also inspect:

• target/run_results.json
• dbt Cloud run history (if applicable)
• warehouse query history (Snowflake Query Profile, BigQuery Job History, Databricks SQL history, etc.)

---

2. Convert expensive full-refresh tables to incremental models

The biggest win is often avoiding rebuilding large tables.

Before:

{{ config(materialized='table') }}

select *
from {{ source('app', 'events') }}

After:

{{ 
  config(
    materialized='incremental',
    unique_key='event_id'
  ) 
}}

select *
from {{ source('app', 'events') }}

{% if is_incremental() %}

where updated_at >= (
    select max(updated_at)
    from {{ this }}
)

{% endif %}

Incremental models process only new or changed data instead of recreating the entire dataset.

Common improvements:

• Use a reliable updated_at or event timestamp column
• Add a lookback window for late-arriving data
• Choose the right incremental strategy (merge, append, delete+insert, etc.)
• Avoid unnecessary full-refresh runs

---

3. Stop running the whole DAG during development

A common mistake is:

dbt run

for every change.

Instead:

dbt build --select my_model+

or:

dbt build --select state:modified+

State-based selection can avoid rebuilding unchanged parts of your project.

For CI pipelines, consider:

dbt build \
  --select state:modified+ \
  --defer \
  --state path/to/prod/artifacts

This lets unchanged models resolve against production artifacts rather than rebuilding everything.

---

4. Reduce expensive SQL patterns

Look for:

Huge unnecessary scans

Bad:

select *
from huge_table

Better:

select
    id,
    created_at,
    customer_id
from huge_table
where created_at >= '2026-01-01'

Late filtering

Bad:

select *
from (
    select *
    from events
    join users using(user_id)
)
where event_date >= current_date - 7

Better:

select *
from events e
join users u using(user_id)
where e.event_date >= current_date - 7

Push filters as close to the source as possible.

---

5. Review your materializations

Typical patterns:

Be careful with long chains of views. A warehouse may repeatedly expand and execute complex SQL when querying downstream models.

---

6. Tune warehouse-specific performance

Depending on your warehouse:

Snowflake

• Cluster large tables with selective filters
• Avoid repeated scans of massive tables
• Watch warehouse size vs concurrency

BigQuery

• Partition large tables
• Cluster frequently filtered columns
• Avoid select *
• Reduce unnecessary joins

Databricks

• Optimize Delta tables
• Use partitioning carefully
• Compact small files

---

7. Increase parallelism carefully

If your warehouse has capacity, increase dbt threads:

profiles.yml

target: prod

outputs:
  prod:
    type: snowflake
    threads: 8

More threads help only when:

• your DAG has parallelizable branches
• your warehouse can handle concurrency

More threads can make things slower if you overload the warehouse.

---

8. Enable faster parsing for large projects

If startup time is the issue (not SQL execution), enable partial parsing:

flags:
  partial_parse: true

Partial parsing avoids reparsing unchanged project files and can significantly reduce startup time in large projects.

---

9. Audit tests

Tests can quietly add hours.

Look for:

• relationships tests on huge tables
• duplicate scans of the same large datasets
• tests running unnecessarily in every environment

Consider:

• running full tests nightly
• running targeted tests in CI
• testing representative subsets in development

---

A practical optimization order

1. Find top 10 slowest models
2. Convert the biggest full-refresh tables to incremental
3. Remove unnecessary dbt run of the entire DAG
4. Optimize the slowest SQL queries in the warehouse
5. Tune materializations
6. Increase threads if warehouse capacity allows
7. Optimize tests and parsing

If you share:

• warehouse (Snowflake/BigQuery/Redshift/Databricks/etc.)
• current runtime
• number of models
• slowest 3 models and their runtimes
• whether runs are full-refresh or incremental

I can suggest a more targeted tuning plan.

Speeding up sluggish dbt (data build tool) runs requires a mix of configuration tweaks, architectural changes, and incremental processing. Here is a prioritized, scannable guide to optimizing your dbt project:

• Use Incremental Models : Instead of rebuilding entire tables every run, use config(materialized='incremental') . Filter for only new or updated records (e.g., using updated_at > (SELECT MAX(updated_at) FROM {this}})) on subsequent runs, and only do a full refresh when necessary via the --full-refresh flag.
• Leverage Ephemeral Materialization : For small, single-use transformations or CTEs that don't need to be queried downstream as physical database objects, use config(materialized='ephemeral') . This injects the code as a CTE directly into the downstream queries, saving storage and creation overhead.
• Optimize Queries and Indexes : Review your heavy SQL models for unnecessary SELECT * , redundant joins, or expensive window functions. On data warehouses like PostgreSQL, Snowflake, or BigQuery, ensure you are defining appropriate clustering keys, partition fields, or sorting keys in your model configs to minimize data scanning.
• Tune Threads : Increase the concurrency of your dbt run by bumping up the threads in your profiles.yml file or via the CLI flag -t 8 (or higher, depending on your data warehouse's concurrency limits and CPU capacity).
• Use Node Selection Safely : Avoid running the entire project every time if you are only working on a specific feature. Use dbt's powerful graph selection syntax—such as dbt run --select +my_model (upstream dependencies) or dbt run --select state:modified (using artifact comparison against a production manifest) to run only what changed.
• Refactor with Slim CI : Implement a "Slim CI" workflow in your CI/CD pipeline. By comparing your current branch's state against a production manifest.json using --select state:modified+ , you build only the models that were modified and their downstream children, drastically cutting down PR build times.
• Audit Heavy Tests : Heavy data tests (like unique/not_null combinations on massive tables) can slow things down. Consider using warn severities for non-critical checks, scoping tests with where clauses (e.g., testing only active records), or using custom generic tests that run more efficiently.

If you want to dive deeper, let me know:

• Which data warehouse are you using (Snowflake, BigQuery, Databricks, Postgres, etc.)?
• Are you currently using incremental models or a CI/CD tool (like GitHub Actions or GitLab CI)?

I can give you more targeted, warehouse-specific recommendations or help you set up a Slim CI workflow.

The biggest dbt speedups usually come from building less data, running fewer models, and making the warehouse do less work. I’d optimize in this order:

1. Find where the time actually goes

Start by separating dbt overhead from warehouse query time.

Look at:

• Slowest models by execution time
• Models scanning huge tables
• Models that rebuild every run despite rarely changing
• Long dependency chains that serialize work
• Tests that query enormous relations
• Seeds/snapshots or hooks taking significant time

If a model takes 20 minutes because the warehouse query takes 20 minutes, changing dbt configuration won't fix the underlying SQL.

2. Convert expensive full-refresh models to incremental

This is often the biggest win.

Instead of:

select *
from {{ ref('events') }}

use an incremental strategy that only processes new/changed records:

{{ config(
    materialized='incremental',
    unique_key='event_id'
) }}

select
    event_id,
    user_id,
    event_timestamp,
    event_type
from {{ ref('events') }}

{% if is_incremental() %}
where event_timestamp >= (
    select max(event_timestamp)
    from {{ this }}
)
{% endif %}

The important part isn't merely setting materialized='incremental': your SQL needs to avoid scanning/reprocessing the historical dataset.

Also consider your warehouse's best incremental strategy—e.g. merge, delete+insert, or insert-overwrite—and partition/cluster the underlying data appropriately.

3. Filter as early as possible

Avoid carrying billions of rows through intermediate CTEs only to filter them at the end.

Prefer:

with events as (
    select *
    from {{ ref('events') }}
    where event_date >= current_date - 30
),

users as (
    select *
    from {{ ref('users') }}
    where is_active = true
)

...

over joining massive unfiltered relations and applying predicates afterward.

Also avoid select * in production models when you only need a subset of columns.

4. Look for accidental repeated work

A common anti-pattern is having many downstream models independently perform the same expensive transformation.

For example:

raw_events
 ├── model_a → expensive deduplication
 ├── model_b → expensive deduplication
 └── model_c → expensive deduplication

Instead:

raw_events
      ↓
deduped_events
 ├── model_a
 ├── model_b
 └── model_c

Materializing an expensive shared transformation once can dramatically reduce total warehouse work.

The opposite can also happen: creating dozens of unnecessary intermediate tables can increase I/O. Benchmark rather than blindly materializing everything.

5. Exploit dbt's parallelism

dbt can execute independent DAG branches concurrently. If your project has:

A → B → C → D

there isn't much dbt can parallelize.

But:

        → B →
A →           → E
        → C →
        → D →

gives the runner more opportunity to execute work concurrently.

Look for unnecessary dependencies such as models referencing another model merely because it was convenient, rather than because the data dependency is real.

Also check your warehouse's concurrency limits—raising dbt's thread count isn't necessarily faster if it causes warehouse contention.

6. Use state-aware selection in CI

Don't rebuild the entire project for every pull request.

dbt's current documentation highlights dbt State as a way to skip unnecessary rebuilds by identifying nodes whose code/data haven't meaningfully changed.

A typical CI pattern is conceptually:

dbt build --select state:modified+

The + is important: it includes downstream dependencies of modified nodes.

For CI, combine this with a previous production manifest/state directory so a PR only builds the affected portion of the DAG.

7. Make development runs much smaller

Developers shouldn't routinely run the entire warehouse.

Useful patterns include:

dbt build --select my_model+

and limiting development data with a variable or environment-aware filter:

where event_date >= current_date - 7

You can also use --defer with a production manifest so unchanged upstream models can resolve to existing production relations instead of being rebuilt locally.

8. Optimize the SQL for your warehouse

This is often more important than dbt itself.

Check for:

• Large table scans
• Inefficient joins
• Exploding many-to-many joins
• Repeated window functions
• distinct over huge datasets
• Unnecessary order by
• Non-sargable predicates
• Excessive JSON/semi-structured-data parsing
• Poor partition pruning
• Poor clustering/sort/distribution choices

For example, this can be disastrous on a large table:

where cast(event_timestamp as date) = current_date - 1

whereas filtering directly on a partition-friendly timestamp/date column can allow the warehouse to prune most of the table.

9. Don't overlook tests

A surprisingly slow dbt project can actually be a testing problem.

Tests such as uniqueness and relationships can scan enormous tables. Consider:

• Testing appropriate subsets during development
• Avoiding redundant tests
• Ensuring tested columns are efficiently queryable
• Separating expensive comprehensive tests from every developer run

10. Reduce dbt parsing overhead

If your project has thousands of models/macros, local startup/parsing can become noticeable. dbt supports partial parsing, which avoids unnecessarily reparsing the entire project when only limited files have changed.

If parsing is a significant portion of your runtime, also inspect:

• Huge macro files
• Excessive generated YAML
• Large numbers of project files
• Environment-dependent parsing behavior
• Packages you don't actually use

dbt's current tooling also emphasizes faster parsing/development through its Fusion engine and VS Code integration.

---

What I'd do first

If you give me your warehouse (Snowflake, BigQuery, Databricks, Redshift, Postgres, etc.), roughly how many models you have, and a snippet of your slowest model, I can help you identify the highest-impact changes.

A good optimization exercise is:

The key principle: don't start by tweaking threads. First determine whether you're spending the time parsing dbt, waiting on the warehouse, rebuilding unnecessary data, or running an unnecessarily large DAG. Then optimize that bottleneck.

Slow dbt runs are usually caused by one (or more) of these: too much data being reprocessed, inefficient SQL, poor DAG design, warehouse constraints, or running more models than necessary. dbt itself mostly orchestrates SQL execution, so the biggest wins usually come from your models and warehouse.

Here’s a practical optimization checklist:

1. Find your slowest models first

Run with timing enabled and identify the bottlenecks:

dbt run --select model_name --log-level debug

Look for:

• Models taking the majority of runtime
• Models scanning huge source tables
• Models with expensive joins/window functions
• Models blocking many downstream models

A model that takes 20 minutes but feeds 2 models may matter less than a 5-minute model feeding 100 models.

---

2. Convert expensive tables to incremental models

If you rebuild large tables every run, incremental models are often the biggest improvement. They process only new/changed rows instead of rebuilding everything.

Example:

{{ config(
    materialized='incremental',
    unique_key='id'
) }}

select *
from {{ ref('events') }}

{% if is_incremental() %}

where updated_at > (
    select max(updated_at)
    from {{ this }}
)

{% endif %}

Good candidates:

• Event tables
• Transaction tables
• Large fact tables
• Append-heavy datasets

Poor candidates:

• Small dimension tables
• Models where nearly every row changes daily

---

3. Stop overusing views

A common slow pattern:

source
  ↓
view
  ↓
view
  ↓
view
  ↓
table

The warehouse may have to execute the entire chain when the final table runs.

For complex, frequently reused models, consider:

models:
  analytics:
    staging:
      +materialized: table

or:

{{ config(materialized='table') }}

Long chains of views can hurt performance because computation is deferred until downstream queries run.

---

4. Reduce unnecessary model execution

Avoid:

dbt run

for every development change.

Use selection:

dbt run --select my_model+

Runs your model plus downstream dependencies.

Or:

dbt run --select +my_model

Runs upstream dependencies.

For CI, consider state-based selection:

dbt build --select state:modified+

so unchanged models do not rebuild.

---

5. Optimize SQL inside models

Common wins:

Filter early

Slow:

select *
from huge_table
join another_table

Better:

select *
from (
    select *
    from huge_table
    where event_date >= current_date - 30
)
join another_table

Avoid unnecessary select *

Instead of:

select *

use:

select
    customer_id,
    created_at,
    revenue

Avoid repeated calculations

If several models calculate the same expensive transformation, materialize it once.

---

6. Improve joins

Check for:

• Joining billions of rows unnecessarily
• Joining before filtering
• Missing clustering/partitioning
• Many-to-many joins accidentally exploding row counts

Example problem:

fact_orders
join customer_events

If both sides have multiple rows per customer, your row count can explode.

---

7. Use warehouse-specific optimizations

Depending on your platform:

Snowflake

Consider:

• Warehouse sizing
• Clustering keys for large tables
• Incremental strategies
• Avoiding unnecessary full scans

Snowflake specifically recommends incremental models for large frequently updated tables because they avoid rebuilding entire datasets.

BigQuery

Consider:

• Partitioning by date
• Clustering by common filters/joins
• Avoiding queries that scan entire tables

Redshift

Consider:

• Distribution keys
• Sort keys
• Vacuum/analyze strategy

---

8. Increase parallelism carefully

dbt can run models concurrently:

profiles.yml

outputs:
  prod:
    threads: 8

More threads can reduce wall-clock time, but only if:

• Your warehouse has enough compute
• Your DAG has enough parallel branches

Too many threads can make everything slower by competing for resources.

---

9. Remove unnecessary tests from every run

Large projects often spend significant time testing.

Instead of:

dbt test

on everything every time, consider:

• Running critical tests in CI
• Running full tests nightly
• Testing modified models only

---

10. Look for DAG bottlenecks

A healthy DAG looks like:

        A
      / | \
     B  C  D
    / \    |
   E   F   G

A slow DAG looks like:

A → B → C → D → E → F → G

Long dependency chains prevent parallel execution.

---

A good first-week optimization plan

1. Export model runtimes.
2. Fix the top 5 slowest models.
3. Convert large facts to incremental.
4. Replace expensive shared views with tables.
5. Add partitioning/clustering where scans are huge.
6. Use --select and state-based CI runs.
7. Tune threads after SQL improvements.

If you share your warehouse (Snowflake/BigQuery/Redshift/Databricks), number of models, and a sample of your slowest model SQL, I can suggest more targeted changes.

🐢 Slow dbt runs can be a massive bottleneck, but there are several high-impact strategies you can implement to speed things up significantly.

Here is a structured checklist to optimize your dbt project for speed:

• Materialization Strategy: Move large, intermediate models from view or table to incremental . Views recompute on every downstream query, and full tables rebuild from scratch every run. Incremental models only process new or updated data.
• Ephemeral Models: Use ephemeral materialization for lightweight, single-use transformations (like simple renames or filters). They get inlined as CTEs into downstream queries, saving you from cluttering your data warehouse with physical tables or views.
• State-Based Execution (--select state:modified ): Use dbt's state comparison to run or build only the models that have actually changed since your last production run. You can pass the artifact directory from your last production run using --state ./path/to/artifact and run dbt build --select state:modified+.
• Deferral (--defer ): When developing locally or running in CI, use --defer to point your environment at a production database/schema for unmodified models. Instead of building upstream dependencies locally, your local run will query the existing production relations. You can configure this easily using the dbt CLI Guide.
• Concurrency Tuning: Increase your thread count in your profiles.yml file. By default, dbt runs on 1 or 4 threads, but most modern data warehouses (like Snowflake, BigQuery, or Databricks) can easily handle 8, 16, or more concurrent queries. Just ensure your warehouse warehouse-size or concurrency limits can support it.
• Cluster/Partition Pruning: Ensure large source tables and incremental models are properly partitioned and clustered (or sorted) in your data warehouse. If dbt is scanning a whole petabyte-scale table on every incremental filter, it won't matter how optimized your dbt code is.
• Heavy Refactoring via ref() : Audit your dependency graphs (using dbt docs generate and dbt docs serve ) to eliminate redundant JOIN s, expensive window functions early in the DAG, or loops that cause unnecessary fan-outs.

To help narrow down the best fix, could you share:

• What data warehouse are you using (Snowflake, BigQuery, Postgres, etc.)?
• Are you currently running full refreshes every time, or do you use incremental models?
• Is this lag happening in local development, CI/CD pipelines , or production scheduled runs?

When a dbt project starts taking forever, the culprit is usually a mix of inefficient materialization strategies, poor concurrency settings, or a bloated Directed Acyclic Graph (DAG).

To get your run times down quickly, tackle the bottlenecks in this prioritized order:

1. Max out your dbt Threads
   • The Fix: Increase the number of threads in your profiles.yml or via the --threads CLI flag. By default, dbt might be running on 1 or 4 threads, meaning models execute sequentially rather than in parallel.
   • Rule of thumb: Scale up your thread count (e.g., 8, 16, or higher depending on your data warehouse concurrency limits) so independent paths in your DAG run simultaneously. Watch out for resource contention if your warehouse size is too small.
2. Audit run_results.json to find your 80/20 bottleneck
   • The Fix: Don't guess which model is slow. Check the run_results.json artifact generated after a run (or use the Model Timing tab in dbt Cloud ). Typically, 10% of your models account for 90% of the runtime. Drop those specific slow queries into your data warehouse's native query analyzer/profile to see where the heavy scans or shuffles happen.
3. Convert heavy tables to Incremental Models
   • The Fix: If you are doing a full-refresh on massive, append-only fact tables on every run, you are wasting massive amounts of compute. Switch those models to materialized='incremental' so they only process new or updated rows. Ensure you write clean is_incremental() blocks or use the newer microbatch strategies to avoid accidental full-table scans.
4. Apply Partitioning and Clustering
   • The Fix: For your largest tables, make sure you are leveraging database-specific performance features inside your dbt config blocks. Use partition_by (BigQuery/Databricks) or cluster_by (Snowflake) configurations. This allows the database engine to prune irrelevant data blocks instead of executing full table scans on every downstream join.
5. Stop using SELECT * and Filter Early
   • The Fix: Passing every column through every layer of staging and intermediate models inflates memory usage and data shuffling. Explicitly project only the columns you need in your staging models, and apply WHERE filters as early as possible in your CTEs to shrink the dataset before heavy joins and aggregations occur.
6. Prune and Optimize your Tests
   • The Fix: Running heavy generic tests (unique, not_null, relationships ) across millions of rows on every single build adds significant overhead. Restrict tests to critical paths, or scope them down using config blocks or customized where clauses so they don't re-scan entire historical tables.

If you want to narrow this down, let me know:

• Which data warehouse are you using (Snowflake, BigQuery, Databricks, Postgres, etc.)?
• Roughly how many models are in your project and what is your current thread count?

I can give you specific adapter configurations or point you toward profiling strategies for your environment.

The biggest dbt speed wins usually come from reducing how much SQL runs, how much data each model scans, and how much of the DAG you rebuild. dbt itself mostly orchestrates SQL; your warehouse does the heavy lifting.

Here’s a practical optimization checklist:

1. Find your slowest models first

Start by measuring before changing things:

dbt build --select state:modified+

and inspect:

• target/run_results.json
• warehouse query history
• model execution times
• bytes scanned / partitions read

Look for:

• models taking >5–10 minutes
• models that run every day but rarely change
• upstream models causing many downstream rebuilds

---

2. Convert expensive views into tables or incrementals

A common dbt anti-pattern is a long chain of views:

source
  ↓
view
  ↓
view
  ↓
view
  ↓
final table

The warehouse may have to execute the entire chain every time. Materializing expensive intermediate models as tables or incremental models can dramatically reduce runtime.

Example:

{{ config(
    materialized='table'
) }}

select *
from {{ ref('stg_orders') }}

For large fact models:

{{ config(
    materialized='incremental',
    unique_key='order_id'
) }}

select *
from {{ ref('orders_source') }}

{% if is_incremental() %}

where updated_at >
(
    select max(updated_at)
    from {{ this }}
)

{% endif %}

Incremental models can avoid reprocessing the full dataset by only handling new or changed rows.

---

3. Stop running the entire DAG during development

Instead of:

dbt build

use targeted selection:

dbt build --select my_model+

or:

dbt build --select +my_model

Useful selectors:

Large dbt projects often improve developer speed by running only relevant graph sections rather than the entire project.

---

4. Optimize your SQL patterns

Avoid repeated expensive joins

Bad:

model_a
  joins huge_customer_table

model_b
  joins huge_customer_table

model_c
  joins huge_customer_table

Better:

huge_customer_table
        ↓
dim_customer
        ↓
model_a
model_b
model_c

Compute expensive transformations once.

---

Filter early

Slow:

select *
from huge_events
join users using(user_id)
where event_date >= '2026-01-01'

Better:

select *
from (
    select *
    from huge_events
    where event_date >= '2026-01-01'
) events
join users using(user_id)

---

Avoid select *

Instead:

select
    order_id,
    customer_id,
    created_at
from {{ ref('orders') }}

This reduces unnecessary scanning and downstream data movement.

---

5. Increase parallelism

If your warehouse has capacity, increase dbt threads:

profiles.yml

outputs:
  prod:
    type: snowflake
    threads: 8

More threads allow independent models to run concurrently. Snowflake specifically recommends tuning threads based on warehouse capacity.

Don't blindly increase it—too many threads can cause warehouse contention.

---

6. Add incremental strategies that match your data

Different workloads need different approaches:

Append-only events

Example:

• clicks
• page views
• logs

Use:

incremental_strategy: append

Updates/deletes

Example:

• customers
• subscriptions

Use:

incremental_strategy: merge
unique_key: customer_id

Partitioned data

Use warehouse-specific optimizations:

• BigQuery: partition + cluster
• Snowflake: clustering where needed
• Databricks: optimize file layout

---

7. Reduce unnecessary tests

Tests can become a hidden bottleneck.

Heavy examples:

• relationships on billion-row tables
• full table uniqueness checks
• frequent freshness checks

Consider:

• running exhaustive tests nightly
• lighter tests on pull requests
• tagging expensive tests

---

8. Use model layers intentionally

A fast dbt architecture usually looks like:

sources
   |
   ↓
staging (views)
   |
   ↓
intermediate (tables/incremental)
   |
   ↓
marts (tables/incremental)

Avoid making every layer a view.

---

9. Check for warehouse bottlenecks

Sometimes dbt is fine and the warehouse is slow.

Look for:

• spilled joins
• insufficient memory
• missing clustering/partitioning
• too-small warehouse size
• concurrency limits

---

10. A quick triage order

If I inherited a slow dbt project, I’d check in this order:

1. Which 10 models consume 80% of runtime?
2. Can any become incremental?
3. Are there chains of expensive views?
4. Are developers running the full DAG unnecessarily?
5. Are joins scanning unnecessary data?
6. Is warehouse sizing/concurrency the limiter?

If you share:

• warehouse (Snowflake/BigQuery/Redshift/Databricks/etc.)
• number of models
• current runtime
• slowest 3 models
• whether you use incremental models

I can suggest more targeted changes.