Optimize Performance#
Cosmos performance can be tuned across two dimensions: how fast DAGs are parsed (affecting how quickly they appear and update in Airflow) and how fast tasks execute (affecting DAG run duration).
Optimize DAG Parsing – Speed up DAG parsing by choosing the right LoadMode, reducing DAG granularity, and skipping stale sources.
Optimize Task Execution – Speed up DAG runs by choosing the right execution mode, sizing workers, and reducing per-task overhead.
Troubleshooting Performance – Diagnose common performance issues such as slow parsing, missing DAGs, and Out of Memory (OOM) errors.
The following pages cover specific optimization mechanisms in more detail:
Memory Optimization Options for Astronomer Cosmos – Reduce memory consumption during DAG parsing and task execution.
Caching – How Cosmos caches dbt ls output, partial parse files, profiles, and YAML selectors.
Invocation Modes – Choose between running dbt as a library or as a subprocess.