Local execution mode

By default, Cosmos uses the local execution mode. It is the fastest way to run Cosmos operators, since it runs dbt either as a library or as a local subprocess. For situations where dbt and Airflow dependencies conflict, Airflow and dbt dependencies conflicts, you most likely have the option to pre-install dbt in an isolated Python virtual environment, either as part of the container image or as part of a pre-start script.

The local execution mode assumes that the Airflow worker node can access a dbt binary. If dbt was not installed alongside Cosmos, you can create a dedicated virtual environment and define a custom path to dbt by declaring the argument ExecutionConfig.dbt_executable_path.

Note

Starting in the 1.4 version, Cosmos tries to leverage the dbt partial parsing (partial_parse.msgpack) to speed up task execution. This feature is bound to dbt partial parsing limitations. Learn more: Partial parsing.

When using the local execution mode, Cosmos converts Airflow Connections into a native dbt profiles file (profiles.yml).

Example of how to use, for instance, when dbt was installed together with Cosmos:

basic_cosmos_dag = DbtDag(
    # dbt/cosmos-specific parameters
    project_config=ProjectConfig(DBT_PROJECT_PATH),
    profile_config=profile_config,
    operator_args={
        "install_deps": True,  # install any necessary dependencies before running any dbt command
        "full_refresh": True,  # used only in dbt commands that support this flag
    },
    # normal dag parameters
    schedule="@daily",
    start_date=datetime(2023, 1, 1),
    catchup=False,
    dag_id="basic_cosmos_dag",
    default_args={"retries": 0},
)