Cosmos Config#
This page lists all available Airflow configurations that affect astronomer-cosmos
Astronomer Cosmos behavior. They can be set in the airflow.cfg file
or using environment variables.
Note
For more information, see Setting Configuration Options.
Sections:
[cosmos]
[openlineage]
[cosmos]#
- cache_dir:
The directory used for caching Cosmos data.
Default:
{TMPDIR}/cosmos_cache
(where{TMPDIR}
is the system temporary directory)Environment Variable:
AIRFLOW__COSMOS__CACHE_DIR
- enable_cache:
Enable or disable caching of Cosmos data.
Default:
True
Environment Variable:
AIRFLOW__COSMOS__ENABLE_CACHE
- enable_cache_dbt_ls:
Enable or disable caching of the dbt ls command in case using
LoadMode.DBT_LS
in an Airflow Variable.Default:
True
Environment Variable:
AIRFLOW__COSMOS__ENABLE_CACHE_DBT_LS
- enable_cache_partial_parse:
Enable or disable caching of dbt partial parse files in the local disk.
Default:
True
Environment Variable:
AIRFLOW__COSMOS__ENABLE_CACHE_PARTIAL_PARSE
- enable_cache_package_lockfile:
Enable or disable caching of dbt project package lockfile.
Default:
True
Environment Variable:
AIRFLOW__COSMOS__ENABLE_CACHE_PACKAGE_LOCKFILE
- propagate_logs:
Whether to propagate logs in the Cosmos module.
Default:
True
Environment Variable:
AIRFLOW__COSMOS__PROPAGATE_LOGS
- dbt_docs_dir:
The directory path for dbt documentation.
Default:
None
Environment Variable:
AIRFLOW__COSMOS__DBT_DOCS_DIR
- dbt_docs_conn_id:
The connection ID for dbt documentation.
Default:
None
Environment Variable:
AIRFLOW__COSMOS__DBT_DOCS_CONN_ID
- enable_cache_profile:
Enable caching for the DBT profile.
Default:
True
Environment Variable:
AIRFLOW__COSMOS__ENABLE_CACHE_PROFILE
- profile_cache_dir_name:
Folder name to store the DBT cached profiles. This will be a sub-folder of
cache_dir
Default:
profile
Environment Variable:
AIRFLOW__COSMOS__PROFILE_CACHE_DIR_NAME
- remote_cache_dir:
The remote directory to store the dbt cache. Starting with Cosmos 1.6.0, you can store the dbt ls output as cache in a remote location (an alternative to the Variable cache approach released previously since Cosmos 1.5.0) using this configuration. The value for the remote cache directory can be any of the schemes that are supported by the Airflow Object Store feature introduced in Airflow 2.8.0 (e.g.
s3://your_s3_bucket/cache_dir/
,gs://your_gs_bucket/cache_dir/
,abfs://your_azure_container/cache_dir
, etc.)This is an experimental feature available since Cosmos 1.6 to gather user feedback and will be merged into the
cache_dir
setting in upcoming releases.Default:
None
Environment Variable:
AIRFLOW__COSMOS__REMOTE_CACHE_DIR
- remote_cache_dir_conn_id:
The connection ID for the remote cache directory. If this is not set, the default Airflow connection ID identified for the scheme will be used.
Default:
None
Environment Variable:
AIRFLOW__COSMOS__REMOTE_CACHE_DIR_CONN_ID
[openlineage]#
- namespace:
The OpenLineage namespace for tracking lineage.
Default: If not configured in Airflow configuration, it falls back to the environment variable
OPENLINEAGE_NAMESPACE
, otherwise it usesDEFAULT_OPENLINEAGE_NAMESPACE
.Environment Variable:
AIRFLOW__OPENLINEAGE__NAMESPACE
Note
For more information, see Openlieage Configuration Options.
Environment Variables#
- LINEAGE_NAMESPACE:
The OpenLineage namespace for tracking lineage.
Default: If not configured in Airflow configuration, it falls back to the environment variable
OPENLINEAGE_NAMESPACE
, otherwise it usesDEFAULT_OPENLINEAGE_NAMESPACE
.