Airflow and dbt dependencies conflicts#

When using the Local Execution Mode, users may face dependency conflicts between Apache Airflow® and dbt. The conflicts may increase depending on the Airflow providers and dbt adapters being used.

If you find errors, we recommend users look into using alternative execution modes.

In the following table, x represents combinations that lead to conflicts (vanilla apache-airflow and dbt-core packages):

Airflow / DBT

1.0

1.1

1.2

1.3

1.4

1.5

1.6

1.7

1.8

2.2

x

x

x

x

x

x

2.3

x

x

x

x

x

x

x

X

2.4

x

x

x

2.5

x

x

x

2.6

x

x

x

x

x

2.7

x

x

x

x

x

2.8

x

x

x

x

x

x

2.9

x

x

x

x

x

Examples of errors#

The conflict is caused by:
  apache-airflow 2.8.0 depends on pydantic>=2.3.0
  dbt-semantic-interfaces 0.4.2 depends on pydantic~=1.10
  apache-airflow 2.8.0 depends on pydantic>=2.3.0
  dbt-semantic-interfaces 0.4.2.dev0 depends on pydantic~=1.10
  apache-airflow 2.8.0 depends on pydantic>=2.3.0
  dbt-semantic-interfaces 0.4.1 depends on pydantic~=1.10
  apache-airflow 2.8.0 depends on pydantic>=2.3.0
  dbt-semantic-interfaces 0.4.0 depends on pydantic~=1.10

ERROR: Cannot install apache-airflow==2.2.4 and dbt-core==1.5.0 because these package versions have conflicting dependencies.
The conflict is caused by:
  apache-airflow 2.2.4 depends on jinja2<3.1 and >=2.10.1
  dbt-core 1.5.0 depends on Jinja2==3.1.2

ERROR: Cannot install apache-airflow==2.6.0 and dbt-core because these package versions have conflicting dependencies.
The conflict is caused by:
  apache-airflow 2.6.0 depends on importlib-metadata<5.0.0 and >=1.7; python_version < "3.9"
  dbt-semantic-interfaces 0.1.0.dev7 depends on importlib-metadata==6.6.0

ERROR: Cannot install apache-airflow, apache-airflow==2.7.0 and dbt-core==1.4.0 because these package versions have conflicting dependencies.

The conflict is caused by:
    dbt-core 1.4.0 depends on pyyaml>=6.0
    connexion 2.12.0 depends on PyYAML<6 and >=5.1
    dbt-core 1.4.0 depends on pyyaml>=6.0
    connexion 2.11.2 depends on PyYAML<6 and >=5.1
    dbt-core 1.4.0 depends on pyyaml>=6.0
    connexion 2.11.1 depends on PyYAML<6 and >=5.1
    dbt-core 1.4.0 depends on pyyaml>=6.0
    connexion 2.11.0 depends on PyYAML<6 and >=5.1
    apache-airflow 2.7.0 depends on jsonschema>=4.18.0
    flask-appbuilder 4.3.3 depends on jsonschema<5 and >=3
    connexion 2.10.0 depends on jsonschema<4 and >=2.5.1

How to reproduce#

The table was created by running nox with the following noxfile.py:

import nox


nox.options.sessions = ["compatibility"]
nox.options.reuse_existing_virtualenvs = True


@nox.session(python=["3.10"])
@nox.parametrize(
    "dbt_version", ["1.0", "1.1", "1.2", "1.3", "1.4", "1.5", "1.6", "1.7", "1.8"]
)
@nox.parametrize(
    "airflow_version", ["2.2.4", "2.3", "2.4", "2.5", "2.6", "2.7", "2.8", "2.9"]
)
def compatibility(session: nox.Session, airflow_version, dbt_version) -> None:
    """Run both unit and integration tests."""
    session.run(
        "pip3",
        "install",
        "--pre",
        f"apache-airflow=={airflow_version}",
        f"dbt-core=={dbt_version}",
    )