GCP GKE execution modes (experimental)#

Added in version 1.15.0.

Warning

This feature is experimental and may change without a deprecation period.

Cosmos provides two execution modes to run dbt in Google Managed Airflow Gen 3 (formerly Cloud Composer 3) with a self-managed custom GKE cluster:

Both modes work the same way as their Kubernetes counterparts but use the GKEStartPodOperator instead of KubernetesPodOperator, which handles GKE authentication automatically and lets you select the cluster to use.

To use either mode, install the Google Cloud provider:

pip install "astronomer-cosmos[google]"

Both modes require three additional parameters in operator_args:

  • project_id: Your GCP project ID

  • location: The GKE cluster location (e.g. us-central1)

  • cluster_name: The GKE cluster name

GCP GKE execution mode#

ExecutionMode.GCP_GKE is the GCP GKE variant of ExecutionMode.KUBERNETES.

from cosmos import DbtDag
from cosmos.config import ExecutionConfig
from cosmos.constants import ExecutionMode

dag = DbtDag(
    dag_id="jaffle_shop_gcp_gke",
    # ... other DAG parameters ...
    execution_config=ExecutionConfig(
        execution_mode=ExecutionMode.GCP_GKE,
        dbt_project_path="dags/dbt/jaffle_shop",
    ),
    operator_args={
        "image": "dbt-jaffle-shop:1.0.0",
        "get_logs": True,
        "project_id": "my-gcp-project",
        "location": "us-central1",
        "cluster_name": "my-gke-cluster",
    },
)

All other configuration (Docker image, secrets, profiles) and the limitations are the same as ExecutionMode.KUBERNETES. For detailed setup instructions, refer to the Kubernetes execution mode documentation.

Watcher GCP GKE execution mode#

ExecutionMode.WATCHER_GCP_GKE is the GCP GKE variant of ExecutionMode.WATCHER_KUBERNETES.

from cosmos import DbtDag
from cosmos.config import ExecutionConfig
from cosmos.constants import ExecutionMode

dag = DbtDag(
    dag_id="jaffle_shop_watcher_gcp_gke",
    # ... other DAG parameters ...
    execution_config=ExecutionConfig(
        execution_mode=ExecutionMode.WATCHER_GCP_GKE,
        dbt_project_path="dags/dbt/jaffle_shop",
    ),
    operator_args={
        "image": "dbt-jaffle-shop:1.0.0",
        "get_logs": True,
        "project_id": "my-gcp-project",
        "location": "us-central1",
        "cluster_name": "my-gke-cluster",
    },
)

All limitations and configuration from ExecutionMode.WATCHER_KUBERNETES apply. For more details, refer to the Watcher Kubernetes execution mode documentation.