Cloud Platform instructions¶

Amazon Web Services (AWS)¶

The below steps are useful if you have your Kubernetes cluster hosted on AWS EKS

1. Add AWS CLI to your Airflow worker

pip install awscli

Or add the package to your requirements.txt file. For other methods to install the CLI, please see here

2. Authenticate with AWS

AWS_ACCESS_KEY_ID=<your-access-key>
AWS_SECRET_ACCESS_KEY=<your-secret-access-key>
AWS_SESSION_TOKEN=<your-session-token>

The Airflow worker will need to access the Kubernetes cluster on AWS. Setup access using one of the methods listed here1

If you have your Kubernetes cluster hosted on Azure Kubernetes Service (AKS), follow these steps:

1. Install Azure CLI on your Airflow worker

pip install azure-cli

For other installation methods, please refer to the Azure CLI installation guide.

2. Authenticate with Azure

To authenticate the Airflow worker with Azure, get the below details from your k8 cluster’s service principal:

AZURE_CLIENT_ID=<your-client-id>
AZURE_CLIENT_SECRET=<your-client-secret>
AZURE_TENANT_ID=<your-tenant-id>

Replace <app-id>, <password-or-cert>, and <tenant> with your specific Azure credentials.

For more authentication options, see the Azure authentication guide.

1. Install the Google Cloud CLI on your Airflow worker

pip install gcloud

2. Authenticate with GCP

GOOGLE_APPLICATION_CREDENTIALS="/path/to/your/service-account-key.json"