Starship Migration DAG¶

The StarshipAirflowMigrationDAG can be used to migrate Airflow Variables, Pools, Connections, and DAG History from one Airflow instance to another.

The StarshipAirflowMigrationDAG should be used in instances where the source Airflow Webserver is unable to correctly host a Plugin. The Target must still have a functioning Starship Plugin installed, be running the same version of Airflow, and have the same set of DAGs deployed.

The StarshipAirflowMigrationDAG should be used if migrating from a Google Cloud Composer 1 (with Airflow 2.x) or MWAA v2.0.2 environment. These environments do not support webserver plugins and will require using the StarshipAirflowMigrationDAG to migrate data.

Installation¶

Add the following lines to your requirements.txt in your source environment:

astronomer-starship
apache-airflow-providers-http

Note

You only need apache-airflow-providers-http if you're using the migration DAG. If you're only using the Starship plugin UI, astronomer-starship alone is enough. If the provider is missing, importing the DAG will raise an ImportError.

Setup¶

Target connection¶

Make a connection in Airflow with the following details:

Conn ID: starship_default
Conn Type: HTTP
Host: the URL of the homepage of Airflow (excluding /home on the end of the URL)
For example, if your deployment URL is https://astronomer.astronomer.run/abcdt4ry/home, you'll use https://astronomer.astronomer.run/abcdt4ry
Schema: https
Extra: {"Authorization": "Bearer <token>"}

Source connection (Airflow 3 only)¶

On Airflow 3 the migration DAG cannot use direct database access from workers, so the source hook now talks to the source Airflow's Starship HTTP API. This requires a second Airflow connection on the source deployment itself:

Conn ID: starship_source (default; overridable via source_http_conn_id)
Conn Type: HTTP
Host: the base URL of the source Airflow (same rules as above -- exclude any /home suffix)
Schema: https
Extra: {"Authorization": "Bearer <token>"}

The token must be valid for the source Airflow's /api/starship/* endpoints. If the connection is missing when the DAG runs, Starship raises a clear RuntimeError naming the connection and explaining the requirement, so you can add the connection and re-run.

On Airflow 2 no source connection is needed -- the source hook reads directly from the local metadata DB, exactly as it did in prior releases.

Usage¶

Add the following DAG to your source environment.

Airflow 3:

dags/starship_airflow_migration_dag.py

from astronomer_starship.providers.starship.operators.starship import (
    StarshipAirflowMigrationDAG,
)

globals()["starship_airflow_migration_dag"] = StarshipAirflowMigrationDAG(
    target_http_conn_id="starship_default",
    source_http_conn_id="starship_source",
)

Airflow 2:

dags/starship_airflow_migration_dag.py

from astronomer_starship.providers.starship.operators.starship import (
    StarshipAirflowMigrationDAG,
)

globals()["starship_airflow_migration_dag"] = StarshipAirflowMigrationDAG(
    target_http_conn_id="starship_default",
)

Unpause the DAG in the Airflow UI
Once the DAG successfully runs, your connections, variables, and environment variables should all be migrated to Astronomer

Configuration¶

The StarshipAirflowMigrationDAG can be configured as follows.

Airflow 3 (preferred)¶

StarshipAirflowMigrationDAG(
    target_http_conn_id="starship_default",
    source_http_conn_id="starship_source",
    variables=None,  # None to migrate all, or ["var1", "var2"] to migrate specific items, or empty list to skip all
    pools=None,  # None to migrate all, or ["pool1", "pool2"] to migrate specific items, or empty list to skip all
    connections=None,  # None to migrate all, or ["conn1", "conn2"] to migrate specific items, or empty list to skip all
    dag_ids=None,  # None to migrate all, or ["dag1", "dag2"] to migrate specific items, or empty list to skip all
)

Airflow 2 (preferred)¶

StarshipAirflowMigrationDAG(
    target_http_conn_id="starship_default",
    variables=None,  # None to migrate all, or ["var1", "var2"] to migrate specific items, or empty list to skip all
    pools=None,  # None to migrate all, or ["pool1", "pool2"] to migrate specific items, or empty list to skip all
    connections=None,  # None to migrate all, or ["conn1", "conn2"] to migrate specific items, or empty list to skip all
    dag_ids=None,  # None to migrate all, or ["dag1", "dag2"] to migrate specific items, or empty list to skip all
)

Airflow 2 (legacy `http_conn_id`, still supported)¶

StarshipAirflowMigrationDAG(
    http_conn_id="starship_default",
    variables=None,  # None to migrate all, or ["var1", "var2"] to migrate specific items, or empty list to skip all
    pools=None,  # None to migrate all, or ["pool1", "pool2"] to migrate specific items, or empty list to skip all
    connections=None,  # None to migrate all, or ["conn1", "conn2"] to migrate specific items, or empty list to skip all
    dag_ids=None,  # None to migrate all, or ["dag1", "dag2"] to migrate specific items, or empty list to skip all
)

Connection kwargs¶

Kwarg	Purpose	Default	Notes
`target_http_conn_id`	HTTP conn id for the target Airflow (where data is written)	falls back to `http_conn_id`	Preferred name
`source_http_conn_id`	HTTP conn id for the source Airflow (where data is read from)	`"starship_source"`	Airflow 3 only; ignored on Airflow 2
`http_conn_id`	Legacy alias for `target_http_conn_id`	`None`	Kept for backward compatibility

You can use this DAG to migrate all items, or specific items by providing a list of names.

You can skip migration by providing an empty list.

Python API¶

Hooks¶

Hooks for interacting with Starship migrations.

Classes:

Name	Description
`StarshipHttpHook`

StarshipHttpHook ¶

Methods:

Name	Description
`get_connections`	Get all connections from the Target Starship instance.
`get_dag_runs`	Get DAG runs from the Target Starship instance.
`get_dags`	Get all DAGs from the Target Starship instance.
`get_pools`	Get all pools from the Target Starship instance.
`get_task_instances`	Get task instances from the Target Starship instance.
`get_variables`	Get all variables from the Target Starship instance.
`set_connection`	Set a connection in the Target Starship instance.
`set_dag_is_paused`	Set the paused status of a DAG in the Target Starship instance.
`set_dag_runs`	Set DAG runs in the Target Starship instance.
`set_pool`	Set a pool in the Target Starship instance.
`set_task_instances`	Set task instances in the Target Starship instance.
`set_variable`	Set a variable in the Target Starship instance.

get_connections ¶

get_connections()

Get all connections from the Target Starship instance.

get_dag_runs ¶

get_dag_runs(dag_id: str, offset: int = 0, limit: int = 10) -> dict

Get DAG runs from the Target Starship instance.

get_dags ¶

get_dags() -> dict

Get all DAGs from the Target Starship instance.

get_pools ¶

get_pools()

Get all pools from the Target Starship instance.

get_task_instances ¶

get_task_instances(dag_id: str, offset: int = 0, limit: int = 10)

Get task instances from the Target Starship instance.

get_variables ¶

get_variables()

Get all variables from the Target Starship instance.

set_connection ¶

set_connection(**kwargs)

Set a connection in the Target Starship instance.

set_dag_is_paused ¶

set_dag_is_paused(dag_id: str, is_paused: bool)

Set the paused status of a DAG in the Target Starship instance.

set_dag_runs ¶

set_dag_runs(dag_runs: List[dict]) -> dict

Set DAG runs in the Target Starship instance.

set_pool ¶

set_pool(**kwargs)

Set a pool in the Target Starship instance.

set_task_instances ¶

set_task_instances(task_instances: list[dict]) -> dict

Set task instances in the Target Starship instance.

set_variable ¶

set_variable(**kwargs)

Set a variable in the Target Starship instance.

Operators, TaskGroups, DAG¶

Operators, TaskGroups, and DAGs for interacting with the Starship migrations.

Classes:

Name	Description
`StarshipConnectionMigrationOperator`	Operator to migrate a single Connection from one Airflow instance to another.
`StarshipDagHistoryMigrationOperator`	Operator to migrate a single DAG from one Airflow instance to another, with it's history.
`StarshipPoolMigrationOperator`	Operator to migrate a single Pool from one Airflow instance to another.
`StarshipVariableMigrationOperator`	Operator to migrate a single Variable from one Airflow instance to another.

Functions:

Name	Description
`StarshipAirflowMigrationDAG`	DAG to fetch and migrate Variables, Pools, Connections, and DAGs with history from one Airflow instance to another.
`assert_source_conn_exists`	Raise a clear error if an HTTP-based source connection is missing.
`starship_connections_migration`	TaskGroup to fetch and migrate Connections from one Airflow instance to another.
`starship_dag_history_migration`	TaskGroup to fetch and migrate DAGs with their history from one Airflow instance to another.
`starship_pools_migration`	TaskGroup to fetch and migrate Pools from one Airflow instance to another.
`starship_variables_migration`	TaskGroup to fetch and migrate Variables from one Airflow instance to another.

StarshipConnectionMigrationOperator ¶

StarshipConnectionMigrationOperator(connection_id: Union[str, None] = None, **kwargs)

Operator to migrate a single Connection from one Airflow instance to another.

StarshipDagHistoryMigrationOperator ¶

StarshipDagHistoryMigrationOperator(target_dag_id: str, unpause_dag_in_target: bool = False, dag_run_limit: int = 10, **kwargs)

Operator to migrate a single DAG from one Airflow instance to another, with it's history.

StarshipPoolMigrationOperator ¶

StarshipPoolMigrationOperator(pool_name: Union[str, None] = None, **kwargs)

Operator to migrate a single Pool from one Airflow instance to another.

StarshipVariableMigrationOperator ¶

StarshipVariableMigrationOperator(variable_key: Union[str, None] = None, **kwargs)

Operator to migrate a single Variable from one Airflow instance to another.

StarshipAirflowMigrationDAG ¶

StarshipAirflowMigrationDAG(http_conn_id: str = None, variables: List[str] = None, pools: List[str] = None, connections: List[str] = None, dag_ids: List[str] = None, source_http_conn_id: str = None, target_http_conn_id: str = None, **kwargs)

DAG to fetch and migrate Variables, Pools, Connections, and DAGs with history from one Airflow instance to another.

assert_source_conn_exists ¶

assert_source_conn_exists(http_conn_id: str) -> None

Raise a clear error if an HTTP-based source connection is missing.

Only meaningful when SourceHook is StarshipHttpHook -- direct DB access needs no connection, so this is a no-op on Airflow 2.

starship_connections_migration ¶

starship_connections_migration(connections: List[str] = None, source_http_conn_id: str = None, **kwargs)

TaskGroup to fetch and migrate Connections from one Airflow instance to another.

starship_dag_history_migration ¶

starship_dag_history_migration(dag_ids: List[str] = None, source_http_conn_id: str = None, **kwargs)

TaskGroup to fetch and migrate DAGs with their history from one Airflow instance to another.

starship_pools_migration ¶

starship_pools_migration(pools: List[str] = None, source_http_conn_id: str = None, **kwargs)

TaskGroup to fetch and migrate Pools from one Airflow instance to another.

starship_variables_migration ¶

starship_variables_migration(variables: List[str] = None, source_http_conn_id: str = None, **kwargs)

TaskGroup to fetch and migrate Variables from one Airflow instance to another.