Home

Astronomer Orbiter can land legacy workloads safely down in a new home on Apache Airflow!

What is Orbiter?

Orbiter is both a CLI and Framework for converting workflows from other orchestration tools to Apache Airflow.

Generally it can be thoughts of as:

flowchart LR
    origin{{ XML/JSON/YAML/Etc Workflows }}
    origin -->| ✨ Translations ✨ | airflow{{ Apache Airflow Project }}

The framework is a set of Rules and Objects that can translate workflows from an Origin system to an Airflow project.

Installation

Install the orbiter CLI, if you have Python >= 3.10 installed via pip:

pip install astronomer-orbiter

If you do not have a compatible Python environment, pre-built binary executables of the orbiter CLI are available for download on the Releases page.

Translate

Utilize the orbiter CLI with existing translations to convert workflows from other systems to an Airflow project.

1. Set up a new workflow/ folder. Add your workflows files to it

   .
   └── workflow/
       ├── workflow_a.json
       ├── workflow_b.json
       └── ...
   

2. Determine the specific translation ruleset and repository via:
   1. the Origins documentation
   2. the orbiter list-rulesets command
   3. or by creating a translation ruleset, if one does not exist
3. Install the translation ruleset via the orbiter install command
4. Translate workloads via the orbiter translate command
5. Review the contents of the output folder (default: output/). If extensions or customizations are required, review how to extend a translation ruleset
6. (optional) Initialize a full Airflow project containing your migrated workloads with the astro CLI
7. (optional) Deploy to Astro to run your translated workflows in production! 🚀

Authoring Rulesets & Customization

Orbiter can be extended to fit specific needs, patterns, or to support additional origins.

Read more specifics about how to use the framework at Rules and Objects

Extend or Customize

To extend or customize an existing ruleset, you can easily modify it with simple Python code.

1. Set up your workspace as described in steps 1+2 of the Translate instructions
2. Create a Python script, named override.py

   .
   ├── override.py
   └── workflow/
       ├── workflow_a.json
       ├── workflow_b.json
       └── ...
   

3. Add contents to override.py:

   from orbiter_community_translations.dag_factory import translation_ruleset  # (1)!
   from orbiter.objects.operators.ssh import OrbiterSSHOperator  # (2)!
   from orbiter.rules import task_rule  # (3)!
   
   
   @task_rule(priority=99)  # (4)!
   def ssh_rule(val: dict):
       """Demonstration of overriding rulesets, by switching DAG Factory BashOperators to SSHOperators"""
       if val.pop("operator", "") == "BashOperator":  # (5)!
           return OrbiterSSHOperator(  # (6)!
               command=val.pop("bash_command"),
               doc="Hello World!",
               **{k: v for k, v in val if k != "dependencies"},
           )
       else:
           return None
   
   
   translation_ruleset.task_ruleset.ruleset.append(ssh_rule)  # (7)!
   

   1. Importing specific translation ruleset, determined via the Origins page
   2. Importing required Objects
   3. Importing required Rule types
   4. Create one or more @rule functions, as required. A higher priority means this rule will be applied first. @task_rule Reference
   5. Rules have an if/else statement - they must always return a single thing or nothing
   6. OrbiterSSHOperator Reference
   7. Append the new Rule to the translation_ruleset

4. Invoke the orbiter translate command, pointing it at your customized ruleset

   orbiter translate --ruleset override.translation_ruleset
   

5. Follow the remaining steps of the Translate instructions

Authoring a new Ruleset

You can utilize the TranslationRuleset Template to create a new TranslationRuleset.

FAQ

• Can this tool convert my workflows from tool X to Airflow?

  If you don't see your tool listed in Supported Origins, contact us for services to create translations, create an issue in the orbiter-community-translations repository, or write a TranslationRuleset and submit a pull request to share your translations with the community.

• Are the results of this tool under any guarantee of correctness?

  No. This tool is provided as-is, with no guarantee of correctness. It is your responsibility to verify the results. We accept Pull Requests to improve parsing, and strive to make the tool easily configurable to handle your specific use-case.

---

Artwork Orbiter logo by Ivan Colic used with permission from The Noun Project under Creative Commons.