Home

Astronomer Orbiter can land legacy workloads safely down in a new home on Apache Airflow!

What is Orbiter?

Orbiter is both a CLI and Framework for converting workflows from other orchestration tools to Apache Airflow.

Generally it can be thoughts of as:

flowchart LR
    origin{{ XML/JSON/YAML/Etc Workflows }}
    origin -->| ✨ Translations ✨ | airflow{{ Apache Airflow Project }}

The framework is a set of Rules and Objects that can translate workflows from an Origin system to an Airflow project.

Installation

Install the orbiter CLI, if you have Python >= 3.10 installed via pip:

pip install astronomer-orbiter

If you do not have a compatible Python environment, pre-built binary executables of the orbiter CLI are available for download on the Releases page.

Translate

Utilize the orbiter CLI with existing translations to convert workflows from other systems to an Airflow project.

1. Set up a new folder, and create a workflow/ folder. Add your workflows files to it

   .
   └── workflow/
       ├── workflow_a.json
       ├── workflow_b.json
       └── ...
   

2. Determine the specific translation ruleset via:
   1. the Origins documentation
   2. the orbiter list-rulesets command
   3. or by creating a translation ruleset, if one does not exist
3. Install the translation ruleset via the orbiter install command (substituting <REPOSITORY> with the value in the last step)

   orbiter install --repo=<REPOSITORY>
   

4. Use the orbiter translate command with the <RULESET> determined in the last step This will produce output to an output/ folder:

   orbiter translate workflow/ --ruleset <RULESET> output/
   

5. Review the contents of the output/ folder. If extensions or customizations are required, review how to extend a translation ruleset
6. (optional) Utilize the astro CLI to run Airflow instance with your migrated workloads
7. (optional) Deploy to Astro to run your translated workflows in production! 🚀

Authoring Rulesets & Customization

Orbiter can be extended to fit specific needs, patterns, or to support additional origins.

Read more specifics about how to use the framework at Rules and Objects

Extend or Customize

To extend or customize an existing ruleset, you can easily modify it with simple Python code.

1. Set up your workspace as described in steps 1+2 of the Translate instructions
2. Create a Python script, named override.py

   .
   ├── override.py
   └── workflow/
       ├── workflow_a.json
       ├── workflow_b.json
       └── ...
   

3. Add contents to override.py:

   from orbiter_community_translations.dag_factory import translation_ruleset  # (1)!
   from orbiter.objects.operators.ssh import OrbiterSSHOperator  # (2)!
   from orbiter.rules import task_rule  # (3)!
   
   
   @task_rule(priority=99)  # (4)!
   def ssh_rule(val: dict):
       """Demonstration of overriding rulesets, by switching DAG Factory BashOperators to SSHOperators"""
       if val.pop("operator", "") == "BashOperator":  # (5)!
           return OrbiterSSHOperator(  # (6)!
               command=val.pop("bash_command"),
               doc="Hello World!",
               **{k: v for k, v in val if k != "dependencies"},
           )
       else:
           return None
   
   
   translation_ruleset.task_ruleset.ruleset.append(ssh_rule)  # (7)!
   

   1. Importing specific translation ruleset, determined via the Origins page
   2. Importing required Objects
   3. Importing required Rule types
   4. Create one or more @rule functions, as required. A higher priority means this rule will be applied first. @task_rule Reference
   5. Rules have an if/else statement - they must always return a single thing or nothing
   6. OrbiterSSHOperator Reference
   7. Append the new Rule to the translation_ruleset

4. Invoke the orbiter CLI, pointing it at your customized ruleset, and writing output to an output/ folder:

   orbiter translate workflow/ output/ --ruleset override.translation_ruleset
   

5. Follow the remaining steps of the Translate instructions

Authoring a new Ruleset

You can utilize the TranslationRuleset Template to create a new TranslationRuleset.

FAQ

• Can this tool convert my workflows from tool X to Airflow?

  If you don't see your tool listed in Supported Origins, contact us for services to create translations, create an issue in the orbiter-community-translations repository, or write a TranslationRuleset and submit a pull request to share your translations with the community.

• Are the results of this tool under any guarantee of correctness?

  No. This tool is provided as-is, with no guarantee of correctness. It is your responsibility to verify the results. We accept Pull Requests to improve parsing, and strive to make the tool easily configurable to handle your specific use-case.

---

Artwork Orbiter logo by Ivan Colic used with permission from The Noun Project under Creative Commons.