Environment variables

Starting release 0.20.0, DAG Factory introduces support for referencing environment variables directly within YAML configuration files. This enhancement enables dynamic configuration paths and enhances workflow portability by resolving environment variables during DAG parsing.

With this feature, DAG Factory removes the reliance on hard-coded paths, allowing for more flexible and adaptable configurations that work seamlessly across various environments.

Example YAML Configuration with Environment Variables

Reference environment variable in YAML

example_dag:
  default_args:
    owner: "custom_owner"
  description: "this is an example dag"
  schedule_interval: "0 3 * * *"
  render_template_as_native_obj: True
  dag_display_name: "Pretty Example DAG"
  tasks:
    task_1:
      operator: airflow.operators.bash_operator.BashOperator
      bash_command: "echo 1"
    task_2:
      operator: airflow.operators.bash_operator.BashOperator
      bash_command: "echo 2"
      dependencies: [task_1]
    task_3:
      operator: airflow.operators.python_operator.PythonOperator
      python_callable_name: print_hello
      python_callable_file: $CONFIG_ROOT_DIR/print_hello.py
      dependencies: [task_1]

In the above example, $CONFIG_ROOT_DIR is used to reference an environment variable that points to the root directory of your DAG configurations. During DAG parsing, it will be resolved to the value specified for the CONFIG_ROOT_DIR environment variable.