.. _managing-sources: Managing sources ================ .. note:: This feature is only available for cosmos >= 1.6.0. By default, Cosmos does not render dbt sources automatically. Instead, you need to configure the rendering of sources explicitly. You can control this behavior using the ``source_rendering_behavior`` field in the ``RenderConfig`` object. This is how it works: - **all**: When set to ``all``, Cosmos renders all sources in the dbt project. It uses three different node types for this: - ``EmptyOperator``: For sources that do not have tests or freshness checks. - ``DbtSourceOperator``: For sources that have freshness checks. - ``DbtTestOperator``: For sources that have tests. This approach aims to create a comprehensive DAG that aligns with dbt documentation, allowing for the rendering of both sources and models for a more detailed visual representation. It also ensures that model dependencies do not run if their sources are not fresh, thus preventing the execution of stale or incomplete data. - **none** (default): When set to ``none``, Cosmos does not automatically render any sources. Note that if node converters are being used for sources, they will still function as intended. - **with_tests_or_freshness**: When set to ``with_tests_or_freshness``, Cosmos only renders sources that have either tests or freshness checks. Sources that have tests but no freshness checks are rendered as ``EmptyOperator``. How the source test task is rendered follows ``test_behavior``: under ``after_each`` and, since Cosmos 1.16.0, under ``build``, source tests run in their own task after the source task, because the build tasks of models, seeds and snapshots select only their own node. Under ``after_all`` they run in the project-level test task, and under ``none`` they are not rendered. Example: .. code-block:: python from cosmos import DbtTaskGroup, RenderConfig from cosmos.constants import SourceRenderingBehavior jaffle_shop = DbtTaskGroup( render_config=RenderConfig( source_rendering_behavior=SourceRenderingBehavior.WITH_TESTS_OR_FRESHNESS, ) ) Source pruning ~~~~~~~~~~~~~~ The ``source_pruning`` is a boolean parameter available in the ``RenderConfig``. When set to ``True``, it automatically removes (or "prunes") any dbt source nodes from your `Apache Airflow® `_ DAG that do not have any downstream dependencies within the selected portion of the dbt graph. This is particularly useful for keeping your DAGs clean and focused, especially in large dbt projects where you might be selecting only a subset of models to run. By default, this option is set to ``False``. Example: .. code-block:: python from cosmos import DbtTaskGroup, RenderConfig jaffle_shop_pruned = DbtTaskGroup( render_config=RenderConfig( select=["+customers"], # select only customers and its parents source_pruning=True, ) ) In this example, if the ``jaffle_shop`` project has multiple sources, but only some of them are upstream of the ``customers`` model, Cosmos will only render the necessary sources and prune the rest. on_warning_callback callback ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The ``on_warning_callback`` is a callback parameter available on the ``DbtSourceLocalOperator``. This callback is triggered when a warning occurs during the execution of the ``dbt source freshness`` command. The callback accepts the task context, which includes additional parameters: test_names and test_results As with test warnings, this callback can be given either as a top-level ``DbtDag`` / ``DbtTaskGroup`` argument or inside ``operator_args``, and the top-level argument wins when both are given. See :ref:`testing-behavior` for details. Example: .. literalinclude:: ../../../dev/dags/example_source_rendering.py :language: python :start-after: [START cosmos_source_node_example] :end-before: [END cosmos_source_node_example]