Kedro is an open-sourced Python framework for creating maintainable and modular data science code.

Kedro

Hey team, I am new to Kedro and I want to know if it is possible to schedule a pipeline to run every day (with a different `configuration YAML` files) ?

I mean the pipeline should be called everyday with a configuration file including the current date

Hi <@U0796U50KFV>, Kedro itself is not an orchestrator, so if you want to schedule your runs you'd have to use something like Airflow to set up the schedule.

Thank you <@U03S12METRA> for your answer. Yes I will be using Databricks and AirFlow, but I am actually worrying about the dynamic configuration + catalog in each run

You could also use different run environments to divide your config: <https://docs.kedro.org/en/stable/configuration/configuration_basics.html#configuration-environments> and then for the run commands you can do `kedro run --env=setup1` , `kedro run --env=setup2` etc. Assuming you want the pipeline to be the same every time and just vary the inputs/outpus.

Or you could leverage namespaces <https://docs.kedro.org/en/latest/nodes_and_pipelines/namespaces.html> (we've just rewritten the docs)

This seems very helpful, thanks a lot <@U03S12METRA> :raised_hands: