Hey team, I am new to Kedro and I want to know if ...
# questions
Hey team, I am new to Kedro and I want to know if it is possible to schedule a pipeline to run every day (with a different
configuration YAML
files) ?
I mean the pipeline should be called everyday with a configuration file including the current date
Hi @Ayoub Chouikha, Kedro itself is not an orchestrator, so if you want to schedule your runs you'd have to use something like Airflow to set up the schedule.
Thank you @Merel for your answer. Yes I will be using Databricks and AirFlow, but I am actually worrying about the dynamic configuration + catalog in each run
You could also use different run environments to divide your config: https://docs.kedro.org/en/stable/configuration/configuration_basics.html#configuration-environments and then for the run commands you can do
kedro run --env=setup1
kedro run --env=setup2
etc. Assuming you want the pipeline to be the same every time and just vary the inputs/outpus.
🥳 1
Or you could leverage namespaces https://docs.kedro.org/en/latest/nodes_and_pipelines/namespaces.html (we've just rewritten the docs)
👍 2
This seems very helpful, thanks a lot @Merel 🙌
You're welcome!
❤️ 1