Hello everyone !
I have some question regarding kedro pipeline for forecasting solution.
is there a way to run kedro sub-pipeline inside a loop so that it generates the forecast.
catch output of one iteration is input for other iteration.
Please let me know.
d
Deepyaman Datta
03/08/2023, 6:49 PM
Not really, since that would violate the requirement that the Kedro pipeline is a DAG.
I'd perhaps use an orchestrator (e.g. Prefect, Airflow, Kubeflow Pipelines, whatever) to orchestrate modular subpipelines if want looping functionality.
s
Suryansh Soni
03/08/2023, 11:37 PM
inst there any way in kedro to do that ?
d
Deepyaman Datta
03/08/2023, 11:44 PM
Not really? A Kedro pipeline must be a DAG. You can dynamically construct unrolled loops, but it's not really a supported use case/isn't ideal for many reasons (but you can see https://github.com/kedro-org/kedro/issues/1993 and other related issues).
s
Sebastian Pehle
03/09/2023, 9:19 AM
Would it be possible to not loop the node but to put the loop inside the node?
s
Suryansh Soni
03/15/2023, 4:29 PM
that is doable. but i wont be able to visualize the process