Hi guys, long time no see! 😊 What would be current recommended way to do fan-out/fan-in processing in Kedro? Let's assume I would like to use either ParallelRunner or ThreadRunner to achieve some level of concurrency in the fan-out part.

Hi @marrrcin, Good to hear back from you. I hope you are doing good. I do not have much experience on concurrent processing apart from what Kedro provides via ParallelRunner or ThreadRunner. I would still recommend using ParallelRunner for this. But I will let someone from the community provide more information. Thank you

@Merel maybe?

Yes,

ParallelRunner

is what you should use to run your nodes in parallel.

Yeah, that I know, but how to do the fan-out Kedro-style?

Hi @marrrcin, what do you mean by fan-out apart from running multiple nodes in parallel ? Do you mean dynamic and increasing number of inputs + nodes ?

Yup