Hi Team , can we make dataset load to a kedro node parallel ? I have 19 datasets that i need to read in a single function but when i see the logs it taking 2 minutes per dataset i.e. ~40 minutes of read time , any way we can make the read parallel ? I dont think they are conflicting in any way so i dont see why it needs to be sequential ?

please don’t tag us directly - someone will answer you. What kind of dataset?

This is a custom dataset we have created

well then Kedro isn’t mandating sequential reads?

this is a dataset agnostic question , i have 19 catalog entries , how can i make sure that they are being loaded at the same time ? im assuming by default it loads step by step ?

oh as in 19 dataset loads to the same node

indeed

that is a simple loop and if you want to implement your own runner you can

its loads like this

00:02 INFO Loading data from xxxxx
00:04 INFO Loading data from yyyyy

oh sorry

aaah lemme check this

Nice ! I was going to ask if this is documented @datajoely but it is, and I'd completely forgotten about it 😆 I think a blog post about optimising Kedro usage, with this kind of tip, could be a good one :)

Fully agree! I would encourage a separate documentation for this too , this is much broader than just "running a pipeline"

A question for @Merel is there an argument to make this default behaviour?

Not that I’m aware of!

as in its been ruled out?

Oh haha I thought you meant argument as in argument to the CLI 😂

what’s the best way to do so? An issue?

calls for a user feature vote ? 😛

Issue sounds good! You can mark it as “technical design” so we’ll discuss with the team

LOL, if you add it as default I won't have the makings of an infomercial blog post any more 😆

@datajoely do you want me to make the issue about this ? I have screenshot et al and can put it up

I’m doing it

Awesome, the only push back i can expect is with the

ThreadRunner

as async calls doesnt work with it , but lets see

I’m proposing a fallback