Sorry, it is me again, lol. Is there any way to configure ParallelRunner's

max_workers

from the command line, in addition to

-r ParallelRunner

? This is essential if you want to avoid loading too much data into RAM or onto the GPU at the same time. From what I've seen, this isn't possible at the moment, and while you could create a runner that executes your pipeline from code, this is probably not want you want as you are losing the rest of the CLI functionality. Also, is it possible to set the project's default runner? I've created this little workaround class, which allows you to set max_workers via an environment variable, but this feels super silly.

Hmmm... it used to be in the run CLI: https://docs.kedro.org/en/stable/getting-started/commands_reference/#kedro-run

But it does not seem to be there right now, is it? Why was it removed? 🙈

I haven’t found any evidence that this was ever part of the CLI. Maybe @Merel or @Rashida Kanchwala might have more context?

I feel like I'm having a Mandela effect moment right now I can't find any evidence it was in the CLI either 😵‍💫 @Bjarne Hiller In the

__main__.py

in your project you can provide a runner to the

run

function and limit the max workers there

I know, right? 😄 All the LLMs also suggest using a

--max-workers

argument, but I didn't find any support for this either. However, that would be nice to have.

I don't think this was ever part of the CLI. But you can indeed set it on the runner yourself. I don't have access to my computer now so can't provide a snippet at this moment .

There's an example of a custom runner in the docs: https://docs.kedro.org/en/stable/build/run_a_pipeline/#custom-runners