is it possible to create and run nodes parallel in...
# questions
t
is it possible to create and run nodes parallel in a loop?
h
Someone will reply to you shortly. In the meantime, this might help:
d
if you do
kedro run --runner ParalellRunner
all nodes that can be run concurrently will be
t
So it is not possible to create multiple nodes from a single node in a loop? e.g. an rss scraper node which for each record submits a single not to preprocess it?
d
so you can do that within a node, but you'd have to manage the parallism with something like
joblib
, if you do that you can't use Kedro's node level parallel splitting
t
Okay! Do you know where in the docs I can find how to crerate the nodes dynamically?
m
We don't officially recommend creating nodes or pipelines dynamically, so we don't have any docs about that. Our collaborators at GetInData have written a blog post https://getindata.com/blog/kedro-dynamic-pipelines/, but it's more about dynamically creating pipelines. It might help though.
👍 1