Hi, just an idea: Sometimes it would be nice to be...
# questions
b
Hi, just an idea: Sometimes it would be nice to be able to pass some function arguments directly when creating a kedro Node. This is especially useful for dynamic pipeline creation. For example, I want to create a preprocessing pipeline for each dataset. I have a function
create_data_pipeline(dataset)
, that downloads and processes the respective dataset via a function
make_data(dataset)
. Right now, I guess I could go for
partial(make_data, dataset)
, register dataset as param in the catalog, or create a separate download function for each dataset. All of these feel a bit silly/convoluted to me, since I already have the
dataset
value right there, in my pipeline creation function. Ideally, I would like to create something like
Node(make_data, python_inputs=dataset)
, i.e., inputs that are not looked up in the catalog. What do you think?
👍 1
👀 1
l
Agreed, it's rather frustrating!
r
Hi @Bjarne Hiller and @Laurens Vijnck, Thank you for the suggestions and letting us know about the not-so-great experience. This will help us improve the package. There has been an old discussion around this - https://github.com/kedro-org/kedro/discussions/1279 We will try to get this in our grooming sessions and spike on the possibility of having a way to pass node inputs. Thank you cc: @Dmitry Sorokin
👍 2
Created a spike ticket to tackle this in coming sprints - https://github.com/kedro-org/kedro/issues/5455 Please add any other information/use-case you would like us to consider @Bjarne Hiller @Laurens Vijnck Thanks for your patience. cc: @Alice Cima
👍 2
thankyou 1
d
As an FYI, this has been discussed several times before (e.g. in #213 and #526). Added a comment to the issue accordingly.
👍 1