https://kedro.org/ logo
Join Slack
Powered by
Hi everyone. Is there a way to pass *args as a par...
# questions
j
Hi everyone. Is there a way to pass *args as a parameter in a pipeline so that I can re-use a function with different number of dataframes?
Copy code
def simulation(
    weighted: bool,
    k: int,
    num_tries: int,
    mapping: Dict[str, str],
    *dataframes: pl.DataFrame
) -> pl.DataFrame:

node(
     func = simulation,            
     inputs = [
        "params:weighted",
        "params:k",
        "params:number_of_tries",
        "params:mapping",
        "*dataframes"   <------- how would this be defined?
    ],
👍 1
d
This is nearly merged, but not in yet https://github.com/kedro-org/kedro/pull/3905
👍🏾 1
j
Excellent. Good to know its possible.
n
What is *dataframes in the node definition in this example?
You can already pass different number of dataset to the node function.
j
Essentially *args as I want to be able to pass different numbers of dataframes to the function.
I want to re-use the function.
n
Have you tried it? There is nothing preventing it and it should work
For example, with a function:
Copy code
def foo(*args):
    ...
Your node can take
Copy code
node(func=foo, inputs=["data_a","data_b","data_c"],...)
or
Copy code
node(func=foo, inputs=["data_a"], ...)
j
so nodes can accept both positional and arbitrary arguments within the same input?
n
It's not arbitrary. We don't exactly map Python Function to Node's input, but they share a lot of similarity. For example, if you have a function
Copy code
def foo(a,b, *args):
   ...
it's valid to pass in foo(1,2,3,4,5,6), where 3,4,5,6 will passed to 
*args
So we are doing something similiar, dataset are being loaded and pass to the underlying node function.
Copy code
return self._func(*(inputs[item] for item in node_inputs))
so nodes can accept both positional and arbitrary arguments
In fact the most common case of 
node
is using the positional arguments, https://docs.kedro.org/en/stable/nodes_and_pipelines/nodes.html#syntax-for-input-variables I see that our docs didn't explicit mention 
*args
, but it has been possible for years. I am opening a PR to add this now
👍🏾 1
2 Views
Open in Slack
PreviousNext
Powered by