Hi Team, I am experimenting with modular pipelines...
# questions
f
Hi Team, I am experimenting with modular pipelines. The pipeline template takes 5 parameters, out of which 2 vary within the different namespaces. The other 3 parameters are static within in my current use-case, but might require adaptions in future. Therefore, I would like to define the 3 static parameters as constant for pipeline template, without having to re-define with each namespaced pipeline. However, because of the namespaces of each instantiation the pipelines do not find the 3 static parameters without defining them for each namespace. How can I do this in a proper way?
d
In the modular pipeline construction, specify the parameters mapping. For these 3 static parameters, you can pass something like
parameter={'p1': 'p1', 'p2': 'p2', 'p3': 'p3'}
https://kedro.readthedocs.io/en/stable/nodes_and_pipelines/modular_pipelines.html#using-the-modular-pipeline-wrapper-to-provide-overrides
f
That's what i have done so far. But I dislike the fact that i have to declare the 3 static parameters within each instantiation.
d
Ah, sorry, I misunderstood; I thought you were copying them as separate parameters like
namespace1.p1, namespace2.p2, namespace3.p3
. In that case, there's not that much better you can do out of the box. You could define a custom
pipeline_with_static_params()
wrapper or something that takes the arguments to
pipeline()
and the list of static params as a convenience wrapper, but I honestly wouldn't, since it provides limited value (you can always use a dictionary comprehension or something to create the mapping of static parameters).
If this is something you feel like should have better first class support for, or even have an idea in terms of how this could look better, I'd suggest raising an issue on GitHub.