Marc Gris
07/18/2023, 4:47 AMMichel van den Berg
07/18/2023, 5:49 AMMarc Gris
07/18/2023, 6:56 AMNok Lam Chan
07/18/2023, 10:18 AMconf/base/parameters.yml
, while leaving the parameters for specific scope in conf/base/parameters/some_pipeline.yml
Marc Gris
07/18/2023, 10:21 AMNok Lam Chan
07/18/2023, 10:41 AMCONFIG_LOADER_ARGS = {
"config_patterns": {
"parameters": ["params*", "params*/**", "**/params*"],
}
}
https://docs.kedro.org/en/latest/configuration/advanced_configuration.html#how-to-change-which-configuration-files-are-loadedMarc Gris
07/18/2023, 10:42 AMconf/base/parameters.yml
I have a top-level key call schema
passing inputs="params:schema"
to a node does work without applying a namespace to the pipeline. But if a namespace is applied, I get the following error:
ValueError: Pipeline input(s) {'params:<namespace>.schema'} not found in the DataCatalog
Nok Lam Chan
07/18/2023, 10:43 AMMarc Gris
07/18/2023, 10:44 AMdef create_pipeline(**kwargs) -> Pipeline:
dummy_node = node(
func=lambda x: print(x),
inputs="params:schema",
outputs=None,
)
return pipeline([ dummy_node], namespace="test")
N.B : ConfigLoader wise, we use OmegaConfigLoader
Nok Lam Chan
07/18/2023, 11:42 AMMarc Gris
07/18/2023, 11:59 AMNok Lam Chan
07/18/2023, 12:43 PMnamespace
so it automatically insert the test.
prefix in front of your inputs/outputs.
https://docs.kedro.org/en/stable/nodes_and_pipelines/modular_pipelines.htmlMarc Gris
07/18/2023, 12:52 PM<namespace-1>.params
, namespace-2.params
etc… especially in the the case where those params are left identical / unchanged across all namespaces.
Do you see what I mean ? 🙂Nok Lam Chan
07/18/2023, 12:58 PMMarc Gris
07/18/2023, 12:59 PMNok Lam Chan
07/18/2023, 12:59 PMMarc Gris
07/18/2023, 1:03 PMtest.schema: ${schema}
in conf/parameters/modular_pipeline_1.yml
Nok Lam Chan
07/18/2023, 1:14 PMnamespace
argument, everything in your pipeline is now prefixed. If you need to escape the namespace, you have to pass the input/output/parameters specificly.
The reason that you are getting this error because you only have params:schema
,
ValueError: Pipeline input(s) {'params:<namespace>.schema'} not found in the DataCatalog
To use this with different parameters , you can do
pipeline(pipe=some_pipeline, parameters = {"params:schema" : "params:my_parameters"}, namesapce="<namespace>")
It may not be immediately obvious, but there is nothing stopping you to pass the same params:schema
parameters so it can escape the namespace (Or the global you are referring to)
pipeline(pipe=some_pipeline, parameters = {"params:schema" : "params:schema"}, namesapce="<namespace>")
Marc Gris
07/18/2023, 2:56 PMparameters
parameter for modular / namespaced pipelines.
I will ponder onto this and try things out on my side before getting back to you.
Thanks (a lot) again 🙏🏼
M."{namespace}.data":
type: ….
filepath: …
Is there anything equivalent for parameters ? 🙂
(I’ve tried… and of course.. failed 😅 )
ThxNok Lam Chan
07/18/2023, 3:24 PMcatalog.yml
specifically, what are your use cases? Maybe you can just give me pseudocode even if this feature doesn’t exist.Marc Gris
07/18/2023, 3:47 PM"{namespace}.invariant_param" : …
combined with multiple namespace_*1*.changing_param:...
, namespace_*2*.changing_param: ...
As soon as things get clearer, I’ll get back to you if needed 👍🏼 🙂Nok Lam Chan
07/18/2023, 4:39 PMMarc Gris
07/18/2023, 4:46 PMcatalog.yml
& parameters.yml
is a desirable thing 🙂Nok Lam Chan
07/19/2023, 12:15 PMparams
was not namespace pre- 0.18, but it was changed, the doc should reflect this.
https://github.com/kedro-org/kedro/pull/2839Marc Gris
07/31/2023, 3:31 PMJuan Luis
09/27/2023, 9:31 AM