hello, I am using pipelines with namespaces and i ...
# questions
g
hello, I am using pipelines with namespaces and i need to retrieve the value of namespace in the nodes. I have a hook that works fine when i specify the namespace kedro run --namespace xx but when i run all my pipelines without specifying the --namespace then this hook returns None. How to acces the namespace (or full pipeline name
<http://namespace.my|namespace.my>_pipeline
) from inside a node? my current hook:
class NamespaceHook:
namespace = None
@hook_impl
def before_pipeline_run(self, run_params, pipeline, catalog):
NamespaceHook.namespace = run_params.get("namespace")
<http://logger.info|logger.info>(f"Running pipeline with namespace: {NamespaceHook.namespace}")
@staticmethod
def get_namespace():
return NamespaceHook.namespace
👀 1
ok. this can be done by setting parameters for each namespace
r
Hi @Gauthier Pierard, glad you resolved the issue. If you could please share the usecase and what you mean by accessing namespace inside a node, that would help us in improving our APIs. Are you using namespace at the pipeline level or node level in your project and does
node.namespace
does not work for you ?
g
sure. basically I need to pass variables into the save path of a partitioned dataset. The best way i found to do this was to dynamically define this inside my final node:
Copy code
# nodes.py
from kedro_datasets.pandas import ParquetDataset
from kedro_datasets.partitions import PartitionedDataset
from kedro.io import DataCatalog
def myfinalnode(ns = mynamespace, ...): # namespace passed through params:mynamespace
    catalog =  DataCatalog({})
    runid = datetime.now().strftime("%Y-%m-%dT%H-%M")
    dynamic_dataset = PartitionedDataset(
        path=f"abfs://{ns}_{runid}/...}",
        dataset=ParquetDataset,
        credentials=credentials["adls_creds"],
        overwrite = False,
        filename_suffix= f".parquet"
    )
    catalog.add("dynamic_dataset", dynamic_dataset)
    catalog.save("dynamic_dataset", res)
    return res
as you can see I want the namespace to be used in the path of the saved files. in case you see a better way, please let me know
in particular, I would prefer to access and update the existing catalog rather than creating a local empty one, but not sure if that's possible