Gauthier Pierard
03/28/2025, 2:37 PM<http://namespace.my|namespace.my>_pipeline
) from inside a node?
my current hook:
class NamespaceHook:
namespace = None
@hook_impl
def before_pipeline_run(self, run_params, pipeline, catalog):
NamespaceHook.namespace = run_params.get("namespace")
<http://logger.info|logger.info>(f"Running pipeline with namespace: {NamespaceHook.namespace}")
@staticmethod
def get_namespace():
return NamespaceHook.namespace
Gauthier Pierard
03/28/2025, 3:42 PMRavi Kumar Pilla
03/28/2025, 4:07 PMnode.namespace
does not work for you ?Gauthier Pierard
03/28/2025, 4:14 PM# nodes.py
from kedro_datasets.pandas import ParquetDataset
from kedro_datasets.partitions import PartitionedDataset
from kedro.io import DataCatalog
def myfinalnode(ns = mynamespace, ...): # namespace passed through params:mynamespace
catalog = DataCatalog({})
runid = datetime.now().strftime("%Y-%m-%dT%H-%M")
dynamic_dataset = PartitionedDataset(
path=f"abfs://{ns}_{runid}/...}",
dataset=ParquetDataset,
credentials=credentials["adls_creds"],
overwrite = False,
filename_suffix= f".parquet"
)
catalog.add("dynamic_dataset", dynamic_dataset)
catalog.save("dynamic_dataset", res)
return res
as you can see I want the namespace to be used in the path of the saved files. in case you see a better way, please let me knowGauthier Pierard
03/28/2025, 4:18 PM