Marie Cordes
03/06/2025, 9:34 AMnamespace
in a python script the same way I can access e.g., parameters?
For example, I can access parameters in a py
script as such:
from kedro.framework.project import settings
CONFIG_LOADER = settings.CONFIG_LOADER_CLASS(conf_source=settings.CONF_SOURCE, **settings.CONFIG_LOADER_ARGS)
params = CONFIG_LOADER["parameters"]
But how can I access the namespace I pass when running kedro run --pipeline data_preprocessing --namespace example_space
to for example - in the simplest form - be able to log "namespace is set to: example_space"? Thank you!!!Hall
03/06/2025, 9:34 AMdatajoely
03/06/2025, 10:34 AMdatajoely
03/06/2025, 10:36 AMMarie Cordes
03/07/2025, 2:41 PMbefore/after_pipeline_run
hooks in any way?
Some more context:
hooks.py
class PipelineStatusHookNew:
def __init__(self):
logger.info("PipelineStatusHookNew instantiated.")
logger.info(f"Class instance: {self}")
...
@hook_impl
def before_pipeline_run(self, run_params, **kwargs):
logger.info("before_pipeline_run triggered with run_params: %s", run_params)
self.start_time = datetime.now(tz=timezone.utc)
logger.info("Logged run start time for state management table update.")
...
CLI
You can see that that the def __init__(self):
logs are being displayed but then the pipeline and nodes start running without triggering the before_pipeline_run
method. And I've tried before/after pipeline/node etc. - logging things but also writing to an output file but nothing's happening.
kedro run --pipeline sample_testing_pipe --namespace industry
[03/07/25 15:33:04] INFO Using 'conf/logging.yml' as logging configuration. You can change this by setting the KEDRO_LOGGING_CONFIG __init__.py:270
environment variable accordingly.
2025-03-07 15:33:19,052 - hook_registry.hooks - INFO - PipelineStatusHookNew instantiated.
[03/07/25 15:33:19] INFO PipelineStatusHookNew instantiated. hooks.py:16
2025-03-07 15:33:19,055 - hook_registry.hooks - INFO - Class instance: <hook_registry.hooks.PipelineStatusHookNew object at 0x162526620>
INFO Class instance: <hook_registry.hooks.PipelineStatusHookNew object at 0x162526620> hooks.py:17
WARNING /Users/Marie_Cordes/.pyenv/versions/3.10.9/lib/python3.10/site-packages/kedro_viz/integrations/kedro/hooks.py:13: warnings.py:109
KedroDeprecationWarning: 'TRANSCODING_SEPARATOR' has been moved to 'kedro.pipeline.transcoding', and the alias
will be removed in Kedro 0.20.0
from kedro.pipeline.pipeline import TRANSCODING_SEPARATOR, _strip_transcoding
INFO Kedro project classifications-v2 session.py:329
INFO Using synchronous mode for loading and saving data. Use the --async flag for potential performance gains. sequential_runner.py:74
<https://docs.kedro.org/en/stable/nodes_and_pipelines/run_a_pipeline.html#load-and-save-asynchronously>
INFO Loading data from model_config_dev (SnowparkTableDataset)... data_catalog.py:390
[03/07/25 15:33:20] INFO Loading data from params:industry.model_configs (MemoryDataset)... data_catalog.py:390
INFO Running node: fetch_model_configs_from_snowflake_node: node.py:367
fetch_model_configs([model_config_dev;params:industry.model_configs]) -> [industry.classifier_profiles]
...
<< proceeds to run the pipeline >>
src/classifications_v2/settings.py
from hook_registry.hooks import PipelineStatusHookNew
HOOKS = (PipelineStatusHookNew(),)
datajoely
03/07/2025, 3:04 PMdatajoely
03/07/2025, 3:04 PMbreakpoint()
in the hook and make sure it actually gets called