https://kedro.org/ logo
Join Slack
Powered by
Hi all, I am developing a namespace pipeline and ...
# questions
a
Hi all, I am developing a namespace pipeline and have come into an issue when using Neptune to track my experiments. When I pass 
"neptune_run"
as an input to a pipeline node I get the following error: 
ValueError: Pipeline input(s) {'NAMESPACE.neptune_run'} not found in the DataCatalog
Where 
"NAMESPACE"
is my namespace pipeline name. Is there a way to use Neptune along with namespace pipelines? Thanks again.
👀 1
n
At which point did you get this error? Did you still have the error if you are not running with neptune. I assume you are using their plugin, if you uninstall it do you still get error?
a
If I set 
enable: false
in 
neptune.yml
I do not get this error.
I can also run the first nodes in my pipeline with 
enable: true
and track the run with Neptune. However, in one of my final nodes I want to upload error metrics. So I pass 
"neptune_run"
as an input to this node. Within the function called at this node I run: 
neptune_run["evaluation/metrics"] = error_metrics
to track the error metrics. This is the point at which I get the 
ValueError
above. Should I implement this in a different way?
Thanks for the response @Nok Lam Chan
n
In that case, I think it’s likely a 
kedro-neptune
issue,. maybe worth asking it in #plugins-integrations. I would also suggest asking in their repository https://github.com/neptune-ai/kedro-neptune since this is a community maintained plugin.
a
Ok will do, thanks @Nok Lam Chan
j
@Andrew Doherty could you share the complete traceback?
a
Of course
Copy code
INFO     Loading data from 'parameters' (MemoryDataSet)...                                                                                                               data_catalog.py:343
╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ /Users/andrew/code │
│ 8 in <module>                                                                                    │
│                                                                                                  │
│ /Users/andrew/code│
│ 3.9/site-packages/kedro/framework/cli/cli.py:211 in main                                         │
│                                                                                                  │
│   208 │   """                                                                                    │
│   209 │   _init_plugins()                                                                        │
│   210 │   cli_collection = KedroCLI(project_path=Path.cwd())                                     │
│ ❱ 211 │   cli_collection()                                                                       │
│   212                                                                                            │
│                                                                                                  │
│ /Users/andrew/code│
│ 3.9/site-packages/click/core.py:1130 in __call__                                                 │
│                                                                                                  │
│ /Users/andrew/code│
│ 3.9/site-packages/kedro/framework/cli/cli.py:139 in main                                         │
│                                                                                                  │
│   136 │   │   )                                                                                  │
│   137 │   │                                                                                      │
│   138 │   │   try:                                                                               │
│ ❱ 139 │   │   │   super().main(                                                                  │
│   140 │   │   │   │   args=args,                                                                 │
│   141 │   │   │   │   prog_name=prog_name,                                                       │
│   142 │   │   │   │   complete_var=complete_var,                                                 │
│                                                                                                  │
│ /Users/andrew/code │
│ 3.9/site-packages/click/core.py:1055 in main                                                     │
│                                                                                                  │
│ /Users/andrew│
│ 3.9/site-packages/click/core.py:1657 in invoke                                                   │
│                                                                                                  │
│ /Users/andrew/code │
│ 3.9/site-packages/click/core.py:1404 in invoke                                                   │
│                                                                                                  │
│ /Users/andrew/code │
│ 3.9/site-packages/click/core.py:760 in invoke                                                    │
│                                                                                                  │
│ /Users/andrew/code │
│ 3.9/site-packages/kedro/framework/cli/project.py:472 in run                                      │
│                                                                                                  │
│   469 │   with KedroSession.create(                                                              │
│   470 │   │   env=env, conf_source=conf_source, extra_params=params                              │
│   471 │   ) as session:                                                                          │
│ ❱ 472 │   │   session.run(                                                                       │
│   473 │   │   │   tags=tag,                                                                      │
│   474 │   │   │   runner=runner(is_async=is_async),                                              │
│   475 │   │   │   node_names=node_names,                                                         │
│                                                                                                  │
│ /Users/andrew/code│
│ 3.9/site-packages/kedro/framework/session/session.py:426 in run                                  │
│                                                                                                  │
│   423 │   │   )                                                                                  │
│   424 │   │                                                                                      │
│   425 │   │   try:                                                                               │
│ ❱ 426 │   │   │   run_result = runner.run(                                                       │
│   427 │   │   │   │   filtered_pipeline, catalog, hook_manager, session_id                       │
│   428 │   │   │   )                                                                              │
│   429 │   │   │   self._run_called = True                                                        │
│                                                                                                  │
│ /Users/andrew/code│
│ 3.9/site-packages/kedro/runner/runner.py:78 in run                                               │
│                                                                                                  │
│    75 │   │                                                                                      │
│    76 │   │   unsatisfied = pipeline.inputs() - set(catalog.list())                              │
│    77 │   │   if unsatisfied:                                                                    │
│ ❱  78 │   │   │   raise ValueError(                                                              │
│    79 │   │   │   │   f"Pipeline input(s) {unsatisfied} not found in the DataCatalog"            │
│    80 │   │   │   )                                                                              │
│    81                                                                                            │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
ValueError: Pipeline input(s) {'dah_market.neptune_run'} not found in the DataCatalog
Due to the namespace adding in the prefix "dah_market" (my namespace) it appears that Kedro then sees this as a dataset rather than the 
neptune_run
object. I wonder if I could pass the 
neptune_run
as a literal to the function using partial. I'll try tomorrow as I need to dig into how this works.
Any thoughts @Juan Luis?
j
no more thoughts for now, haven't looked into this in depth yet but I intend to do so. if you have any further findings drop them here @Andrew Doherty, there's a chance this is an actual bug
a
OK thanks @Juan Luis . Shall I raise this as an issue in 
kedro-neptune
Github?
j
yes, go ahead - the worst that can happen is that it isn't an actual bug 🙂 but will be good to draw their attention. I don't see them in this Slack.
👍 1
a
Hi @Juan Luis, I got a response on the Github issue which doesn't touch on the specific Namespace pipeline challenge I am experiencing - https://github.com/neptune-ai/kedro-neptune/issues/66#issuecomment-1554557741. I think I need to create a basic example to share. Do you have any advice on how to create a simple reproducible Namespace Pipeline for this?
j
@Andrew Doherty thanks for your patience. maybe following the Spaceflights example? https://docs.kedro.org/en/0.18.0/tutorial/namespace_pipelines.html#adding-a-namespace-to-the-data-processing-pipeline you could put that in a repository and try to reproduce the problem. then write in the issue how to do so (
git clone spaceflights-modular && cd spaceflights-modular && kedro run
)
a
Nice @Juan Luis, that's very helpful. I'll work on that.
🙌🏼 1
Hi @Juan Luis. I have added the following comment with a reproducible example. https://github.com/neptune-ai/kedro-neptune/issues/66#issuecomment-1561367556
Let me know your thoughts
j
that looks perfect 💯 thanks @Andrew Doherty! hoping for the maintainers to respond, otherwise we can do some debugging
a
Great, thanks @Juan Luis. If the "neptune_run" object could be imported into the file where the pipeline is defined we could use partial to pass it to the function rather than as a node input here: https://github.com/adoherty21/kedro_neptune_namespace_issue/blob/6abd300ff5980b953[…]/src/basic_namespace/common/pipelines/common_pipeline_01_raw.py If you have any thoughts on how we could do that this would be a neat work around.
s
Hey all, Siddhant here from neptune.ai 👋 Here's an update: https://github.com/neptune-ai/kedro-neptune/issues/66#issuecomment-1564608359 I've asked our devs to have a look, but since the same behavior works fine with normal pipelines, I think we might need support from Kedro's engg. team as well. I'll keep the thread updated. Have a great weekend!
👍 1
j
thanks @Siddhant Sadangi for chiming in!
n
Sure, happy to help if needed
👍 1
🤝 1
a
Hi all, just to confirm that @Siddhant Sadangi provided a workaround which has been tested and works. https://github.com/neptune-ai/kedro-neptune/issues/66 Thanks all for your time and effort.
🎉 2
j
Fantastic news!
we have an issue for Kedro now by the way https://github.com/kedro-org/kedro/issues/2646
👏 1
4 Views
Open in Slack
PreviousNext
Powered by