Hi All I am developing a Kedro Namespace pipeline `kedro=0 1 Kedro #questions

Hi All, I am developing a Kedro Namespace pipelin...

Andrew Doherty

07/06/2023, 2:08 PM

Hi All, I am developing a Kedro Namespace pipeline (

kedro=0.18.7

). When running from the CLI I would like to pass the location of the parameters file. From the documentation I believe I should be able to run:

Copy code

kedro run  --pipeline=namespace_1 --config=<config_file_name>.yml

However, when I run this I get

KeyError: 'run'

(traceback in thread). Is there a way to use

--config

with namespace pippelines? The alternative is to create a copy of the

conf

folder and use the following:

kedro run  --pipeline=namespace_1 --conf-source=moved_params

This works but the first option where we only override the single parameter file would be much neater for our purpose. Thanks again.

Andrew Doherty

07/06/2023, 2:08 PM

Copy code

╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ /Users/andrew/code_dev/kedro_namespace_example/kedro_neptune_namespace_issue/.venv/bin/kedro:8   │
│ in <module>                                                                                      │
│                                                                                                  │
│ /Users/andrew/code_dev/kedro_namespace_example/kedro_neptune_namespace_issue/.venv/lib/python3.9 │
│ /site-packages/kedro/framework/cli/cli.py:211 in main                                            │
│                                                                                                  │
│   208 │   """                                                                                    │
│   209 │   _init_plugins()                                                                        │
│   210 │   cli_collection = KedroCLI(project_path=Path.cwd())                                     │
│ ❱ 211 │   cli_collection()                                                                       │
│   212                                                                                            │
│                                                                                                  │
│ /Users/andrew/code_dev/kedro_namespace_example/kedro_neptune_namespace_issue/.venv/lib/python3.9 │
│ /site-packages/click/core.py:1130 in __call__                                                    │
│                                                                                                  │
│ /Users/andrew/code_dev/kedro_namespace_example/kedro_neptune_namespace_issue/.venv/lib/python3.9 │
│ /site-packages/kedro/framework/cli/cli.py:139 in main                                            │
│                                                                                                  │
│   136 │   │   )                                                                                  │
│   137 │   │                                                                                      │
│   138 │   │   try:                                                                               │
│ ❱ 139 │   │   │   super().main(                                                                  │
│   140 │   │   │   │   args=args,                                                                 │
│   141 │   │   │   │   prog_name=prog_name,                                                       │
│   142 │   │   │   │   complete_var=complete_var,                                                 │
│                                                                                                  │
│ /Users/andrew/code_dev/kedro_namespace_example/kedro_neptune_namespace_issue/.venv/lib/python3.9 │
│ /site-packages/click/core.py:1055 in main                                                        │
│                                                                                                  │
│ /Users/andrew/code_dev/kedro_namespace_example/kedro_neptune_namespace_issue/.venv/lib/python3.9 │
│ /site-packages/click/core.py:1655 in invoke                                                      │
│                                                                                                  │
│ /Users/andrew/code_dev/kedro_namespace_example/kedro_neptune_namespace_issue/.venv/lib/python3.9 │
│ /site-packages/click/core.py:920 in make_context                                                 │
│                                                                                                  │
│ /Users/andrew/code_dev/kedro_namespace_example/kedro_neptune_namespace_issue/.venv/lib/python3.9 │
│ /site-packages/click/core.py:1378 in parse_args                                                  │
│                                                                                                  │
│ /Users/andrew/code_dev/kedro_namespace_example/kedro_neptune_namespace_issue/.venv/lib/python3.9 │
│ /site-packages/click/core.py:2360 in handle_parse_result                                         │
│                                                                                                  │
│ /Users/andrew/code_dev/kedro_namespace_example/kedro_neptune_namespace_issue/.venv/lib/python3.9 │
│ /site-packages/click/core.py:2322 in process_value                                               │
│                                                                                                  │
│ /Users/andrew/code_dev/kedro_namespace_example/kedro_neptune_namespace_issue/.venv/lib/python3.9 │
│ /site-packages/kedro/framework/cli/utils.py:414 in _config_file_callback                         │
│                                                                                                  │
│   411 │   section = ctx.info_name                                                                │
│   412 │                                                                                          │
│   413 │   if value:                                                                              │
│ ❱ 414 │   │   config = anyconfig.load(value)[section]                                            │
│   415 │   │   ctx.default_map.update(config)                                                     │
│   416 │                                                                                          │
│   417 │   return value                                                                           │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
KeyError: 'run'

datajoely

07/06/2023, 2:39 PM

the chances are you’re not calling Kedro from the right directory

Andrew Doherty

07/06/2023, 3:14 PM

The following works:

kedro run  --pipeline=namespace_1

Andrew Doherty

07/06/2023, 3:14 PM

The following errors as above:

kedro run  --pipeline=namespace_1 --config=/Users/andrew/code_dev/moved_params/base/parameters/parameters.yml

datajoely

07/06/2023, 3:14 PM

oooh this may be bug

datajoely

07/06/2023, 3:15 PM

would you raise one on GitHub

Andrew Doherty

07/06/2023, 3:15 PM

Yeah, no problem.

Andrew Doherty

07/06/2023, 3:16 PM

I'll post here shortly. There also appears to be another bug if you use

--conf-source=moved_params

along with Neptune.

neptune.yml

and

credentials_neptune.yml

are still required in

conf/base

and are not taken from

--conf-source

Andrew Doherty

07/07/2023, 9:46 AM

@datajoely https://github.com/kedro-org/kedro/issues/2778

Andrew Doherty

07/07/2023, 9:46 AM

Let me know if I can do anything to help.

Andrew Doherty

07/07/2023, 10:02 AM

@datajoely I was misinterpreting the documentation - see response in the issue. Thanks again for your quick response.

datajoely

07/07/2023, 12:53 PM

No worries! Your issue is still super useful for us to see how users are trying to use the tool

datajoely

07/07/2023, 12:53 PM

thank you for the time you put into explaining what you were trying to do

👍 1

4 Views

Open in Slack

Previous Next