hello team, Working on a kedro project (18.4) with...
# questions
t
hello team, Working on a kedro project (18.4) with omega conf, and when the pipeline is executed hitting the below error,
Copy code
/home/hadoop/miniconda3/envs/mldev/lib/python3.8/site-packages/kedro/framework/cli/project.py:4 │
│ 53 in run                                                                                        │
│                                                                                                  │
│   450 │   with KedroSession.create(                                                              │
│   451 │   │   env=env, conf_source=conf_source, extra_params=params                              │
│   452 │   ) as session:                                                                          │
│ ❱ 453 │   │   session.run(                                                                       │
│   454 │   │   │   tags=tag,                                                                      │
│   455 │   │   │   runner=runner(is_async=is_async),                                              │
│   456 │   │   │   node_names=node_names,                                                         │
│                                                                                                  │
│ /home/hadoop/miniconda3/envs/mldev/lib/python3.8/site-packages/kedro/framework/session/session. │
│ py:418 in run                                                                                    │
│                                                                                                  │
│   415 │   │   │   "runner": getattr(runner, "__name__", str(runner)),                            │
│   416 │   │   }                                                                                  │
│   417 │   │                                                                                      │
│ ❱ 418 │   │   catalog = context._get_catalog(  # noqa: protected-access                          │
│   419 │   │   │   save_version=save_version,                                                     │
│   420 │   │   │   load_versions=load_versions,                                                   │
│   421 │   │   )                                                                                  │
│                                                                                                  │
│ /home/hadoop/miniconda3/envs/mldev/lib/python3.8/site-packages/kedro/framework/context/context. │
│ py:277 in _get_catalog                                                                           │
│                                                                                                  │
│   274 │   │   )                                                                                  │
│   275 │   │   conf_creds = self._get_config_credentials()                                        │
│   276 │   │                                                                                      │
│ ❱ 277 │   │   catalog = settings.DATA_CATALOG_CLASS.from_config(                                 │
│   278 │   │   │   catalog=conf_catalog,                                                          │
│   279 │   │   │   credentials=conf_creds,                                                        │
│   280 │   │   │   load_versions=load_versions,                                                   │
│                                                                                                  │
│ /home/hadoop/miniconda3/envs/mldev/lib/python3.8/site-packages/kedro/io/data_catalog.py:289 in  │
│ from_config                                                                                      │
│                                                                                                  │
│   286 │   │   layers: dict[str, set[str]] = defaultdict(set)                                     │
│   287 │   │                                                                                      │
│   288 │   │   for ds_name, ds_config in catalog.items():                                         │
│ ❱ 289 │   │   │   ds_config = _resolve_credentials(  # noqa: redefined-loop-name                 │
│   290 │   │   │   │   ds_config, credentials                                                     │
│   291 │   │   │   )                                                                              │
│   292 │   │   │   if cls._is_pattern(ds_name):                                                   │
│                                                                                                  │
│ /home/hadoop/miniconda3/envs/mldev/lib/python3.8/site-packages/kedro/io/data_catalog.py:87 in   │
│ _resolve_credentials                                                                             │
│                                                                                                  │
│    84 │   │   │   return {k: _map_value(k, v) for k, v in value.items()}                         │
│    85 │   │   return value                                                                       │
│    86 │                                                                                          │
│ ❱  87 │   return {k: _map_value(k, v) for k, v in config.items()}                                │
│    88                                                                                            │
│    89                                                                                            │
│    90 def _sub_nonword_chars(data_set_name: str) -> str:                                         │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
AttributeError: 'str' object has no attribute 'items'
Any help on why its hitting
👀 1
a
What is the catalog looking like?
t
an example,
Copy code
_data_source.int_pq: &int_pq
  type: spark.SparkDataSet
  layer: int
  file_format: parquet
  save_args:
    mode: overwrite


data_source.int_detail:
  <<: *int_pq
  filepath: ${base_path_spark}/detail

data_source.int_summary:
  <<: *int_pq
  filepath: ${base_path_spark}/summary
n
you probably forget to use
_
somewhere in the catalog
t
but there are a lot of them
n
a
Yep, the interpolation keys should begin with
_
in the catalog
t
aah ok ok got it
thanks both or the quick reply
n
Actually - I am not sure if it works with YAML anchor (
<<
)if you are using OmegaConf, worth double checking because the preferred way to do so with OmegaConf is using interpolation.
t
Thanks for the help, the solution worked really well. and anchoring is working as well 🙂
K 1
👀 1
n
Thanks for reporting! I didn't know that it works too!