Hi all, I’m looking into Kedro 0.19.2 and I created a project with Spark support. I have just one

spark.SparkDataset

in my catalog and I’m getting this error:

File "/opt/conda/envs/personas/lib/python3.10/site-packages/kedro/io/data_catalog.py", line 83, in _resolve_credentials
    return {k: _map_value(k, v) for k, v in config.items()}
AttributeError: 'str' object has no attribute 'items'

I normally don’t use credentials but in the past this wasn’t an issue. Any ideas what I’m doing wrong?

Which version were you using? And what does your catalog.yml look like?

catalog.yml

transactions:
  type: spark.SparkDataset
  filepath: ...
  file_format: parquet

Would u be able to create a sanitised repo that we can try to reproduce?

I was putting a repo together and noticed that I added a

catalog_globals.yml

as per the docs. If I keep it empty, everything works, if I add a few entries to it, I get the above error.

Would you mind showing how your catalog_globals look like? I could have a look shortly and see if we can get a more explicit error

catalog_globals.yml

looks something like this

exhibitor: abc
my_api_url: my.api.abc
data_bucket: some/s3/location
kedro_s3_bucket: data/science/bucket/kedro-datasets/personas/

and

catalog.yml

dataset1:
  type: personas.extras.datasets.mantle.MantleSparkDataset
  dataset: some.dataset
  exhibitor: ${exhibitor}
  api_url: ${my_api_url}
  tenant_parameter: 'tenant'

dataset2:
  type: personas.extras.datasets.mantle.MantleSparkDataset
  dataset: other.dataset
  exhibitor: ${exhibitor}
  api_url: ${my_api_url}
  tenant_parameter: 'tenant'

Thanks, this is very helpful. I can reproduce the same behavior on my end. Let's see if we can do better to surface the error.