https://kedro.org/ logo
#questions
Title
# questions
w

William Caicedo

01/23/2024, 9:00 PM
Hi all, I’m looking into Kedro 0.19.2 and I created a project with Spark support. I have just one
spark.SparkDataset
in my catalog and I’m getting this error:
Copy code
File "/opt/conda/envs/personas/lib/python3.10/site-packages/kedro/io/data_catalog.py", line 83, in _resolve_credentials
    return {k: _map_value(k, v) for k, v in config.items()}
AttributeError: 'str' object has no attribute 'items'
I normally don’t use credentials but in the past this wasn’t an issue. Any ideas what I’m doing wrong?
n

Nok Lam Chan

01/23/2024, 10:26 PM
Which version were you using? And what does your catalog.yml look like?
w

William Caicedo

01/23/2024, 10:29 PM
catalog.yml
Copy code
transactions:
  type: spark.SparkDataset
  filepath: ...
  file_format: parquet
n

Nok Lam Chan

01/23/2024, 10:37 PM
Would u be able to create a sanitised repo that we can try to reproduce?
w

William Caicedo

01/24/2024, 12:46 AM
I was putting a repo together and noticed that I added a
catalog_globals.yml
as per the docs. If I keep it empty, everything works, if I add a few entries to it, I get the above error.
Seems I missed the new underscore requirement 🤔
Yeah, that seems to be it. I messed up by not using the trailing underscore. It’s a rather obscure error though
n

Nok Lam Chan

01/24/2024, 1:26 PM
Would you mind showing how your catalog_globals look like? I could have a look shortly and see if we can get a more explicit error
w

William Caicedo

01/24/2024, 8:12 PM
catalog_globals.yml
looks something like this
Copy code
exhibitor: abc
my_api_url: my.api.abc
data_bucket: some/s3/location
kedro_s3_bucket: data/science/bucket/kedro-datasets/personas/
and
catalog.yml
Copy code
dataset1:
  type: personas.extras.datasets.mantle.MantleSparkDataset
  dataset: some.dataset
  exhibitor: ${exhibitor}
  api_url: ${my_api_url}
  tenant_parameter: 'tenant'

dataset2:
  type: personas.extras.datasets.mantle.MantleSparkDataset
  dataset: other.dataset
  exhibitor: ${exhibitor}
  api_url: ${my_api_url}
  tenant_parameter: 'tenant'
n

Nok Lam Chan

01/25/2024, 3:04 PM
Thanks, this is very helpful. I can reproduce the same behavior on my end. Let's see if we can do better to surface the error.
FYR, https://github.com/kedro-org/kedro/issues/3555 I created an issue on Github to track this.
👍 1
5 Views