Hi team, We have a datalake and I would like to download images from there to my local place. So I use a function and put credentials as parameters into the function. I also put it when I create the node. But when I run it, it shows that I need to setup something in my catalog. Can I know how should I do it?

datalake:
  type: partitions.PartitionedDataset
  filepath: <s3://datalake>
  credentials: datalake_credentials
  dataset:
    type: pillow.ImageDataset

I put above into catalog, but I still got error message.

ValueError: Pipeline input(s) {'params:datalake_credentials.PROJECTKEY', 'params:datalake_credentials.USER', 'params:datalake_credentials.PASSWORD'} not found in the DataCatalog

on the other hand, I got this message as well:

Dataset 'datalake' has an invalid type 'partitions.PartitionedDataset'. "Unable to find credentials 'datalake_credentials': check your data catalog and credentials configuration. See <https://docs.kedro.org/en/stable/api/kedro.io.DataCatalog.html> for an example."

Someone will reply to you shortly. In the meantime, this might help:

so this isn't how credentials work in kedro there are a couple of ways of doing it, we have them documented here: https://docs.kedro.org/en/stable/configuration/credentials.html ^ I think this is all you need https://docs.kedro.org/en/stable/data/how_to_create_a_custom_dataset.html#how-to-handle-credentials-and-different-filesystems https://docs.kedro.org/en/stable/hooks/common_use_cases.html#use-hooks-to-load-external-credentials

Follow to similar question. How could I load mlflow parameters that I put in mlflow.yml file? Using the same method is not working for me. should I use

os.getenv

?

those aren't seen by kedro unless you pull them in, possibly a hook would work

I don't understand. How should I do exactly? 😟