Hi everyone is it possible to pass credentials of the underl Kedro #questions

Hi everyone, is it possible to pass credentials of...

Lim H.

07/26/2023, 6:51 PM

Hi everyone, is it possible to pass credentials of the underlying dataset when using it with CachedDataSet? e.g.

Copy code

test:
  type: CachedDataset
  versioned: true
  dataset:
    type: pandas.JSONDataSet
    filepath: ...
    load_args:
      lines: True
    credentials: ...

doesn’t work but this works

Copy code

test:
  type: pandas.JSONDataSet
  filepath: ...
  load_args:
    lines: True
  credentials: ...

I thought this was working at some point? I might be hallucinating though. Just want to double check quickly before I create my own CachedDataSet

✅ 1

👀 1

Lim H.

07/26/2023, 7:33 PM

Sorry my bad. I included versioned: true and that tried to glob through the directory which my credentials didn’t have access to

👍🏼 1

Nok Lam Chan

07/26/2023, 9:04 PM

Actually I don’t know if

versioned

work with any of these wrapper dataset like

PartitionedDataSet

Lim H.

07/26/2023, 9:17 PM

The official doc includes versioned: true there for CachedDataSet

👀 1

Nok Lam Chan

07/27/2023, 4:34 PM

Just to clarify it is only a permission issue, versioning still works right? or is this a doc issue.

Lim H.

07/27/2023, 4:36 PM

• My issue was a permission issue. I turned off versioning and it works. I haven’t tested versioning yet. • My comment about versioning was in the doc was because you mentioned

don’t know if
versioned
work with any of these wrapper dataset like
PartitionedDataSet

If that’s the case, we should probably remove that from the doc for CachedDataSet. I haven’t verified that claim though

👍🏼 1

Nok Lam Chan

07/27/2023, 4:41 PM

Cool, just to report back • CacheDataset

versioned

flag does work •

versioned

is not supported for

PartitionedDataset

🆒 1

Open in Slack

Previous Next