hello team I want to add a few new keywords in data catalog Kedro #questions

hello team! I want to add a few new keywords in da...

Youning Xia

07/30/2024, 9:50 AM

hello team! I want to add a few new keywords in data catalog, i.e.

stage

in the following example.

Copy code

companies:
  type: pandas.CSVDataset
  filepath: data/01_raw/companies.csv
  stage: input

reviews:
  type: pandas.CSVDataset
  filepath: data/01_raw/reviews.csv
  stage: output

Is there a way to ask kedro to ignore any new keywords such as

stage

in catalog when running pipelines? If not, how feasible it is to create a kedro plugin that removes/ignores a set of predefined keywords from a data catalog?

datajoely

07/30/2024, 9:53 AM

Copy code

companies:
  type: pandas.CSVDataset
  filepath: data/01_raw/companies.csv
  metadata:
    stage: input

datajoely

07/30/2024, 9:54 AM

move it into the

metadata

key, you can put what you like there and even pick it up in hooks

👍 2

datajoely

07/30/2024, 9:55 AM

that being said, based on what I’m guessing you’re trying to do… we do have a whole concept that is similar to

stage

called

layers

https://docs.kedro.org/projects/kedro-viz/en/stable/kedro-viz_visualisation.html#visualise-layers

Youning Xia

07/30/2024, 9:58 AM

awesome! was not aware of that, thanks for the pointer!

Hamza

07/30/2024, 11:02 AM

Thanks for the assist here @datajoely

🚀 1

Hamza

07/30/2024, 11:56 AM

@datajoely Are there docs for the metadata tag? Couldn't seems to find any docs on it

datajoely

07/30/2024, 11:56 AM

so yes, but you need to go to the kedro-viz section of the docs

datajoely

07/30/2024, 11:57 AM

actually it’s a bit of a mess

datajoely

07/30/2024, 11:57 AM

https://docs.kedro.org/en/stable/hooks/common_use_cases.html#use-hooks-to-read-metadata-from-datacatalog

👍 1

datajoely

07/30/2024, 11:57 AM

it’s here too

Hamza

07/30/2024, 11:57 AM

thankyou

4 Views

Open in Slack

Previous Next