Hi All I have a catalog entry question Is there a way to dyn Kedro #questions

Hi All. I have a catalog entry question. Is there ...

Zubin Roy

05/21/2024, 3:23 PM

Hi All. I have a catalog entry question. Is there a way to dynamically create your catalog entries based on a parameters? At the moment I use two different environments when I run kedro projects

dev

and

test

where currently I've hardcoded all my file paths. But is there a way that these catalog entries read from a set of parameters which then determine the file paths? Thanks! Present

Copy code

# dev
example_file:
  type: pandas.CSVDataSet
  filepath: <s3://dev_folder_path/example_file.csv>

# test
example_file:
  type: pandas.CSVDataSet
  filepath: <s3://test_folder_path/example_file.csv>

Ideal

Copy code

# dev_param: "<s3://dev_folder_path>"

# dev
example_file:
  type: pandas.CSVDataSet
  filepath: {dev_param}/example_file.csv

# test_param: "<s3://test_folder_path>"

# test
example_file:
  type: pandas.CSVDataSet
  filepath: {test_param}/example_file.csv

Merel

05/21/2024, 3:25 PM

You can use globals for this https://docs.kedro.org/en/stable/configuration/advanced_configuration.html#how-to-use-global-variables-with-the-omegaconfigloader

Merel

05/21/2024, 3:32 PM

Another thing you can do is provide the filepath at runtime depending on which env you're executing: https://docs.kedro.org/en/stable/configuration/parameters.html#how-to-specify-parameters-at-runtime. We're actually about to release new docs about how to combine the globals with the runtime parameters if you do end up needing that.

Zubin Roy

05/21/2024, 4:19 PM

Cool. Thanks @Merel that's helpful!

2 Views

Open in Slack

Previous Next