Hello Team I d like to know if I can change the data directo Kedro #questions

Hello Team, I’d like to know if I can change the d...

Jonghyun Yun

03/22/2024, 2:05 PM

Hello Team, I’d like to know if I can change the data directory when I run Kedro. The current data catalog is like

Copy code

companies:
  type: pandas.CSVDataset
  filepath: ${data}/01_raw/companies.csv

reviews:
  type: pandas.CSVDataset
  filepath: ${data}/01_raw/reviews.csv

where

data

is defined in conf/base/globals.yml. Can I pass different

data

value at the run time?

datajoely

03/22/2024, 2:08 PM

here you go! https://docs.kedro.org/en/stable/configuration/advanced_configuration.html#how-to-override-configuration-with-[…]rameters-with-the-omegaconfigloader

👍 1

Jonghyun Yun

03/22/2024, 2:11 PM

Thank you @*datajoely!*

Nok Lam Chan

03/22/2024, 2:16 PM

https://github.com/kedro-org/kedro/issues/3706 Should we prioritise this a bit? I have seen question about mix uses of global and runtime_params every week. Cc @Ankita Katiyar @Merel

💯 1

1000 1

1000000 1

👍 2

plusone 1

Jonghyun Yun

03/22/2024, 10:35 PM

Hi @Nok Lam Chan, if I want

data

be a run-time parameter to change the filepath, where should I place it?

Jonghyun Yun

03/22/2024, 10:40 PM

I changed the catalog as follows, but Kedro didn’t evaluate it correctly, and tried to read a file from its literal path

./{runtime_params:folder, '01_raw'}/reviews.csv

and failed. I’m using Kedro 0.18.14.

Copy code

companies:
  type: pandas.CSVDataset
  filepath: "${runtime_params:folder, '01_raw'}/companies.csv"

reviews:
  type: pandas.CSVDataset
  filepath: "${runtime_params:folder, '01_raw'}/reviews.csv"

Jonghyun Yun

03/23/2024, 4:04 AM

Please disregard the above. It works fine with

OmegaConfigLoader

🚀 1

👍🏼 1

7 Views

Open in Slack

Previous Next