Heyy guys I m trying to avoid doing some ugly non kedronic s Kedro #questions

Heyy guys! I'm trying to avoid doing some ugly, no...

Wojciech Szenic

02/10/2023, 6:37 AM

Heyy guys! I'm trying to avoid doing some ugly, non kedronic solutions so perhaps You could help me with my problem. I would like for kedro to take in a command line arguments such as date or country and then do processing based on these arguments. So for example, I have a trained machine learning model, and a

predict

pipeline can output predictions. Ideally, this predict pipeline can be run as

kedro run --pipeline=predict --date=2023-01-05

and this would ingest the dataset for 05th of Jan 2023 and run the prediction on it. I'm wondering how can I pass the CLI argument into the dataset catalog?

✅ 1

Nok Lam Chan

02/10/2023, 7:10 AM

I think this could be achieved with

TemplatedConfigLoader

where you specify a template variable in your

catalog.yml

and override the value via CLI. See an example here: https://kedro.readthedocs.io/en/stable/kedro.config.TemplatedConfigLoader.html

👍 1

Wojciech Szenic

02/10/2023, 7:24 AM

Thanks a lot, I'll try it out 🙂

Wojciech Szenic

02/10/2023, 8:01 AM

Ok, so I did manage to insert parameter from

globals.yml

into a query but I cannot override it using CLI, I'm trying to do it using

kedro run --pipeline=predict --params table:correct_name

when the

table

is a wrong name in the globals file. Is there another step which needs to be done to overwrite global variables?

Wojciech Szenic

02/10/2023, 9:17 AM

I manged to partially do it using the VersionedDataset but it doesn't seem to be very clean, is there a perhaps better way to do it? It seems that versioned dataset forces a certain pattern in the file path which might be problematic for me

Nok Lam Chan

02/10/2023, 9:24 AM

https://github.com/kedro-org/kedro/discussions/1782 Can you try this slight variation of TemplatesConfigaloader instead?

Wojciech Szenic

02/10/2023, 9:37 AM

It works! Thanks a lot @Nok Lam Chan thankyou

😁 1

✅ 1

2 Views

Open in Slack

Previous Next