https://kedro.org/ logo
Powered by
#questions
Title
# questions
a

Andreas_Kokolantonakis

06/27/2023, 8:06 AM
Hi everyone, I will like to pass a date parameter from the command line when I am executing kedro run, so the catalog paths can point to the specified date. What is the best way to do so? (e.g args with python)
it doesnt work with --params as it seems to initialize the default parameters first and then replacing the specified values
j

Juan Luis

06/27/2023, 8:31 AM
hi @Andreas_Kokolantonakis!
so that the catalog paths can point to the specific date
do you mean, you have several catalogs/catalog directories with a date in the filename, and you want to pick the appropriate one from the 
kedro run
CLI?
a

Andreas_Kokolantonakis

06/27/2023, 8:41 AM
hello, correct, lets say that we have a folder with a date (partition) and I want to access only the specified date, e.g ${root_path}/${date}/cars.csv, but for ${date} variable I want to change it every time
from CLI: kedro run 20230627
j

Juan Luis

06/27/2023, 8:50 AM
I don't think there's a very good answer to that at the moment. left a comment in this issue: https://github.com/kedro-org/kedro/issues/1606#issuecomment-1609063931 McK has an internal solution to this called multi-runner, you might want to check it out
(and upvote #1606 top comment to increase visibility of this issue 🙏🏼 )
a

Andreas_Kokolantonakis

06/27/2023, 8:55 AM
@Juan Luis what would you suggest if you would like to schedule your kedro code on Airflow and run it daily, accessing data for the specific date (with the option to run historically)
j

Juan Luis

06/27/2023, 8:57 AM
I'm thinking of an alternative using 
OmegaConfigLoader
loading
maybe you could create a custom resolver so that your catalog entries look like this:
Copy code
cars:
  type: pandas.CSVDataSet
  filepath: data/01_primary/${today:%Y_%m_%d}/cars.csv
and then in `settings.py`:
Copy code
import datetime as dt
from omegaconf import OmegaConf

def get_today(date_fmt: str) -> str:
  return dt.date.today().strftime(date_fmt)

if not OmegaConf.has_resolver("today"):
  OmegaConf.register_new_resolver("today", get_date)
(I didn't test this, but you get the idea) how does that sound?
a

Andreas_Kokolantonakis

06/27/2023, 9:06 AM
yep that was my solution to run it for today’s date, but if my job fails, I want to have the option to run it for previous dates
j

Juan Luis

06/27/2023, 9:17 AM
this maybe could be a good use of environment variables, by levearging the 
oc.env
resolver. it's disabled by default but you can enable it.
a

Andreas_Kokolantonakis

06/27/2023, 9:40 AM
that could work! I will give it a try, you either use TemplatedConfigLoader or OmegaConfigLoader right?
j

Juan Luis

06/27/2023, 9:53 AM
it should be the 
OmegaConfigLoader
in this case yes
19 Views
Powered by