https://kedro.org/ logo
Join Slack
Powered by
Hi everyone, I will like to pass a date parameter ...
# questions
a
Hi everyone, I will like to pass a date parameter from the command line when I am executing kedro run, so the catalog paths can point to the specified date. What is the best way to do so? (e.g args with python)
it doesnt work with --params as it seems to initialize the default parameters first and then replacing the specified values
j
hi @Andreas_Kokolantonakis!
so that the catalog paths can point to the specific date
do you mean, you have several catalogs/catalog directories with a date in the filename, and you want to pick the appropriate one from the 
kedro run
CLI?
a
hello, correct, lets say that we have a folder with a date (partition) and I want to access only the specified date, e.g ${root_path}/${date}/cars.csv, but for ${date} variable I want to change it every time
from CLI: kedro run 20230627
j
I don't think there's a very good answer to that at the moment. left a comment in this issue: https://github.com/kedro-org/kedro/issues/1606#issuecomment-1609063931 McK has an internal solution to this called multi-runner, you might want to check it out
(and upvote #1606 top comment to increase visibility of this issue 🙏🏼 )
a
@Juan Luis what would you suggest if you would like to schedule your kedro code on Airflow and run it daily, accessing data for the specific date (with the option to run historically)
j
I'm thinking of an alternative using 
OmegaConfigLoader
loading
maybe you could create a custom resolver so that your catalog entries look like this:
Copy code
cars:
  type: pandas.CSVDataSet
  filepath: data/01_primary/${today:%Y_%m_%d}/cars.csv
and then in `settings.py`:
Copy code
import datetime as dt
from omegaconf import OmegaConf

def get_today(date_fmt: str) -> str:
  return dt.date.today().strftime(date_fmt)

if not OmegaConf.has_resolver("today"):
  OmegaConf.register_new_resolver("today", get_date)
(I didn't test this, but you get the idea) how does that sound?
a
yep that was my solution to run it for today’s date, but if my job fails, I want to have the option to run it for previous dates
j
this maybe could be a good use of environment variables, by levearging the 
oc.env
resolver. it's disabled by default but you can enable it.
a
that could work! I will give it a try, you either use TemplatedConfigLoader or OmegaConfigLoader right?
j
it should be the 
OmegaConfigLoader
in this case yes
141 Views
Open in Slack
PreviousNext
Powered by