Hi everyone, I will like to pass a date parameter ...
# questions
Hi everyone, I will like to pass a date parameter from the command line when I am executing kedro run, so the catalog paths can point to the specified date. What is the best way to do so? (e.g args with python)
it doesnt work with --params as it seems to initialize the default parameters first and then replacing the specified values
hi @Andreas_Kokolantonakis!
so that the catalog paths can point to the specific date
do you mean, you have several catalogs/catalog directories with a date in the filename, and you want to pick the appropriate one from the
kedro run
hello, correct, lets say that we have a folder with a date (partition) and I want to access only the specified date, e.g ${root_path}/${date}/cars.csv, but for ${date} variable I want to change it every time
from CLI: kedro run 20230627
I don't think there's a very good answer to that at the moment. left a comment in this issue: https://github.com/kedro-org/kedro/issues/1606#issuecomment-1609063931 McK has an internal solution to this called multi-runner, you might want to check it out
(and upvote #1606 top comment to increase visibility of this issue 🙏🏼 )
@Juan Luis what would you suggest if you would like to schedule your kedro code on Airflow and run it daily, accessing data for the specific date (with the option to run historically)
I'm thinking of an alternative using
maybe you could create a custom resolver so that your catalog entries look like this:
Copy code
  type: pandas.CSVDataSet
  filepath: data/01_primary/${today:%Y_%m_%d}/cars.csv
and then in `settings.py`:
Copy code
import datetime as dt
from omegaconf import OmegaConf

def get_today(date_fmt: str) -> str:
  return dt.date.today().strftime(date_fmt)

if not OmegaConf.has_resolver("today"):
  OmegaConf.register_new_resolver("today", get_date)
(I didn't test this, but you get the idea) how does that sound?
yep that was my solution to run it for today’s date, but if my job fails, I want to have the option to run it for previous dates
this maybe could be a good use of environment variables, by levearging the
resolver. it's disabled by default but you can enable it.
that could work! I will give it a try, you either use TemplatedConfigLoader or OmegaConfigLoader right?
it should be the
in this case yes