https://kedro.org/ logo
#questions
Title
# questions
d

David Ampudia

01/05/2024, 6:45 PM
Hi Kedro Team, I'm currently working on my first project using Kedro and have a question regarding dynamic path configuration in the DataCatalog. I understand that nodes typically don't have knowledge of IO configuration. My question is: is it still possible to define a subfolder in the DataCatalog path (for a partitioned dataset) by using one of the parameters? I know this can be done by overriding runtime parameters (ie.
"$path/to/folder/{runtime_params:subfolder}"
), but I'm struggling to achieve this with static parameters. Any guidance or suggestions on how to approach this would be really helpful.
Let me be a bit more specific: • Current Setup: I'm using
runtime_params
to create subfolders dynamically in an S3 path for a dataset
downstream_incoming_citations
. The intention is to use the parameter
work_id
as part of the path, which I define at runtime.
downstream_incoming_citations:
type: partitions.PartitionedDataset
path: "s3://.../cited_by/downstream/${runtime_params:work_id, None}/"
Issue: I currently depend on the runtime parameter to do dynamic path creation. Instead, I'd like for it to use parameters in my
conf/base/parameters.yml
, or even better, the output string for the previous node in the pipeline this runs in.
y

Yolan Honoré-Rougé

01/05/2024, 9:20 PM
You can use the "globals" resolver instead of the "runtime_params" in order to inject this path from a static file
👍 1
🙏 1
Using the output string of the previous node is not recommended, but possible with the after_node_run hook I guess
2 Views