Hi Kedro Team I m currently working on my first project usin Kedro #questions

Hi Kedro Team, I'm currently working on my first p...

David Ampudia

01/05/2024, 6:45 PM

Hi Kedro Team, I'm currently working on my first project using Kedro and have a question regarding dynamic path configuration in the DataCatalog. I understand that nodes typically don't have knowledge of IO configuration. My question is: is it still possible to define a subfolder in the DataCatalog path (for a partitioned dataset) by using one of the parameters? I know this can be done by overriding runtime parameters (ie.

"$path/to/folder/{runtime_params:subfolder}"

), but I'm struggling to achieve this with static parameters. Any guidance or suggestions on how to approach this would be really helpful.

David Ampudia

01/05/2024, 8:18 PM

Let me be a bit more specific: • Current Setup: I'm using

runtime_params

to create subfolders dynamically in an S3 path for a dataset

downstream_incoming_citations

. The intention is to use the parameter

work_id

as part of the path, which I define at runtime.

downstream_incoming_citations:

type: partitions.PartitionedDataset

path: "s3://.../cited_by/downstream/${runtime_params:work_id, None}/"

• Issue: I currently depend on the runtime parameter to do dynamic path creation. Instead, I'd like for it to use parameters in my

conf/base/parameters.yml

, or even better, the output string for the previous node in the pipeline this runs in.

Yolan Honoré-Rougé

01/05/2024, 9:20 PM

You can use the "globals" resolver instead of the "runtime_params" in order to inject this path from a static file

👍 1

🙏 1

Yolan Honoré-Rougé

01/05/2024, 9:22 PM

Using the output string of the previous node is not recommended, but possible with the after_node_run hook I guess

2 Views

Open in Slack

Previous Next