Hi everyone, I want to use a somewhat dynamic dataset. I am using an internal base package that has a custom dataset with optional partitioning. The table contains the historic data up until the current date. I want to be able to set the partition value to any date in the past (including current date) to run the pipeline on the historic data of that date only. Preferably as a parameter. My catalog entry looks like this:

a_catalog_entry:
  type: internal.package.with.custom.Dataset
  database: MY_DB
  schema: MY_SCHEMA
  name: MY_TABLE
  load_args:
    partition_key: DT
    partition_value:
      - 20260924

The custom Dataset translates it into:

SELECT *
FROM MY_DB.MY_SCHEMA.MY_TABLE
WHERE <PARTITION_KEY> IN (<PARTITION_VALUE_1>, <PARTITION_VALUE_2>, ...)

1. What is the best way (most kedro like way) to do this? ◦ hooks, ... 2. Is using the partition the best way at all? Is there a better way to filter the data for the date the pipeline is running on? Thank you

Hey Jessica, you can do this with runtime params and catalog templating. Kedro's catalog supports OmegaConf's

${runtime_params:...}

resolver, so you can reference a runtime parameter directly in your catalog entry:

a_catalog_entry:
    type: internal.package.with.custom.Dataset
    database: MY_DB
    schema: MY_SCHEMA
    name: MY_TABLE
    load_args:
      partition_key: DT
      partition_value:
        - "${runtime_params:partition_date}"

If you need multiple partition values, you could pass a comma-separated string and split it inside your node or dataset, or define several params such as (partition_date_1, partition_date_2). Let me know if this helps!

Yes thanks that helped!

1. What is the best way (most kedro like way) to do this?

◦ hooks, ...

2. Is using the partition the best way at all? Is there a better way to filter the data for the date the pipeline is running on?

@jessica.schulze Glad you have a solution! To provide an alternative, if it were me, I'd use

ibis.TableDataset

. Rather than filtering in the catalog entry, you can filter in the node itself. Since Ibis provides deferred SQL execution, the whole table is never loaded, even though you provide a reference to an unfiltered table in the catalog. Number of resources out there on doing this, e.g. kedro.org/blog/building-scalable-data-pipelines-with-kedro-and-ibis