Hi guys! I'm a bit new to the data engineering that kedro is bulding on, have limited experience in database management, and have a use-case that I'm struggling to find any straight-forward solution to. My pipeline is in essence creating datasets of DNA-sequences, which I send to an API (AlphaGenome) for property predictions, and then preform data analysis on. All of these sequence and predictions also needs to be stored in a global database for later use. The quirk of this workflow is that the API is enforces rate limiting, which not only motivates the need to only request predictions for new sequences (as predictions take a lot of time to produce), but also to have a failsafe if the API terminates the request, so that I can at least store the predictions I've received before that point. Additionally, the amount of sequences are substantial and this creates strict requirements on efficency in memory and speed. The consequences of this is that I always first need to check what's in the database before updating it, creating a circular dependency. I also need some sort of batch logic for updating my database to store progress. And because of this being a big database I can't really have any version logic that duplicates data. I know that one can create workaround by having two different dataset share the same dataset path but I have a feeling that there might be other design practices that are more aligned with kedros. Either way, I'm guessing that there at least exist a lot of similar use-cases that perform database updating who might have solutions to this. Thankful for any help, and if you have additional recommendation on good kedro-datasets for this, even better! :)

Hi @Marcus Warnerfjord, Thank you for trying out Kedro. In general Kedro assumes data flows one way through immutable snapshots. My suggestions would be - 1. Create a custom dataset: Hide the complexity inside a custom dataset class. It loads only unpredicted sequences and saves via upsert. Your pipeline nodes stay pure; the dataset handles the state complexity internally. 2. Two entries as you mentioned — Two catalog entries, one table. A "reader" dataset queries what's missing; a "writer" dataset upserts results. 3. Incremental-Native Storage — Use a format built for this (Delta Lake, DuckDB). They handle "add new, update existing, don't duplicate" natively, so Kedro can just point at them normally. Dataset recommendations: For 2 entries approach you can use pandas.SQLQueryDataset Read-only; lets you filter with

WHERE prediction IS NULL

pandas.SQLTableDataset Use

save_args: {if_exists: "append"}

for new rows If you want to get support for incremental checkpoints, you can explore Incremental Datasets or Partitioned Datasets If you want incremental native storage, you can explore DuckDB - ibis.TableDataset - Local/lightweight, great pandas interop Delta Lake - spark.DeltaTableDataset - Distributed, ACID transactions, time travel Polars - polars.LazyPolarsDataset - Memory-efficient, fast local analytics I am not sure if this is helpful for your use-case but we have a native BioSequenceDataset I will also let the community provide their suggestions. Please let me know if you need further information. Thank you

Went for a custom dataset with delta lake and sparks and everything works great and makes much more sense now, thanks for the help!