How to Incrementally Append and Upsert Rows in Kedro directly to PostgreSQL DBt?
I'm working on a Kedro project where I have a dataset defined in catalog.yml as follows:
daily_stats_dataset:
type: ${datasets.orm_table}
orm_model: my_proj.schemas.sqla_schemas.DailyStats
credentials: my_database
monthly_stats_dataset:
type: ${datasets.orm_table}
orm_model: my_proj.schemas.sqla_schemas.MonthlyStats
credentials: my_database
I need to incrementally append new rows from my daily_stats_dataset to my monthly_stats_dataset. However the daily stats are aggregated...