Hey! Quick question, is there any easy way to read a delta table in kedro, but the final object is a pandas dataframe? Or should I have to work in a custom DataSet in order to make this happen? I would like to avoid using .toPandas() function inside every node for every input, or have to add a decorator to every func to achieve this. Main goal is only use kedro catalog to manage this problem

The

DeltaTableDataSet

we’ve got in Kedro is based on Spark (https://github.com/kedro-org/kedro/blob/main/kedro/extras/datasets/spark/deltatable_dataset.py) If you need different behaviour you would indeed need to create a custom version.

Ok! Yes, reading about delta and pandas integrations in their repos sounds like i would need to create a custom dataset for this need. Thanks for the answer @Merel!