I would like to bring `polars` to our `kedro` pipeline but I

Question

I would like to bring `polars` to our `kedro` pipeline but I don t want to make a data catalog load files only with polars `type polars CSVDataSet` doing so everyone else must load it as `polars` too Is there a way in `kedro` to be able to use both formats in some way or should I create a custom dataset

Jose Nuñez · Answer

You could create 2 entries in the data catalog one for `polars` and one for let s say `pandas` but both pointing to the same ` csv` file Then the user can choose which one to use when loading data to a notebook for instance

Jose Nuñez · Answer

Nonetheless for the actual pipelines you will need to choose which one to use as default your node s code will depend on which dataframe library you are using

marrrcin · Answer

marrrcin · Answer

```my dataframe pandas type pandas CSVDataSet filepath data csv my dataframe polars type POLARS filepath data csv```

saeed choobani · Answer

Well Thanks this is what I wanted