https://kedro.org/ logo
#questions
Title
# questions
s

saeed choobani

10/05/2023, 1:59 PM
I would like to bring
polars
to our
kedro
pipeline, but I don't want to make a data catalog load files only with polars (
type: polars.CSVDataSet
); doing so, everyone else must load it as
polars
too. Is there a way in
kedro
to be able to use both formats in some way or should I create a custom dataset?
👍 1
j

Jose Nuñez

10/05/2023, 2:11 PM
You could create 2 entries in the data catalog, one for
polars
and one for (let's say)
pandas
but both pointing to the same
.csv
file. Then the user can choose which one to use when loading data to a notebook (for instance)
👍 1
Nonetheless, for the actual pipelines you will need to choose which one to use as default, your node's code will depend on which dataframe library you are using.
👍 1
Copy code
my_dataframe@pandas:
  type: pandas.CSVDataSet
  filepath: data.csv

my_dataframe@polars:
  type: POLARS
  filepath: data.csv
K 1
s

saeed choobani

10/05/2023, 2:14 PM
Well, Thanks! this is what I wanted.
😎 2