I would like to bring `polars` to our `kedro` pipe...
# questions
s
I would like to bring
polars
to our
kedro
pipeline, but I don't want to make a data catalog load files only with polars (
type: polars.CSVDataSet
); doing so, everyone else must load it as
polars
too. Is there a way in
kedro
to be able to use both formats in some way or should I create a custom dataset?
👍 1
j
You could create 2 entries in the data catalog, one for
polars
and one for (let's say)
pandas
but both pointing to the same
.csv
file. Then the user can choose which one to use when loading data to a notebook (for instance)
👍 1
Nonetheless, for the actual pipelines you will need to choose which one to use as default, your node's code will depend on which dataframe library you are using.
👍 1
Copy code
my_dataframe@pandas:
  type: pandas.CSVDataSet
  filepath: data.csv

my_dataframe@polars:
  type: POLARS
  filepath: data.csv
K 1
s
Well, Thanks! this is what I wanted.
😎 2