Hi team!!
I’m trying to convert a fairly large spa...
# questionsp
Priyanka Patil
03/30/2023, 12:08 PMHi team!!
I’m trying to convert a fairly large spark dataframe to pandas.
This is a super expensive operation, so I’m trying to convert chunks of spark dataframe to pandas dataframes.
Is there a kedro dataset that allows us to save multiple pandas dataframes to csv part files? thanks!
d
datajoely
03/30/2023, 12:08 PMPartitioned dataset
datajoely
03/30/2023, 12:08 PMand you can also do this in a lazy way
m
marrrcin
03/30/2023, 12:11 PMp
Priyanka Patil
03/30/2023, 2:01 PMthanks @datajoely @marrrcin!!