Hi team I m trying to convert a fairly large spark dataframe

Question

Hi team I m trying to convert a fairly large spark dataframe to pandas This is a super expensive operation so I m trying to convert chunks of spark dataframe to pandas dataframes Is there a kedro dataset that allows us to save multiple pandas dataframes to csv part files thanks

datajoely · Answer

Partitioned dataset

datajoely · Answer

and you can also do this in a lazy way

marrrcin · Answer

point up <https stackoverflow com a 74066977 1955346>

Priyanka Patil · Answer

thanks < datajoely> < marrrcin>