Hello team, I was wondering if theres an approach ...
# questionss
Sid Shetty
07/24/2023, 2:13 PMHello team, I was wondering if theres an approach to break a pandas dataframe into chunks, run a few operations on it and write each chunk to a parquet in append mode(without concatenating the chunks back)? So the kedro node would have multiple writes.
d
datajoely
07/24/2023, 2:28 PMYes!
PartitionedDataSets
Sid Shetty
07/24/2023, 2:30 PMGotcha thank you!
Sid Shetty
07/25/2023, 12:03 PMThank you this approach worked great! Was wondering how the lazy loading works and maybe this is just a python concept and not related to the partition dataset implementation. Looks like multiple partitions are being saved at the same time and therefore their callables being executed in parallel, is that right? Is there a way to control the number of executions at one time and control RAM usage?
d
datajoely
07/25/2023, 12:12 PMhey so generators are really powerful piece of python where you can defer execution to when it’s needed rather than at runtime. The opposite is called eager.
If I do
range(1,1000000000000)list(range(1,1000000000000))datajoely
07/25/2023, 12:13 PMOne clarification is that the partitions aren’t processed in parallel, they are still sequential unless you do something custom
s
Sid Shetty
07/25/2023, 12:21 PMAhh I see, thank you, very cool stuff!
d
datajoely
07/25/2023, 12:22 PMmy pleasure