https://kedro.org/ logo
#user-research
Title
# user-research
f

Filip Panovski

11/10/2022, 1:36 PM
Hi! I'm not sure if this is quite the right place for something like this, but I created a feature request (https://github.com/kedro-org/kedro/issues/2016) to gauge interest in Avro support for the Kedro
dask.ParquetDataSet
. I had a use case where I needed to parse an existing Avro schema and transform it into a pyarrow schema so that the
<http://dask.to|dask.to>_parquet
function behaves nicely. I was wondering whether this is something the community would be interested in and would appreciate feedback.
d

datajoely

11/10/2022, 1:58 PM
@Deepyaman Datta is our dask dude - I’m sure he has opinions here!
👍 1
d

Deepyaman Datta

11/10/2022, 2:09 PM
Would this be in addition to #1746? I honestly don't know off the top of my head (don't actually use Dask now, just did a but of prototyping in the past) but I can remind myself how the schema parsing works tonight.
f

Filip Panovski

11/10/2022, 2:12 PM
The idea was for it to be in addition to #1746, yes.. but I haven't looked at the code there in detail (just going off the discussions), so it might be possible to integrate it there instead. It probably should not be part of that same PR though, IMO, as it would blow up the scope a bit.
d

Deepyaman Datta

11/10/2022, 2:16 PM
For sure
I will try and take a look over the weekend, sorry for the delay
f

Filip Panovski

11/11/2022, 12:21 PM
No worries, whenever you've got time.
2 Views