Hello guys, I'm struggling for weeks on a splitting problem with kedro : I have a generator that is lazy loading my data (huge webdataset) and i want to split it in 2 dataset. How to performe this since i can only get data on a sample when I access it ? Thanks in advance for your answers !!

Have you seen the

PartitionedDataset

?

Yes but it does not fit my needs. I have a lot of small files so i use tar archive and webdataset to save and process them. But I struggle for the sppliting operation since i can't divide an iterator in 2 output objects without accessing each sample

Would like to help but I am afraid I don't understand the question, any chance you can show some code snippets even that api could be imaginary?

Hello ! Thanks for the answer ! Here is an exemple :

# Here is what my loader is doing
dataset = (
            wds.WebDataset(files, nodesplitter=self.split_by_node)
            .decode(
                wds.torch_audio,
            )
        )

def preprocess(sample):
    # Do some operations
    return sample

# Here is my node code
def compute_node(dataset):
    dataset.map(preprocess)
    return dataset


# Here is the target node output
def compute_node_and_split(dataset):
    dataset.map(preprocess)

    # How to split webdataset with kedro without increasing complexity
    return split1, split2

https://stackoverflow.com/questions/2425270/how-to-look-ahead-one-element-peek-in-a-python-generator Would this work?

mmmh i don't think the first link is working since the goal of webdataset is to avoid loading elements in memory all at once

Can you split an iterator with pure Python? It may be easier to forget about Kedro for a second. Can you split it in the first place when you generate the iterator instead?

No, webdataset is a streaming, so it read on the fly the file as soon as it's downloaded

What does these two different return actually do? Are they consumed differently?

Well it's to return to one dataset or the other but it's horrible ahah

The downstream consumer has to understand the difference anyway, you can create a data class instead

yes but i'll have to change the runner right ?

Hm, not sure if I am following. Why do you need to modify runner? After all it’s just function that takes input and produce output. Maybe you can elaborate what are that two different outputs? The idea of catalog and datasets is all about I/O abstraction, the node (function) operate on data directly. Just like a node will take pandas data frame as input but not a pandas.csvdataset.

Yes i know about that but maybe i was not clear. Let's say you have a partitioned dataset. You don't want to load everything in memory so you iterate on samples and you yield each result in order to clear the memory right ? What i want to do is mostly the same. I want to choose the dataset to write on for each sample. But in kedro i can't yield only on one dataset if my node output on 2 dataset

So the question is about dynamic saving an output to different datasets?

Yes exactly

If you treat it as memory dataset, it works out of the box. The problem is when you want to use Kedro Datacatalog, there is no dataset that you can use directly to save as different type.

But it's again the kedro way no ? It's moving the logic in the dataset

Is it an IO logic?

By IO logic you mean the split is defined dynamicaly depending of the sample ? If yes so yes it is

In my mind it is handling how to save a Python object and it fits the purpose of what Dataset supposed to do. Only in this case you have a dynamic output which could be different type of objects. This makes sense to me to have your node return a certain DataClass and a dataset that understand how to handle the DataClass

Even for purpose like train test split ? Like you suggest to add a variable path to the dataclass so that de customdataset will know were to save ?

Train test split is beyond the job of a dataset, this should be done by a node.

yes but if this is done by a node, how can i do it ? I don't have, like a partitioned dataset, the list of the sample (i discover it when i iterate over the iterator)