https://kedro.org/ logo
Join Slack
Powered by
hey everyone, I am building a system where i retu...
# questions
f
hey everyone, I am building a system where i return the key/filepath of final dataset in the kedro pipeline. What's the ideal way of doing this? A method that also works for partitioned datasets where i get a list of filepaths? I have a catalog instance but somehow all methods are protected so im wondering if im missing something obvious here. I was doing catalog._get_dataset(output)._filepath which works only for non partitioned datasets
n
Hi @Fazil Topal, I am slightly outdated here, but I suggest looking at https://docs.kedro.org/en/0.19.14/data/index.html#kedrodatacatalog-experimental-feature This will be more close to the API in kedro 1.0. I recalled these protected method/filepath are part of the discussion, maybe it's resolved already
👍 1
There are also a 1.0rc release that you can try. New docs are here: https://docs.kedro.org/en/1.0.0rc1/catalog-data/deprecated_api/ https://pypi.org/project/kedro/1.0.0rc1/
r
Not aware of any other way. This is how we do in kedro-viz - https://github.com/kedro-org/kedro-viz/blob/main/package/kedro_viz/integrations/kedro/hooks.py#L168 For partitioned datasets, since it is a directory path, I see we do not have _filepath there. You can use -
Copy code
# Partitioned dataset
    if isinstance(dataset_obj, PartitionedDataset):
        partition_dir = Path(dataset_obj._path)
        filepaths = list(partition_dir.glob("*"))
        return filepaths
f
I end up doing this for now:
Copy code
def get_filepaths(catalog: DataCatalog, name: str) -> str | list[str]:
    ds = catalog._get_dataset(name)

    if isinstance(ds, PartitionedDataset):
        return ds._list_partitions()
    else:
        return ds._filepath
👍 1
Okay coming back to this as this does not work on versioned path and somehow the protocol is not returned (s3://) I am not sure where kedro parses this but im lost, can't find the functions that is being used to create the full path 😅
And tops here on how to work with versioned files and also on remote ones? I have this one but somehow it does not work for the partitioned remote files:
Copy code
def get_filepaths(catalog: DataCatalog, name: str) -> str | list[str]:
    def append_protocol(fpath: str, protocol: str) -> str:
        return f"{protocol}://{str(fpath)}" if protocol != "file" else str(fpath)

    ds = catalog._get_dataset(name)

    if isinstance(ds, PartitionedDataset):
        fpaths = []

        for partition in ds._list_partitions():
            kwargs = deepcopy(ds._dataset_config)
            kwargs[ds._filepath_arg] = partition
            pds = ds._dataset_type(**kwargs)
            ppath = append_protocol(pds._get_load_path(), pds._protocol)
            fpaths.append(ppath)
        return fpaths
    else:
        return append_protocol(ds._get_load_path(), ds._protocol)
Any ideas @Nok Lam Chan @Ravi Kumar Pilla?
r
Hi @Fazil Topal, have you tried _list_partitions and _join_protocol methods instead of custom appending of protocol ? What's not working ? Is it not appending the protocol or not finding the path to partition ? - May be you can use - 
_path_to_partition
f
kwargs[ds._filepath_arg] = partition
-> this line was the problem, 
kwargs[ds._filepath_arg] = ds._join_protocol(partition)
seems to fix the issue. Before, it was not passing the protocol so it was inferred as local file, not it works correctly. It's a bit weird to access every protected method to get the final path that the file will be read. I wonder if there is plan to expose a public method where we can just fetch the path (load/save path)?
💡 1
I also noticed another potential issue. This code is copied from partitioned dataset load method. If i have versioned enabled, this somehow does not work as expected. For instance i have 2 partition1 and 1 partition2 file. The list partitions are returning 3 items but without the versions so we have a list of [partition1, partition1, partition2] but the rest of the code always pull the latest partition so i ended up with a duplicate list.
r
yes I think there should be a simpler way of accessing the path. May be I am missing something. @Dmitry Sorokin do we have an easy way of accessing file paths when it comes to partitioned datasets (versioned/non-versioned) ?
10 Views
Open in Slack
PreviousNext
Powered by