fmfreeze
05/26/2023, 12:24 PMAbstractDataSet
, kedro-viz
does not display the Dataset Type and the File Path property in the details section for that Dataset.
How can I make them show up?Antony Milne
05/26/2023, 12:30 PMAbstractDataSet
look like? I would expect this to work already 🤔fmfreeze
05/26/2023, 12:35 PMfrom typing import Dict
from pathlib import Path, PurePosixPath
import dask.dataframe as dd
from <http://kedro.io|kedro.io> import AbstractDataSet
class DaskCSVDataSet(AbstractDataSet):
def __init__(self, filepath, param1, param2=True):
self._filepath = PurePosixPath(filepath)
self._param1 = param1
self._param2 = param2
def _load(self) -> Dict[str, dd.DataFrame]:
# Implement logic to load multiple CSV files as Dask DataFrame
first_row = dd.read_csv(self._filepath, include_path_column=True, sep=';').head(n=0)
headers = dd.read_csv(self._filepath, include_path_column=True, sep=';', skiprows=[0]).head(n=2)
ddf = dd.read_csv(self._filepath, include_path_column=True, sep=';', skiprows=[0,1,3])
return {"first_row": first_row, "headers": headers, "ddf": ddf}
def _save(self, data: dd.DataFrame) -> None:
raise NotImplementedError("Saving Dask DataFrame as CSV is not supported")
def _describe(self):
return "hello csv"
def _exists(self) -> bool:
return Path(self._filepath.as_posix()).exists()
DaskCSVDataSet
) but instantly disappears again and is replaced with -
.Antony Milne
05/26/2023, 1:52 PM_describe
method like is done here? https://docs.kedro.org/en/stable/extend_kedro/custom_datasets.html#the-complete-examplefmfreeze
05/26/2023, 2:16 PM