#questions
Title
# questions
j
Juan Luis
03/30/2023, 8:40 AMhi folks, I created a custom dataset to see if I could understand the documentation and how it works, but I feel I'm doing some unconventional things and I'd need some advice:
I created a
KaggleDataSet_loadKaggleBundle_savecatalog Copy code
In [2]: catalog.list()
Out[2]: ['spaceship_titanic_kaggle', 'head_titanic_train', 'parameters']
In [3]: bundle = catalog.load("spaceship_titanic_kaggle")
[03/30/23 10:21:01] INFO Loading data from 'spaceship_titanic_kaggle' (KaggleDataSet)... data_catalog.py:343
[03/30/23 10:21:02] WARNING /Users/juan_cano/.micromamba/envs/kaggle310-dev/lib/python3.10/site-packages/kaggle/rest.py:62: DeprecationWarning: warnings.py:109
HTTPResponse.getheaders() is deprecated and will be removed in urllib3 v2.1.0. Instead access HTTPResponse.headers
directly.
return self.urllib3_response.getheaders()
In [4]: bundle
Out[4]: KaggleBundle(dataset_or_competition='spaceship-titanic', members=['sample_submission.csv', 'train.csv', 'test.csv'], is_competition=True, single_file=False)
In [5]: catalog.save("spaceship_titanic_kaggle", data=bundle)
[03/30/23 10:21:51] INFO Saving data to 'spaceship_titanic_kaggle' (KaggleDataSet)... data_catalog.py:382
[03/30/23 10:21:52] WARNING /Users/juan_cano/.micromamba/envs/kaggle310-dev/lib/python3.10/site-packages/kaggle/api_client.py:181: warnings.py:109
DeprecationWarning: HTTPResponse.getheaders() is deprecated and will be removed in urllib3 v2.1.0. Instead access
HTTPResponse.headers directly.
response_data.getheaders())
Downloading spaceship-titanic.zip to data/01_raw
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 299k/299k [00:00<00:00, 988kB/s]
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 299k/299k [00:00<00:00, 986kB/s]
In [6]: !ls data/01_raw
sample_submission.csv test.csv train.csvKaggleDataSet Copy code
# catalog.yaml
spaceship_titanic_kaggle_orig:
type: kedro_kaggle_dataset.KaggleDataSet
dataset: spaceship-titanic
directory: data/01_raw/
is_competition: True
spaceship_titanic_kaggle:
type: kedro_kaggle_dataset.KaggleDataSet
dataset: spaceship-titanic
directory: data/01_raw/
is_competition: True
# nodes.py
from kedro_kaggle_dataset.kaggle_dataset import KaggleBundle
def download_titanic(data: KaggleBundle) -> KaggleBundle:
return data
# pipeline.py
def create_pipeline(**kwargs) -> Pipeline:
return pipeline([
node(
func=download_titanic,
inputs="spaceship_titanic_kaggle_orig",
outputs="spaceship_titanic_kaggle",
name="download_titanic_node",
),
...this is the code of the dataset in case you want to try it https://github.com/astrojuanlu/kedro-kaggle-dataset
is there a better way of doing this?
d
datajoely
03/30/2023, 8:56 AMso i think the trick here is to be explicit with
load_argssave_argsin truth this is a higher level abstraction than we normally do in the standard bundle
so not all of the concepts map neatly
a
Antony Milne
03/30/2023, 9:53 AMYeah, this is a very interesting idea but I think might be tricky to do within the kedro catalog paradigm as it stands. I think the simplest way would be to have a dataset a bit like
APIDataSet_load_saveload_args_loadpandas.CSVDataSet💡 1
Curious if @Nok Lam Chan has any ideas here
d
datajoely
03/30/2023, 9:56 AMI’ll forward my invite to you two
✔️ 1
n
Nok Lam Chan
03/30/2023, 12:32 PMI’ll need some time to think about it more, APIDataSet approach seems fine to me.
A bit off topic, I think we need some kind of
PyTorchDataLoaderDataSet_save_loadd
datajoely
03/30/2023, 12:38 PMyeah seen lots of people talk about that
I’ve considered a special type of node that processes each partition 1:1
cos image processing today is a bit weird in one node
j
Juan Luis
03/30/2023, 2:37 PMor maybe a dataset that needs no inputs, because what I'm doing here is essentially downloading a thing from the internet and putting it in
01_rawbut the "dummy node" problem would still be there
(I also think the "custom datasets" page needs better docs but I'll tackle that when I fully understand them 😅 )
n
Nok Lam Chan
03/30/2023, 2:39 PMI can’t run the code for some reason - keep getting 403 error, running on CLI is fine tho
j
Juan Luis
03/30/2023, 2:40 PMauthentication problems? I was using environment variables for that
n
Nok Lam Chan
03/30/2023, 2:42 PMDownloading seems to be a
_load_savesavej
Juan Luis
03/30/2023, 2:51 PMyeah I was struggling with that. the problem of that particular case is that the Kaggle download could be anything, so I don't know up front how should I handle that
👍🏼 1
n
Nok Lam Chan
03/30/2023, 2:51 PMI don’t have much better idea than the one Antony has, it feels repeating though to opening up these file in a node and converting them as a dataframe
j
Juan Luis
03/30/2023, 2:53 PMthat works as long as you assume that everything are dataframes 🤔 but maybe we could return a dictionary of
io.BytesIOand then downstream nodes decide what to do
by the way this approach worked 😄 and now I do understand what
_load_saveK 1
Copy code
def _load(self) -> KaggleBundle:
...
members = {}
for member_filename in members_list:
with open(self._directory / member_filename, "rb") as fh:
members[member_filename] = BytesIO(fh.read())
return KaggleBundle(..., members)
def _save(self, KaggleBundle): -> None:
raise NotImplementedError("Cannot save back to Kaggle")KaggleDataSetpandas.CSVDataSet Copy code
def head_titanic(bundle: KaggleBundle) -> pd.DataFrame:
df_test = pd.read_csv(bundle["test.csv"])
return ...notes for Monday: what to do if I don't want to download this data every time I run a pipeline
d
datajoely
04/01/2023, 8:21 AMI made an expiring cahced dataset a long time ago
I should contribute it
it essentially pickled something with the date it was last downloaded and then only triggered a new request after some threshold passed
💡 1
j
Juan Luis
04/03/2023, 3:05 PMwaiting folks!
@Nok Lam Chan @datajoely
d
datajoely
04/03/2023, 3:06 PMsorry
n
Nok Lam Chan
04/03/2023, 3:10 PMSorry just finish an adhox meeting
d
datajoely
04/03/2023, 3:13 PMUntitled.py
This my super old ExpiringHTTPDataSet
and it could be used like this
j
Juan Luis
04/03/2023, 3:59 PM Copy code
birdclef_kaggle:
type: BagOfFilesDataSet # .zip/heterogeneous/generic/whatever
path: <kaggle://birdclef-2023/>
credentials: kaggle_credentials
patterns: "**/*.ogg"
# birdclef_kaggle ----> Bundle ----> audio_files
audio_files:
type: PartitionedDataSet
path: data/02_intermediate/audio
dataset: audio.OGGDataSet👍🏼 1
n
Nok Lam Chan
04/03/2023, 4:03 PMTotally unrelated to the KaggleDataSet - I like the
patterns☝🏼 1
j
Juan Luis
04/03/2023, 4:04 PMthe best kind of meeting! 😂
last thing we discussed @datajoely not sure if you were still there is that datasets seem to involve formats, whereas protocols feel more appropriate for transports or locations
👍 1
n
Nok Lam Chan
04/03/2023, 4:07 PMAlso, I like the wrapper DataSet concept - which I haven’t seen much people doing it? Essential it’s a MixIn approach for datasets. (Not sure if the format is the best though, since it’s weird to have more than 2 MixIn as your catalog start to look like arbitary nested)
I used to have a wrapper
CacheArtifactDataSetpathj
Juan Luis
04/15/2023, 9:44 AMFWIW I've been trying to implement a Kaggle filesystem - still unsure how it will allow me to download the whole
.ziplsinfosize