Jordan
05/02/2023, 11:16 AM%load_ext kedro.ipython
.
However, in a standalone file when I am creating the catalog as follows:
from kedro.framework.session import KedroSession
from kedro.framework.startup import bootstrap_project
project_path = Path(".").resolve()
metadata = bootstrap_project(project_path)
with KedroSession.create(metadata.package_name, project_path) as session:
context = session.load_context()
catalog = context.catalog
data = catalog.load("my_metrics")
I get the following error:
DataSetError: Loading not supported for 'MetricsDataSet'
If this is true, why does it load in a notebook?Juan Luis
05/02/2023, 4:19 PM_load
implementation:
https://github.com/kedro-org/kedro-plugins/blob/435689f0eba9f643e1f9e8ac75f2151279[β¦]7e476/kedro-datasets/kedro_datasets/tracking/metrics_dataset.py
are you certain that it's working inside the notebook?Jordan
05/02/2023, 4:50 PMJuan Luis
05/03/2023, 11:22 AM.exists
method returns True
, whereas in my case returns False
. could you please run this code in the notebook session (where the loading is working for you):
catalog._get_dataset("original_total_power")._version
and also
_ds._load??
and paste or show the outputs here?Jordan
05/03/2023, 2:32 PMJuan Luis
05/03/2023, 2:36 PM_ds
was supposed to be this:
_ds = catalog._get_dataset("original_total_power")
_ds._load??
should workJordan
05/03/2023, 2:39 PMJuan Luis
05/03/2023, 2:43 PMload
proceeds π€ but still unclear why MetricsDataSet
is not therefrom kedro_datasets.tracking.metrics_dataset import MetricsDataSet
and then show the full source using MetricsDataSet??
finally, printing the kedro_datasets
version will definitely help: import kedro_datasets; print(kedro_datasets.__version__)
Jordan
05/03/2023, 2:45 PMJuan Luis
05/03/2023, 2:57 PMkedro_datasets
1.2.0.
yes, MetricsDataSet
inherits from JSONDataSet
, but I was expecting to see a _load
method there that would override the parent one. I see ...
there, maybe the output was truncated?print(MetricsDataSet._load) # I expect <function JSONDataSet._load at ...> in your case, should be <function MetricsDataSet._load at ...> instead
print(MetricsDataSet.__init__) # I expect <function JSONDataSet.__init__ at ...> in both
Jordan
05/03/2023, 3:04 PM_load
method that should be overriding:
def _load(self) -> NoReturn:
raise DataSetError(f"Loading not supported for '{self.__class__.__name__}'")
You are right!Juan Luis
05/03/2023, 3:06 PMJordan
05/03/2023, 3:10 PMkedro-datasets
, what do I get for free, as it were? ie. If I just install kedro-datasets
without passing any extras, I should get all the built in datasets like JSON, Metrics, etc, but not things like Parquet, and so on. Is this assumption correct?Juan Luis
05/03/2023, 3:19 PMkedro
, like AbstractDataSet
, PartitionedDataSet
and so on. if you install kedro-datasets
without any extras, you don't get anythingJordan
05/03/2023, 3:24 PMkedro-datasets
implementation of a particular dataset if it is also available in the kedro
core framework?Juan Luis
05/03/2023, 3:24 PMkedro.extras.datasets
is deprecated and will go away in Kedro 0.19. we just added a DeprecationWarning
in 0.18.8Jordan
05/03/2023, 3:26 PMJuan Luis
05/03/2023, 3:28 PMJordan
05/03/2023, 4:03 PMJuan Luis
05/03/2023, 4:06 PMconda-lock.yml
and poetry.lock
, how should I try to recreate the environment? also, I see you were using VSCode right? or does this happen in a normal ipython
shell too?Jordan
05/03/2023, 4:10 PMenvironment.yml
has the non-standard platforms
key. I've been installing with Micromamba like:
micromamba env create -f conda-lock.yml -p ./.env
micromamba activate ./.env
poetry install
Then everything should be good to go.Juan Luis
05/03/2023, 4:13 PMJordan
05/03/2023, 4:13 PMipython
shell, so I can't speak to thatJuan Luis
05/03/2023, 4:14 PM$ ipython
>>> %load_ext kedro.ipython
>>> ...
displays the same weird behavior? namely, that MetricsDataSet._load
seems to be goneJordan
05/03/2023, 4:18 PMJuan Luis
05/03/2023, 4:19 PMJordan
05/03/2023, 4:19 PMJuan Luis
05/03/2023, 4:33 PMJordan
05/03/2023, 8:52 PMblack
,conda-lock
and pytest
from the dev deps to slim down builds while debugging if you wantJuan Luis
05/03/2023, 9:46 PMkedro
, kedro-viz
, and kedro-datasets[tracking]
would probably suffice, if you want to try that out (and maybe reproduce it without the need of a full Docker container) feel free to add more information to the issue @Jordan!Jordan
05/03/2023, 9:55 PMAntony Milne
05/04/2023, 10:07 PM