Sergey S
11/14/2023, 12:10 AMclass KedroCocoDataset(AbstractDataset):
...
Then in catalog.yaml
coco_dataset:
type: kedro-example.datasets.KedroCocoDataset
filepath: data/01_raw/014-playment-parts-100-images-variant-a.json
However in the notebook (started by kedro jupyter lab
) when running %reload_kedro
I get an error:
DatasetError: An exception occurred when parsing config for dataset 'coco_dataset':
Class 'kedro-example.datasets.KedroCocoDataset' not found or one of its dependencies has not been installed.
Any help is appreciated.marrrcin
11/14/2023, 7:50 AMkedro-example/src/kedro-example/datasets.py
? Looks like a Python-level issue rather than Kedro issue.Nok Lam Chan
11/14/2023, 8:46 AMkedro_example
instead of kedro-example
, this is a python convention you cannot have -
as a namespace.
p.s. Most notable example is probably scikit-learn
, you do pip install scikit-learn
but you do import sklearn
instead of import scikit-learn
kedro new
this should be handled properly, if not then it’s a bug on our side👀The project name 'kedro-example' has been applied to:
- The project title in /Users/Nok_Lam_Chan/GitHub/kedro/kedro-example/README.md
- The folder created for your project in /Users/Nok_Lam_Chan/GitHub/kedro/kedro-example
- The project's python package in /Users/Nok_Lam_Chan/GitHub/kedro/kedro-example/src/kedro_example
I quickly test it with the latest release, it is working as expected.Lukas Innig
11/14/2023, 10:23 AMNok Lam Chan
11/14/2023, 10:41 AMSergey S
11/14/2023, 12:14 PMkedro-example
instead of kedro_example
in the code.
The issue #2 remains. I followed the project structure of the iris-pandas
example (pipelines.py
and nodes.py
in the src
directory) and my test pipeline works 🥳
However if I move pipelines.py
and nodes.py
to a sub directory called pipelines
(see image below) I am getting an error of ValueError: Pipeline contains no nodes after applying all provided filters
The issue seems to be that find_pipelines()
defined in pipeline_registry.py
does not find my pipeline if its defined in a sub folder. If I manually import my pipeline (from kedro_example.pipelines.pipeline import create_pipeline)
from the sub-directory and manually assign it (pipelines["__default__"] = create_pipeline()
) then everything works.spaceflight
example, I've added
from .pipeline import create_pipeline
to __init.py__
in the pipelines directory but that did not help when using pipelines = find_pipelines()
in the pipeline_registry.py
spaceflight
works:
src/
kedro-proj/
pipelines/
data_processing/
__init__.py # with 'from .pipeline import create_pipeline'
pipeline.py
nodes.py
Defining my pipeline inside pipelines
directory does not work:
src/
kedro-proj/
pipelines/
__init__.py # with 'from .pipeline import create_pipeline'
pipeline.py
nodes.py
Nok Lam Chan
11/14/2023, 1:32 PMSergey S
11/14/2023, 7:52 PMfind_pipeline
recursively iterates over the whole src
folder to find my pipelines.
It requiring a default folder structure makes sense as to why it didn't work, thanks!Nok Lam Chan
11/15/2023, 9:49 AM