Marc Gris
08/22/2023, 10:39 AMkedro run -p data_processing --tags=users
[08/21/23 16:39:45] WARNING /home/kedro_docker/.local/lib/python3.9/site-packages/kedro/framework/project/__init__.py:359: UserWarning: warnings.py:109
An error occurred while importing the 'dodo_kedro.pipelines.filtering' module. Nothing defined therein will
be returned by 'find_pipelines'.
Traceback (most recent call last):
File "/home/kedro_docker/.local/lib/python3.9/site-packages/kedro/framework/project/__init__.py", line 357,
in find_pipelines
pipeline_module = importlib.import_module(pipeline_module_name)
File "/usr/local/lib/python3.9/importlib/__init__.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 850, in exec_module
File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
File "/workspaces/dodo-kedro/src/dodo_kedro/pipelines/filtering/__init__.py", line 6, in <module>
from .pipeline import create_pipeline
File "/workspaces/dodo-kedro/src/dodo_kedro/pipelines/filtering/pipeline.py", line 7, in <module>
from . import nodes
File "/workspaces/dodo-kedro/src/dodo_kedro/pipelines/filtering/nodes.py", line 41, in <module>
for _filter in catalog.load("params:filters")]
File "/home/kedro_docker/.local/lib/python3.9/site-packages/kedro/io/data_catalog.py", line 473, in load
dataset = self._get_dataset(name, version=load_version)
File "/home/kedro_docker/.local/lib/python3.9/site-packages/kedro/io/data_catalog.py", line 406, in
_get_dataset
raise DatasetNotFoundError(error_msg)
kedro.io.core.DatasetNotFoundError: Dataset 'params:filters' not found in the catalog - did you mean one of
these instead: parameters, params:models, params:columns.users
warnings.warn(
I’m wondering why it is trying to load the filters params for the filtering pipeline while I’m running only the data_processing pipeline ?
Those filters
are indeed not defined, since I had “commented-out” those in order to debug something…
Granted: This is not a “drama” since it only raises a warning and still allows to run the data_processing pipeline.
Yet, I must say it’s a little confusing to see a warning for a pipeline that was not called
This seems to me to related to the lazy loading of datasets for which I have opened a feature request on github.
Looking forwards to reading your thoughts / comments.
Regards
Mmarrrcin
08/22/2023, 11:18 AMAleksander Jaworski
08/22/2023, 11:24 AMMarc Gris
08/22/2023, 11:57 AMJuan Luis
08/23/2023, 8:11 AMMarc Gris
08/23/2023, 8:18 AMfind_pipelines()
Thanks for your comment 🙏🏼
Mdatajoely
08/23/2023, 8:19 AMfind_pipelines
and register them yourself for nowMarc Gris
08/23/2023, 8:44 AMkedro run -p filtering
I end up with
TypeError: Inputs of 'recommend' function expected ['scores', 'k'], but got ['filtered_scores', 'params:k', 'params:user_id_col', 'params:item_id_col', 'params:score_col']
This is a node in another pipeline…
I’m therefore re-posting what I’ve just posted previously:
My initial feature request was indeed originally for datasets.
But couldn’t, or even shouldn’t, it be broadened to pipelines as well ?
The spirit being:
Why load something that is not needed / called for ?
& Why should one be blocked from working on pipeline a because things are broken in pipeline b ?
Many thanks in advance for you comments / suggestions.
Cheers
M.Juan Luis
08/25/2023, 9:32 AMpipeline_registry:find_pipelines
function?
about the historical context on why things work this way, we lack that insight at the moment, but we haven't received similar reports in the past. so maybe users were finding ways of working around this without telling us.Marc Gris
08/25/2023, 10:40 AMfind_pipelines()
does the job of course 🙏🏼
My posts are not so much about “complaining” than they are about “giving feedbacks” that hopefully / humbly could be helpful to you guys 🙂
What I described above, seems to be to be a reasonable expectation from a first time kedro user…
Hence my sharing this here.
Thanks again & have a nice day,
M.Juan Luis
08/25/2023, 11:58 AMMarc Gris
08/25/2023, 12:10 PM