https://kedro.org/ logo
Join Slack
Powered by
Hello, I have a question about pytest import error...
# questions
m
Hello, I have a question about pytest import error: This is my project's structure:
Copy code
src
│   ...
└───<package_name>
│   └───pipelines
│       └───data_engineering
|           └───nodes
│               │   node_X.py
└───tests
│   └───pipelines
│       └───data_engineering
|           └───nodes
│               │   test_node_X.py
In my test file, I have to add the first 2 lines else I will get an Import error:
Copy code
# Uncommenting the following 2 lines solves the error
# import sys
# sys.path.append("path_to/src")

import pytest

from <package_name>.pipelines.data_engineering.nodes.node_X import (
    func_X,
)
Is there a cleaner way than having to append the path for each test file? ERROR Message:
Copy code
==================================== ERRORS ======================================================================================
________________________________________________ ERROR collecting src/tests/pipelines/data_engineering/nodes/test_node_X.py ________________________________________________
ImportError while importing test module 'path_to/src/tests/pipelines/data_engineering/nodes/test_node_X.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/home/ec2-user/miniconda3/envs/python310/lib/python3.10/importlib/__init__.py:126: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
src/tests/pipelines/data_engineering/nodes/test_node_X.py:7: in <module>
    from <package_name>.pipelines.data_engineering.nodes.node_X import (
E   ModuleNotFoundError: No module named '<package_name>'
Thank you.
d
Hi @Mouna Balghouthi you shouldn’t have to do any sys path appending
we have some docs on what you need to do here https://docs.kedro.org/en/stable/development/automated_testing.html
m
@datajoely yes exactly it wasn't mentioned in teh official documentation, that's why I am wondering why I get the import error just for running tests 🤔
d
so another question is this python installation the same one you are using in your regular work 
/home/ec2-user/miniconda3/envs/python310/lib/python3.10
👍 1
m
Yes a conda environment (in a Cloud9 environment) where the code runs smoothly 👌
Following the documentation, the src/tests/test_run.py example runs successfully. The issue is in my added test_node_X.py file
d
Awesome
is there anything in the docs we could have done to make this journey cleaner / easier?
m
If I remove my test_node_X.py file and just run pytest for the src/tests/test_run.py example: I can see my node_X.py file in the coverage:
Copy code
=============================================================================== test session starts ===============================================================================
platform linux -- Python 3.10.13, pytest-7.4.3, pluggy-1.2.0
rootdir: path_to_directory_containing_src
configfile: pyproject.toml
plugins: cov-4.1.0
collected 1 item                                                                                                                                                                  

src/tests/test_run.py .                                                                                                                                                     [100%]/home/ec2-user/miniconda3/envs/python310/lib/python3.10/site-packages/coverage/control.py:883: CoverageWarning: No data was collected. (no-data-collected)
  self._warn("No data was collected.", slug="no-data-collected")


---------- coverage: platform linux, python 3.10.13-final-0 ----------
Name                                                                                 Stmts   Miss  Cover   Missing
------------------------------------------------------------------------------------------------------------------
src/markdown/__init__.py                                                                 1      1     0%   3
src/markdown/__main__.py                                                                30     30     0%   2-46
src/markdown/pipeline_registry.py                                                       36     36     0%   3-95
src/markdown/pipelines/__init__.py                                                       0      0   100%
src/markdown/pipelines/data_engineering/__init__.py                                      3      3     0%   3-7
src/markdown/pipelines/data_engineering/nodes/__init__.py                                0      0   100%
src/markdown/pipelines/data_engineering/nodes/node_X.py                         203    203     0%   3-789
However, when I put back my test file test_node_X.py I'd still get the import error. 🤷🏻‍♀️
d
are you doing anything unusual in your imports of that file?
m
Just using the function I am importing:
Copy code
# import sys
# sys.path.append("rootdir_path/src")

import pytest

from <package_name>.pipelines.data_engineering.nodes.node_X import (
    func_X,
)
....
@pytest.mark.parametrize(...)
def test_funcX(...):
   .....
d
that is odd
it shouldn’t be anything in the kedro stuff, it should just be regular Kedro stuff. I wonder if doing 
pip install -e .
(installing the local package in dynamic editable mode) at the top level of your project will make things easier
👀 1
m
I'll try that!
Current config: rootdir: path_to_directory_containing_src I can also double check that rootdir is already in the sys.path from the terminal:
Copy code
> python print_sys_path.py
['rootdir_path', '/home/ec2-user/miniconda3/envs/python310/lib/python310.zip', '/home/ec2-user/miniconda3/envs/python310/lib/python3.10', '/home/ec2-user/miniconda3/envs/python310/lib/python3.10/lib-dynload', '/home/ec2-user/miniconda3/envs/python310/lib/python3.10/site-packages']
The line I am adding in my test file:
Copy code
import sys
sys.path.append("rootdir_path/src")
That's why I am doubting my project structure but looks similar to the documentation
Copy code
pip install -e .
outputs this error that I'd need more time to look into:
Copy code
Obtaining file:///home/ec2-user/environment/project_rootdir
  Installing build dependencies ... done
  Checking if build backend supports build_editable ... done
  Getting requirements to build editable ... error
  error: subprocess-exited-with-error
  
  × Getting requirements to build editable did not run successfully.
  │ exit code: 1
  ╰─> [14 lines of output]
      error: Multiple top-level packages discovered in a flat-layout: ['common', 'markdown'].
      
      To avoid accidental inclusion of unwanted files or directories,
      setuptools will not proceed with this build.
      
      If you are trying to create a single distribution with multiple packages
      on purpose, you should not rely on automatic discovery.
      Instead, consider the following options:
      
      1. set up custom discovery (`find` directive with `include` or `exclude`)
      2. use a `src-layout`
      3. explicitly set `py_modules` or `packages` with a list of names
      
      To find more information, look for "package discovery" on setuptools docs.
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

× Getting requirements to build editable did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.
d
I’m really not sure! at least that error is different, try this? https://stackoverflow.com/questions/72294299/multiple-top-level-packages-discovered-in-a-flat-layout
👀 1
m
Thanks for sharing the above link, I tried suggested solutions there but still similar error 
error: subprocess-exited-with-error
To give you more context about the project structure:
Copy code
package_name= my_ds_project
root_dir
|   ...
|   requirements.txt
|   pyproject.toml
|   my_ds_project
|   │   ...
|   └───spec_folder
|       │   ...
|       |   requirements.txt
|       └───my_ds_project
|           │   ...
            └───src
                │   ...
                |   setup.py
                └───<my_ds_project
                │   └───pipelines
                │       └───data_engineering
                |           └───nodes
                │               │   node_X.py
                └───tests
                │   └───pipelines
                │       └───data_engineering
                |           └───nodes
                │               │   test_node_X.py
b
How are you running your tests? This path 
path_to/src/tests/pipelines/data_engineering/nodes/test_node_X.py
looks wrong - I would expect 
tests/pipelines/data_engineering/nodes/test_node_X.py
. My hunch is you are either running the tests from the wrong location, with the wrong command, or in an IDE without adding 
src
as a source root
j
Copy code
Multiple top-level packages discovered in a flat-layout: ['common', 'markdown'].
      
      To avoid accidental inclusion of unwanted files or directories,
      setuptools will not proceed with this build.
the project structure is non-standard, that's why 
pip install -e .
doesn't work
or you're running it from the wrong directory. @Mouna Balghouthi do you recognize those 
common
and 
markdown
directories?
👍 1
m
For: 
pip install -e .
@Juan Luis Yes, I tried from different directories. Yes 
common
and 
markdown
are on the same level inside the root_dir and is 
markdown
is my_ds_project directory
in an IDE without adding 
src
as a source root
@Ben Horsburgh I am looking into this
So 
kedro test
runs the tests successfully but 
pytest
not and I get this warning: 
DeprecationWarning: Command 'kedro test' is deprecated and will not be available from Kedro 0.19.0. Use the command 'pytest' instead.
d
unless you upgrade to 
kedro 0.19.x
it won’t go away, but it should be doing the same thing
m
Current version: 0.18.14 I guess we'll have to update at some point next year 🤔
[SOLVED] pytest scans directories until it finds a directory where there is no init.py We need an init.py in 
/src/tests/pipelines
and in 
/src/tests
We need to delete the init.py in 
/src/
(not sure why we had one there in the first place) This would then mirror also the structure in https://github.com/WaylonWalker/kedro-spaceflights/tree/feature/project-template/src Then it should work to just import the package_name under src since it automatically then takes the src folder as root. Thank you @datajoely @Ben Horsburgh @Juan Luis for all your contributions .gratitude thank you
d
amazing
is there anything we could have done in the docs to help your journey?
3 Views
Open in Slack
PreviousNext
Powered by