https://kedro.org/ logo
#questions
Title
# questions
m

Mate Scharnitzky

07/28/2023, 11:21 AM
Hi Team, I’m working in a SageMaker notebook which is in the
/notebook
directory. I’m trying to load some nodes that I created locally but it doesn’t find path. Two questions: • How can load kedro context to this notebook? • How can I load python modules developed as part of the kedro project? Thank you! I looked into this but somehow I can’t make it work: https://docs.kedro.org/en/0.18.11/notebooks_and_ipython/kedro_and_notebooks.html
d

datajoely

07/28/2023, 11:25 AM
do you get an error?
j

Juan Luis

07/28/2023, 11:37 AM
yeah, what does
%load_ext kedro.ipython
give?
m

Mate Scharnitzky

07/28/2023, 12:05 PM
When I’m trying to load the below:
Copy code
from gci_pricing.pipelines.feature_engineering.treasury import (
    create_treasury_total_features,
    create_treasury_trx_type_features
)
I’m getting
ModuleNotFoundError: No module named 'gci_pricing'
.
j

Juan Luis

07/28/2023, 12:07 PM
oh okay I misunderstood, this must be a
sys.path
thing. you're trying that inside a notebook right?
m

Mate Scharnitzky

07/28/2023, 12:07 PM
For
%load_ext kedro.ipython
, I’m getting
ModuleNotFoundError: No module named 'kedro'
.
j

Juan Luis

07/28/2023, 12:08 PM
so, looks like the kernel of the notebook you launched doesn't have kedro nor your project installed. can you do
Copy code
import sys
print(sys.prefix, sys.executable)
and see if it matches what you expected?
m

Mate Scharnitzky

07/28/2023, 12:09 PM
My pwd is
/home/ec2-user/SageMaker/<my project name>/notebooks/data_engineering
This is for sys prerix and executable:
Copy code
/home/ec2-user/SageMaker/pyspark /home/ec2-user/SageMaker/pyspark/bin/python
From the terminal, kedro commands work as expected
j

Juan Luis

07/28/2023, 12:18 PM
and from the terminal, what's the output of
kedro info
,
which kedro
, and
which python
@Mate Scharnitzky?
m

Mate Scharnitzky

07/28/2023, 12:19 PM
image.png
Copy code
which kedro
/anaconda3/bin/kedro
j

Juan Luis

07/28/2023, 12:21 PM
okay so we got it.
kedro
is installed in the
base
environment of Anaconda, which is not the same as
/home/ec2-user/SageMaker/pyspark
m

Mate Scharnitzky

07/28/2023, 12:21 PM
found it
yes exactly
j

Juan Luis

07/28/2023, 12:21 PM
you can try
/home/ec2-user/SageMaker/pyspark/bin/python -m pip install kedro
and then in your notebook you should be able to do
%load_ext kedro.ipython
let us know if it works
m

Mate Scharnitzky

07/28/2023, 12:21 PM
I installed it to the
pyspark
env
now it works
j

Juan Luis

07/28/2023, 12:22 PM
🎉
m

Mate Scharnitzky

07/28/2023, 12:24 PM
so loading the context now works
j

Juan Luis

07/28/2023, 12:25 PM
excellent!
m

Mate Scharnitzky

07/28/2023, 12:26 PM
then if I load the context then I load the modules from the package as well
Thank you, both! I should have known this
Sorry for that!
d

datajoely

07/28/2023, 12:26 PM
is there anything we could have added to the documentation you linked that would have helped in this situaiton?
☝🏼 1
m

Mate Scharnitzky

07/28/2023, 12:27 PM
I don’t think so
This happened because for some time my conda env
pyspark
wasn’t persistent in SageMaker
Every single time I had to create a new env if the instance was stopped before.
I did install kedro originally into the right env, but it kept disappearing hence I didn’t notice that would take it from the base env in this particular case
d

datajoely

07/28/2023, 12:30 PM
this isn’t quite the same thing, but maybe helpful
m

Mate Scharnitzky

07/28/2023, 12:31 PM
thank you!
n

Nok Lam Chan

07/28/2023, 12:44 PM
https://github.com/kedro-org/kedro/issues/2412 This is related, I put Joel’s link there for reference.
3 Views