Hi all i am using kedro airflow plugin to convert kedro proj

Question

Hi all i am using kedro airflow plugin to convert kedro project into airflow project I have executed steps 1 3 as in the documentation In my case i have installed `pip3 install test fi 0 1 py3 none any whl` into the EC2 instance where airflow is installed in Step 3 I have tested to run in the EC2 command line `python m test fi`and it gets executed However when the DAG is triggered it throws the error `File lt frozen importlib bootstrap gt line 973 in find and load unlocked` `ModuleNotFoundError No module named test fi ` in the airflow UI What could be the issue here Can anyone help to fix this < Merel> < Ankita Katiyar>

Ankita Katiyar · Answer

Can you share a full stacktrace for this error

meharji arumilli · Answer

` Found local files ` ` home ec2 user airflow ml logs dag id=test fi run id=manual 2023 10 09T12 15 39 798547+00 00 task id=define project parameters attempt=2 log` ` 2023 10 09 12 20 43 UTC taskinstance py 1157 INFO Dependencies all met for dep context=non requeueable deps ti= lt TaskInstance test fi define project parameters manual 2023 10 09T12 15 39 798547+00 00 queued gt ` ` 2023 10 09 12 20 43 UTC taskinstance py 1157 INFO Dependencies all met for dep context=requeueable deps ti= lt TaskInstance test fi define project parameters manual 2023 10 09T12 15 39 798547+00 00 queued gt ` ` 2023 10 09 12 20 43 UTC taskinstance py 1359 INFO Starting attempt 2 of 2` ` 2023 10 09 12 20 43 UTC taskinstance py 1380 INFO Executing lt Task KedroOperator define project parameters gt on 2023 10 09 12 15 39 798547+00 00` ` 2023 10 09 12 20 43 UTC standard task runner py 57 INFO Started process 25099 to run task` ` 2023 10 09 12 20 43 UTC standard task runner py 84 INFO Running airflow tasks run test fi define project parameters manual 2023 10 09T12 15 39 798547+00 00 job id 47 raw subdir DAGS FOLDER test fi py cfg path tmp tmps09iubre ` ` 2023 10 09 12 20 43 UTC standard task runner py 85 INFO Job 47 Subtask define project parameters` ` 2023 10 09 12 20 43 UTC task command py 415 INFO Running lt TaskInstance test fi define project parameters manual 2023 10 09T12 15 39 798547+00 00 running gt on host ip 172 31 14 127 eu west 1 compute internal` ` 2023 10 09 12 20 43 UTC taskinstance py 1660 INFO Exporting env vars AIRFLOW CTX DAG OWNER= airflow AIRFLOW CTX DAG ID= test fi AIRFLOW CTX TASK ID= define project parameters AIRFLOW CTX EXECUTION DATE= 2023 10 09T12 15 39 798547+00 00 AIRFLOW CTX TRY NUMBER= 2 AIRFLOW CTX DAG RUN ID= manual 2023 10 09T12 15 39 798547+00 00 ` ` 2023 10 09 12 20 43 UTC taskinstance py 1935 ERROR Task failed with exception` `Traceback most recent call last ` `File home ec2 user airflow ml dags test fi py line 36 in execute` `with KedroSession create self package name ` `File home ec2 user airflow ml venv lib64 python3 8 site packages kedro framework session session py line 138 in create` `validate settings ` `File home ec2 user airflow ml venv lib64 python3 8 site packages kedro framework project init py line 223 in validate settings` `importlib import module f PACKAGE NAME settings ` `File usr lib64 python3 8 importlib init py line 127 in import module` `return bootstrap gcd import name level package level ` `File lt frozen importlib bootstrap gt line 1014 in gcd import` `File lt frozen importlib bootstrap gt line 991 in find and load` `File lt frozen importlib bootstrap gt line 961 in find and load unlocked` `File lt frozen importlib bootstrap gt line 219 in call with frames removed` `File lt frozen importlib bootstrap gt line 1014 in gcd import` `File lt frozen importlib bootstrap gt line 991 in find and load` `File lt frozen importlib bootstrap gt line 973 in find and load unlocked` `ModuleNotFoundError No module named test fi ` ` 2023 10 09 12 20 43 UTC taskinstance py 1398 INFO Marking task as FAILED dag id=test fi task id=define project parameters execution date=20231009T121539 start date=20231009T122043 end date=20231009T122043` ` 2023 10 09 12 20 43 UTC standard task runner py 104 ERROR Failed to execute job 47 for task define project parameters No module named test fi 25099 ` ` 2023 10 09 12 20 43 UTC local task job runner py 228 INFO Task exited with return code 1` < Ankita Katiyar> here it is

meharji arumilli · Answer

got it solved the package is not installed in the virtual env in which airflow is running

Ankita Katiyar · Answer

Ah perfect reach out if you face any more problems slightly smiling face

meharji arumilli · Answer

< Ankita Katiyar> kedro airflow create used to earlier to create dag Currently it gives the error $ kedro airflow create 2023 10 10 11 37 16 588 kedro framework session store INFO `read ` not implemented for `BaseSessionStore` Assuming empty store 2023 10 10 11 37 30 331 kedro framework session store INFO `save ` not implemented for `BaseSessionStore` Skipping the step Traceback most recent call last File Users opt anaconda3 envs gem models bin kedro line 8 in lt module gt sys exit main File Users opt anaconda3 envs gem models lib python3 8 site packages kedro framework cli cli py line 217 in main cli collection File Users opt anaconda3 envs gem models lib python3 8 site packages click core py line 1157 in call return self main args kwargs File Users opt anaconda3 envs gem models lib python3 8 site packages kedro framework cli cli py line 145 in main super main File Users opt anaconda3 envs gem models lib python3 8 site packages click core py line 1078 in main rv = self invoke ctx File Users opt anaconda3 envs gem models lib python3 8 site packages click core py line 1688 in invoke return process result sub ctx command invoke sub ctx File Users opt anaconda3 envs gem models lib python3 8 site packages click core py line 1688 in invoke return process result sub ctx command invoke sub ctx File Users opt anaconda3 envs gem models lib python3 8 site packages click core py line 1434 in invoke return ctx invoke self callback ctx params File Users opt anaconda3 envs gem models lib python3 8 site packages click core py line 783 in invoke return callback args kwargs File Users opt anaconda3 envs gem models lib python3 8 site packages click decorators py line 45 in new func return f get current context obj args kwargs File Users opt anaconda3 envs gem models lib python3 8 site packages kedro airflow plugin py line 104 in create dag config = load config context pipeline name File Users opt anaconda3 envs gem models lib python3 8 site packages kedro airflow plugin py line 37 in load config if airflow not in context config loader config patterns keys AttributeError MyTemplatedConfigLoader object has no attribute config patterns

meharji arumilli · Answer

im using kedro==0 18 1

Ankita Katiyar · Answer

This is a known problem unfortunately <https github com kedro org kedro plugins issues 321> You either have to upgrade Kedro to version 0 18 3 or higher or downgrade `kedro airflow` to less than `0 6 0` There is a fix in the works soon to offer backwards compatibility

meharji arumilli · Answer

okay was suspecting some version compatibility and would like to ask that the logging in 0 18 3 does not allow to click on the traceback code is different from 0 18 1 allows to click on the traceback lines due to which we sticked to 0 18 1 Is there already an updated version that allows to click on the error traceback

Nok Lam Chan · Answer

< meharji arumilli> I would suggest try to upgrade your IDE and see if it works there is no change in Kedro Recent report from users newer version of Pycharm works fine VScode has always been working for me