Hi all, i am using kedro-airflow plugin to convert kedro project into airflow project. I have executed steps 1-3 as in the documentation https://github.com/kedro-org/kedro-plugins/tree/main/kedro-airflow In my case, i have installed

pip3 install test_fi-0.1-py3-none-any.whl

into the EC2 instance where airflow is installed in Step 3. I have tested to run in the EC2 command line, `python -m test_fi`and it gets executed. However, when the DAG is triggered it throws the error

File "<frozen importlib._bootstrap>", line 973, in _find_and_load_unlocked

ModuleNotFoundError: No module named 'test_fi'

in the airflow UI. What could be the issue here. Can anyone help to fix this? @Merel @Ankita Katiyar

Can you share a full stacktrace for this error?

*** Found local files:

***   * /home/ec2-user/airflow_ml/logs/dag_id=test-fi/run_id=manual__2023-10-09T12:15:39.798547+00:00/task_id=define-project-parameters/attempt=2.log

[2023-10-09, 12:20:43 UTC] {taskinstance.py:1157} INFO - Dependencies all met for dep_context=non-requeueable deps ti=<TaskInstance: test-fi.define-project-parameters manual__2023-10-09T12:15:39.798547+00:00 [queued]>

[2023-10-09, 12:20:43 UTC] {taskinstance.py:1157} INFO - Dependencies all met for dep_context=requeueable deps ti=<TaskInstance: test-fi.define-project-parameters manual__2023-10-09T12:15:39.798547+00:00 [queued]>

[2023-10-09, 12:20:43 UTC] {taskinstance.py:1359} INFO - Starting attempt 2 of 2

[2023-10-09, 12:20:43 UTC] {taskinstance.py:1380} INFO - Executing <Task(KedroOperator): define-project-parameters> on 2023-10-09 12:15:39.798547+00:00

[2023-10-09, 12:20:43 UTC] {standard_task_runner.py:57} INFO - Started process 25099 to run task

[2023-10-09, 12:20:43 UTC] {standard_task_runner.py:84} INFO - Running: ['airflow', 'tasks', 'run', 'test-fi', 'define-project-parameters', 'manual__2023-10-09T12:15:39.798547+00:00', '--job-id', '47', '--raw', '--subdir', 'DAGS_FOLDER/test_fi.py', '--cfg-path', '/tmp/tmps09iubre']

[2023-10-09, 12:20:43 UTC] {standard_task_runner.py:85} INFO - Job 47: Subtask define-project-parameters

[2023-10-09, 12:20:43 UTC] {task_command.py:415} INFO - Running <TaskInstance: test-fi.define-project-parameters manual__2023-10-09T12:15:39.798547+00:00 [running]> on host ip-172-31-14-127.eu-west-1.compute.internal

[2023-10-09, 12:20:43 UTC] {taskinstance.py:1660} INFO - Exporting env vars: AIRFLOW_CTX_DAG_OWNER='airflow' AIRFLOW_CTX_DAG_ID='test-fi' AIRFLOW_CTX_TASK_ID='define-project-parameters' AIRFLOW_CTX_EXECUTION_DATE='2023-10-09T12:15:39.798547+00:00' AIRFLOW_CTX_TRY_NUMBER='2' AIRFLOW_CTX_DAG_RUN_ID='manual__2023-10-09T12:15:39.798547+00:00'

[2023-10-09, 12:20:43 UTC] {taskinstance.py:1935} ERROR - Task failed with exception

Traceback (most recent call last):

File "/home/ec2-user/airflow_ml/dags/test-fi.py", line 36, in execute

with KedroSession.create(self.package_name,

File "/home/ec2-user/airflow_ml/.venv/lib64/python3.8/site-packages/kedro/framework/session/session.py", line 138, in create

validate_settings()

File "/home/ec2-user/airflow_ml/.venv/lib64/python3.8/site-packages/kedro/framework/project/__init__.py", line 223, in validate_settings

importlib.import_module(f"{PACKAGE_NAME}.settings")

File "/usr/lib64/python3.8/importlib/__init__.py", line 127, in import_module

return _bootstrap._gcd_import(name[level:], package, level)

File "<frozen importlib._bootstrap>", line 1014, in _gcd_import

File "<frozen importlib._bootstrap>", line 991, in _find_and_load

File "<frozen importlib._bootstrap>", line 961, in _find_and_load_unlocked

File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed

File "<frozen importlib._bootstrap>", line 1014, in _gcd_import

File "<frozen importlib._bootstrap>", line 991, in _find_and_load

File "<frozen importlib._bootstrap>", line 973, in _find_and_load_unlocked

ModuleNotFoundError: No module named 'test_fi'

[2023-10-09, 12:20:43 UTC] {taskinstance.py:1398} INFO - Marking task as FAILED. dag_id=test-fi, task_id=define-project-parameters, execution_date=20231009T121539, start_date=20231009T122043, end_date=20231009T122043

[2023-10-09, 12:20:43 UTC] {standard_task_runner.py:104} ERROR - Failed to execute job 47 for task define-project-parameters (No module named 'test_fi'; 25099)

[2023-10-09, 12:20:43 UTC] {local_task_job_runner.py:228} INFO - Task exited with return code 1

@Ankita Katiyar here it is

Ah perfect, reach out if you face any more problems! 🙂

@Ankita Katiyar kedro airflow create used to earlier to create dag. Currently, it gives the error: $ kedro airflow create 2023-10-10 113716,588 - kedro.framework.session.store - INFO -

read()

not implemented for

BaseSessionStore

. Assuming empty store. 2023-10-10 113730,331 - kedro.framework.session.store - INFO -

save()

not implemented for

BaseSessionStore

. Skipping the step. Traceback (most recent call last): File "/Users/opt/anaconda3/envs/gem-models/bin/kedro", line 8, in <module> sys.exit(main()) File "/Users/opt/anaconda3/envs/gem-models/lib/python3.8/site-packages/kedro/framework/cli/cli.py", line 217, in main cli_collection() File "/Users/opt/anaconda3/envs/gem-models/lib/python3.8/site-packages/click/core.py", line 1157, in call return self.main(*args, **kwargs) File "/Users/opt/anaconda3/envs/gem-models/lib/python3.8/site-packages/kedro/framework/cli/cli.py", line 145, in main super().main( File "/Users/opt/anaconda3/envs/gem-models/lib/python3.8/site-packages/click/core.py", line 1078, in main rv = self.invoke(ctx) File "/Users/opt/anaconda3/envs/gem-models/lib/python3.8/site-packages/click/core.py", line 1688, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/Users/opt/anaconda3/envs/gem-models/lib/python3.8/site-packages/click/core.py", line 1688, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/Users/opt/anaconda3/envs/gem-models/lib/python3.8/site-packages/click/core.py", line 1434, in invoke return ctx.invoke(self.callback, **ctx.params) File "/Users/opt/anaconda3/envs/gem-models/lib/python3.8/site-packages/click/core.py", line 783, in invoke return __callback(*args, **kwargs) File "/Users/opt/anaconda3/envs/gem-models/lib/python3.8/site-packages/click/decorators.py", line 45, in new_func return f(get_current_context().obj, *args, **kwargs) File "/Users/opt/anaconda3/envs/gem-models/lib/python3.8/site-packages/kedro_airflow/plugin.py", line 104, in create dag_config = _load_config(context, pipeline_name) File "/Users/opt/anaconda3/envs/gem-models/lib/python3.8/site-packages/kedro_airflow/plugin.py", line 37, in _load_config if "airflow" not in context.config_loader.config_patterns.keys(): AttributeError: 'MyTemplatedConfigLoader' object has no attribute 'config_patterns'

This is a known problem unfortunately https://github.com/kedro-org/kedro-plugins/issues/321. You either have to upgrade Kedro to version 0.18.3 or higher or downgrade

kedro-airflow

to less than

0.6.0

. There is a fix in the works soon to offer backwards compatibility

okay, was suspecting some version compatibility and would like to ask that the logging in 0.18.3 (does not allow to click on the traceback code) is different from 0.18.1 (allows to click on the traceback lines), due to which we sticked to 0.18.1. Is there already an updated version that allows to click on the error traceback

@meharji arumilli I would suggest try to upgrade your IDE and see if it works, there is no change in Kedro. Recent report from users newer version of Pycharm works fine, VScode has always been working for me.