meharji arumilli
10/09/2023, 11:56 AMpip3 install test_fi-0.1-py3-none-any.whl
into the EC2 instance where airflow is installed in Step 3. I have tested to run in the EC2 command line, `python -m test_fi`and it gets executed. However, when the DAG is triggered it throws the error File "<frozen importlib._bootstrap>", line 973, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'test_fi'
in the airflow UI. What could be the issue here. Can anyone help to fix this? @Merel @Ankita KatiyarAnkita Katiyar
10/09/2023, 12:22 PMmeharji arumilli
10/09/2023, 12:32 PM*** Found local files:
*** * /home/ec2-user/airflow_ml/logs/dag_id=test-fi/run_id=manual__2023-10-09T12:15:39.798547+00:00/task_id=define-project-parameters/attempt=2.log
[2023-10-09, 12:20:43 UTC] {taskinstance.py:1157} INFO - Dependencies all met for dep_context=non-requeueable deps ti=<TaskInstance: test-fi.define-project-parameters manual__2023-10-09T12:15:39.798547+00:00 [queued]>
[2023-10-09, 12:20:43 UTC] {taskinstance.py:1157} INFO - Dependencies all met for dep_context=requeueable deps ti=<TaskInstance: test-fi.define-project-parameters manual__2023-10-09T12:15:39.798547+00:00 [queued]>
[2023-10-09, 12:20:43 UTC] {taskinstance.py:1359} INFO - Starting attempt 2 of 2
[2023-10-09, 12:20:43 UTC] {taskinstance.py:1380} INFO - Executing <Task(KedroOperator): define-project-parameters> on 2023-10-09 12:15:39.798547+00:00
[2023-10-09, 12:20:43 UTC] {standard_task_runner.py:57} INFO - Started process 25099 to run task
[2023-10-09, 12:20:43 UTC] {standard_task_runner.py:84} INFO - Running: ['airflow', 'tasks', 'run', 'test-fi', 'define-project-parameters', 'manual__2023-10-09T12:15:39.798547+00:00', '--job-id', '47', '--raw', '--subdir', 'DAGS_FOLDER/test_fi.py', '--cfg-path', '/tmp/tmps09iubre']
[2023-10-09, 12:20:43 UTC] {standard_task_runner.py:85} INFO - Job 47: Subtask define-project-parameters
[2023-10-09, 12:20:43 UTC] {task_command.py:415} INFO - Running <TaskInstance: test-fi.define-project-parameters manual__2023-10-09T12:15:39.798547+00:00 [running]> on host ip-172-31-14-127.eu-west-1.compute.internal
[2023-10-09, 12:20:43 UTC] {taskinstance.py:1660} INFO - Exporting env vars: AIRFLOW_CTX_DAG_OWNER='airflow' AIRFLOW_CTX_DAG_ID='test-fi' AIRFLOW_CTX_TASK_ID='define-project-parameters' AIRFLOW_CTX_EXECUTION_DATE='2023-10-09T12:15:39.798547+00:00' AIRFLOW_CTX_TRY_NUMBER='2' AIRFLOW_CTX_DAG_RUN_ID='manual__2023-10-09T12:15:39.798547+00:00'
[2023-10-09, 12:20:43 UTC] {taskinstance.py:1935} ERROR - Task failed with exception
Traceback (most recent call last):
File "/home/ec2-user/airflow_ml/dags/test-fi.py", line 36, in execute
with KedroSession.create(self.package_name,
File "/home/ec2-user/airflow_ml/.venv/lib64/python3.8/site-packages/kedro/framework/session/session.py", line 138, in create
validate_settings()
File "/home/ec2-user/airflow_ml/.venv/lib64/python3.8/site-packages/kedro/framework/project/__init__.py", line 223, in validate_settings
importlib.import_module(f"{PACKAGE_NAME}.settings")
File "/usr/lib64/python3.8/importlib/__init__.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
File "<frozen importlib._bootstrap>", line 991, in _find_and_load
File "<frozen importlib._bootstrap>", line 961, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
File "<frozen importlib._bootstrap>", line 991, in _find_and_load
File "<frozen importlib._bootstrap>", line 973, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'test_fi'
[2023-10-09, 12:20:43 UTC] {taskinstance.py:1398} INFO - Marking task as FAILED. dag_id=test-fi, task_id=define-project-parameters, execution_date=20231009T121539, start_date=20231009T122043, end_date=20231009T122043
[2023-10-09, 12:20:43 UTC] {standard_task_runner.py:104} ERROR - Failed to execute job 47 for task define-project-parameters (No module named 'test_fi'; 25099)
[2023-10-09, 12:20:43 UTC] {local_task_job_runner.py:228} INFO - Task exited with return code 1
@Ankita Katiyar here it isAnkita Katiyar
10/09/2023, 12:41 PMmeharji arumilli
10/10/2023, 8:54 AMread()
not implemented for BaseSessionStore
. Assuming empty store.
2023-10-10 113730,331 - kedro.framework.session.store - INFO - save()
not implemented for BaseSessionStore
. Skipping the step.
Traceback (most recent call last):
File "/Users/opt/anaconda3/envs/gem-models/bin/kedro", line 8, in <module>
sys.exit(main())
File "/Users/opt/anaconda3/envs/gem-models/lib/python3.8/site-packages/kedro/framework/cli/cli.py", line 217, in main
cli_collection()
File "/Users/opt/anaconda3/envs/gem-models/lib/python3.8/site-packages/click/core.py", line 1157, in call
return self.main(*args, **kwargs)
File "/Users/opt/anaconda3/envs/gem-models/lib/python3.8/site-packages/kedro/framework/cli/cli.py", line 145, in main
super().main(
File "/Users/opt/anaconda3/envs/gem-models/lib/python3.8/site-packages/click/core.py", line 1078, in main
rv = self.invoke(ctx)
File "/Users/opt/anaconda3/envs/gem-models/lib/python3.8/site-packages/click/core.py", line 1688, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/Users/opt/anaconda3/envs/gem-models/lib/python3.8/site-packages/click/core.py", line 1688, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/Users/opt/anaconda3/envs/gem-models/lib/python3.8/site-packages/click/core.py", line 1434, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/Users/opt/anaconda3/envs/gem-models/lib/python3.8/site-packages/click/core.py", line 783, in invoke
return __callback(*args, **kwargs)
File "/Users/opt/anaconda3/envs/gem-models/lib/python3.8/site-packages/click/decorators.py", line 45, in new_func
return f(get_current_context().obj, *args, **kwargs)
File "/Users/opt/anaconda3/envs/gem-models/lib/python3.8/site-packages/kedro_airflow/plugin.py", line 104, in create
dag_config = _load_config(context, pipeline_name)
File "/Users/opt/anaconda3/envs/gem-models/lib/python3.8/site-packages/kedro_airflow/plugin.py", line 37, in _load_config
if "airflow" not in context.config_loader.config_patterns.keys():
AttributeError: 'MyTemplatedConfigLoader' object has no attribute 'config_patterns'Ankita Katiyar
10/10/2023, 8:59 AMkedro-airflow
to less than 0.6.0
. There is a fix in the works soon to offer backwards compatibilitymeharji arumilli
10/10/2023, 9:05 AMNok Lam Chan
10/10/2023, 11:57 AM