meharji arumilli
08/01/2023, 8:46 AM*** Found local files:
*** * /opt/airflow/logs/dag_id=test-fi/run_id=scheduled__2023-07-02T08:24:20.451204+00:00/task_id=preprocess/attempt=1.log
[2023-08-01, 08:24:21 UTC] {taskinstance.py:1103} INFO - Dependencies all met for dep_context=non-requeueable deps ti=<TaskInstance: test-fi.preprocess scheduled__2023-07-02T08:24:20.451204+00:00 [queued]>
[2023-08-01, 08:24:21 UTC] {taskinstance.py:1103} INFO - Dependencies all met for dep_context=requeueable deps ti=<TaskInstance: test-fi.preprocess scheduled__2023-07-02T08:24:20.451204+00:00 [queued]>
[2023-08-01, 08:24:21 UTC] {taskinstance.py:1308} INFO - Starting attempt 1 of 2
[2023-08-01, 08:24:21 UTC] {taskinstance.py:1327} INFO - Executing <Task(KedroOperator): preprocess> on 2023-07-02 08:24:20.451204+00:00
[2023-08-01, 08:24:21 UTC] {standard_task_runner.py:57} INFO - Started process 114 to run task
[2023-08-01, 08:24:21 UTC] {standard_task_runner.py:84} INFO - Running: ['***', 'tasks', 'run', 'test-fi', 'preprocess', 'scheduled__2023-07-02T08:24:20.451204+00:00', '--job-id', '486', '--raw', '--subdir', 'DAGS_FOLDER/test_fi_dag.py', '--cfg-path', '/tmp/tmpzsz4yrlp']
[2023-08-01, 08:24:21 UTC] {standard_task_runner.py:85} INFO - Job 486: Subtask preprocess
[2023-08-01, 08:24:21 UTC] {task_command.py:410} INFO - Running <TaskInstance: test-fi.preprocess scheduled__2023-07-02T08:24:20.451204+00:00 [running]> on host 829fb522c236
[2023-08-01, 08:24:21 UTC] {taskinstance.py:1545} INFO - Exporting env vars: AIRFLOW_CTX_DAG_OWNER='***' AIRFLOW_CTX_DAG_ID='test-fi' AIRFLOW_CTX_TASK_ID='preprocess-rre' AIRFLOW_CTX_EXECUTION_DATE='2023-07-02T08:24:20.451204+00:00' AIRFLOW_CTX_TRY_NUMBER='1' AIRFLOW_CTX_DAG_RUN_ID='scheduled__2023-07-02T08:24:20.451204+00:00'
[2023-08-01, 08:24:21 UTC] {test_fi_dag.py:61} INFO - Executing task preprocess, using model version: 20230801
[2023-08-01, 08:37:16 UTC] {local_task_job_runner.py:225} INFO - Task exited with return code 1
Can anyone make a configuration suggestion that could show the complete process log in the airflow UI. Thanks!!datajoely
08/01/2023, 9:05 AMmeharji arumilli
08/01/2023, 9:06 AMdatajoely
08/01/2023, 9:07 AMlogging.yml
https://docs.kedro.org/en/stable/logging/logging.html#default-framework-side-logging-configurationmeharji arumilli
08/01/2023, 9:08 AMversion: 1
disable_existing_loggers: False
formatters:
simple:
format: "%(asctime)s - %(name)s - %(levelname)s - %(message)s"
handlers:
console:
class: logging.StreamHandler
level: INFO
formatter: simple
stream: <ext://sys.stdout>
info_file_handler:
class: logging.handlers.RotatingFileHandler
level: INFO
formatter: simple
filename: logs/info.log
maxBytes: 10485760 # 10MB
backupCount: 20
encoding: utf8
delay: True
error_file_handler:
class: logging.handlers.RotatingFileHandler
level: ERROR
formatter: simple
filename: logs/errors.log
maxBytes: 10485760 # 10MB
backupCount: 20
encoding: utf8
delay: True
loggers:
kedro:
level: INFO
test-fi:
level: INFO
root:
handlers: [info_file_handler, error_file_handler]
this is my logging.ymldatajoely
08/01/2023, 9:08 AMtail -f logs/info.log
in a different window?meharji arumilli
08/01/2023, 9:10 AMdatajoely
08/01/2023, 9:12 AMmeharji arumilli
08/01/2023, 9:14 AMdatajoely
08/01/2023, 9:16 AM