When a kedro project is run in the command line or IDE it gi

Question

When a kedro project is run in the command line or IDE it gives the logs as ` home ec2 user airflow ml logs dag id=gem models run id=manual 2023 09 28T07 51 39 714757+00 00 task id=model rre value de attempt=1 log` `2023 09 28 07 52 47 132 matplotlib font manager INFO generated new fontManager` `2023 09 28 07 57 47 503 kedro io data catalog INFO Saving data to `df preprocessed` PickleDataSet ` `2023 09 28 07 58 41 157 common methods feature engineering process categorical features WARNING Missing value in ordinal dict for feature energy class` `2023 09 28 08 07 19 535 kedro io data catalog INFO Saving data to `master table` PickleDataSet ` `2023 09 28 08 07 28 766 root INFO Starting hyperparameter tuning with feature fraction 0 7 0 8 bagging fraction 0 7 0 8 min data in leaf 10 30 lambda l1 0 5 lambda l2 0 5 min child weight 0 0001 0 99 num leaves 10 300 max depth 3 20 and num iterations 1000 early stopping rounds 5 bagging freq 1 learning rate 0 05 verbosity 1 objective regression ` `2023 09 28 09 09 00 691 kedro io data catalog INFO Saving data to `parameters final` YAMLDataSet ` `2023 09 28 09 09 46 000 root INFO Customer report created forRREvaluationwith model version 20230928 075244` ` 2023 09 28 07 51 41 UTC taskinstance py 1157 INFO Dependencies all met for dep context=non requeueable deps ti=` ` 2023 09 28 07 51 41 UTC taskinstance py 1157 INFO Dependencies all met for dep context=requeueable deps ti=` ` 2023 09 28 07 51 41 UTC taskinstance py 1359 INFO Starting attempt 1 of 2` ` 2023 09 28 07 51 41 UTC taskinstance py 1380 INFO Executing on 2023 09 28 07 51 39 714757+00 00` ` 2023 09 28 07 51 41 UTC standard task runner py 57 INFO Started process 12206 to run task` ` 2023 09 28 07 51 41 UTC standard task runner py 84 INFO Running airflow tasks run gem models model rre value de manual 2023 09 28T07 51 39 714757+00 00 job id 11 raw subdir DAGS FOLDER gem models dag py cfg path tmp tmpageoucml ` ` 2023 09 28 07 51 41 UTC standard task runner py 85 INFO Job 11 Subtask model rre value de` ` 2023 09 28 09 09 47 UTC local task job runner py 228 INFO Task exited with return code 0` When the project is run in airflow using the KedroOperator then it is giving only the below lines in the logs It is not showing the logs that come from kedro runner root in the airflow UI ` Found local files ` ` home ec2 user airflow ml logs dag id=rre value de run id=manual 2023 10 11T07 51 39 197845+00 00 task id=create report attempt=11 log` ` 2023 10 11 10 52 26 UTC taskinstance py 1157 INFO Dependencies all met for dep context=non requeueable deps ti=` ` 2023 10 11 10 52 26 UTC taskinstance py 1157 INFO Dependencies all met for dep context=requeueable deps ti=` ` 2023 10 11 10 52 26 UTC taskinstance py 1359 INFO Starting attempt 11 of 12` ` 2023 10 11 10 52 26 UTC taskinstance py 1380 INFO Executing on 2023 10 11 07 51 39 197845+00 00` ` 2023 10 11 10 52 26 UTC standard task runner py 57 INFO Started process 22175 to run task` ` 2023 10 11 10 52 26 UTC standard task runner py 84 INFO Running airflow tasks run rre value de create report manual 2023 10 11T07 51 39 197845+00 00 job id 195 raw subdir DAGS FOLDER rre value de dag py cfg path tmp tmpetucf v2 ` ` 2023 10 11 10 52 26 UTC standard task runner py 85 INFO Job 195 Subtask create report` ` 2023 10 11 10 52 26 UTC task command py 415 INFO Running on host ip 172 31 14 127 eu west 1 compute internal` ` 2023 10 11 10 52 26 UTC taskinstance py 1660 INFO Exporting env vars AIRFLOW CTX DAG OWNER= airflow AIRFLOW CTX DAG ID= rre value de AIRFLOW CTX TASK ID= create report AIRFLOW CTX EXECUTION DATE= 2023 10 11T07 51 39 197845+00 00 AIRFLOW CTX TRY NUMBER= 11 AIRFLOW CTX DAG RUN ID= manual 2023 10 11T07 51 39 197845+00 00 ` ` 2023 10 11 10 52 27 UTC store py 32 INFO `read ` not implemented for `BaseSessionStore` Assuming empty store ` ` 2023 10 11 10 52 40 UTC local task job runner py 228 INFO Task exited with return code 0` The KedroOperator is generated from the kedro airflow v0 5 1 which looks like below `class KedroOperator BaseOperator ` ` apply defaults` `def init ` `self ` `package name str ` `pipeline name str ` `node name str ` `project path str ` `env str ` ` args kwargs` ` > None ` `super init args kwargs ` `self package name = package name` `self pipeline name = pipeline name` `self node name = node name` `self project path = project path` `self env = env` `def execute self context ` `configure project self package name ` `with KedroSession create self package name ` `self project path ` `env=self env as session ` `session run self pipeline name node names= self node name ` It appears like the tasks executed by the KedroOperator are not generating or capturing the logs Any help to capture the logs from KedroOperator tasks to the airflow UI kedro airflow

meharji arumilli · Answer

Hi Any fix on this Does the `execute`method of class `KedroOperator` lack the functionality to emit the logs Using kedro version 0 18 1 and `kedro airflow create` command to create the DAG

Juan Luis · Answer

hi < meharji arumilli> there have been some changes in the logging are you able to upgrade to Kedro 0 18 14

Dmitry Sorokin · Answer

< meharji arumilli> hi also what is your kedro logging configuration

meharji arumilli · Answer

< Dmitry Sorokin> here is my logging ```version 1 disable existing loggers True formatters simple format asctime s name s levelname s message s json formatter format asctime s name s levelname s message s class pythonjsonlogger jsonlogger JsonFormatter handlers console class logging StreamHandler level INFO formatter simple stream info file handler class logging handlers RotatingFileHandler level INFO formatter simple filename logs info log maxBytes 10485760 10MB backupCount 20 encoding utf8 delay True error file handler class logging handlers RotatingFileHandler level ERROR formatter simple filename logs errors log maxBytes 10485760 10MB backupCount 20 encoding utf8 delay True loggers anyconfig level WARNING handlers console info file handler error file handler propagate no level INFO handlers console info file handler error file handler propagate no kedro pipeline level INFO handlers console info file handler error file handler propagate no kedro journal level INFO propagate no root level INFO handlers console info file handler error file handler ```

meharji arumilli · Answer

< Juan Luis> yes gave a try with 0 18 14 still the same behaviour KedroOperator is not capturing the logs that come from kedro runner root in the airflow UI

meharji arumilli · Answer

To add more information when the kedro project is dockerized and run as Docker image in airflow using DockerOperator it fetches the complete logs from base py and docker py file sources as shown in the screenshot

meharji arumilli · Answer

When it is packaged as whl and run using KedroOperator it does not give the kedro task project specific logs and emits the logs only from the file sources shown in the screen shot It certainly sounds like KedroOperator is not configured enough to capture the logs As it does not show base py in the file sources for the logs Any help from the kedro engineering or developer community will be highly appreciated as it is crucial to have logs seen in the airflow UI inorder to use kedro airflow or kedro package plugins to its full capability

meharji arumilli · Answer

Does adding kedroOperator to airflow providers could have a solution to this logging issue Has it been in the pipeline to be included < Nok Lam Chan> < datajoely> < Merel>

meharji arumilli · Answer

< Dmitry Sorokin> does the logging yml look fine

Nok Lam Chan · Answer

Can you try to use the default logging that Kedro generate for you It looks like you removed the project logging thus you can t see it when it s packaged as a wheel

Nok Lam Chan · Answer

And I see `kedro journal` which signals that your logging file is quite old

Nok Lam Chan · Answer

Can you try a new kedro project with airflow to see if logging shows up

Nok Lam Chan · Answer

Re <https github com kedro org kedro issues 1717> There isn t any progress for this ticket yet it s not related to the problem as it only help discoverability to put this KedroOperator into `airflow` codebase it doesn t change any functionalities

meharji arumilli · Answer

< Nok Lam Chan> I did created a new logging file with the kedro new command which looks like ```version 1 disable existing loggers False formatters simple format asctime s name s levelname s message s json formatter format asctime s name s levelname s message s class pythonjsonlogger jsonlogger JsonFormatter handlers console class logging StreamHandler level INFO formatter simple stream info file handler class logging handlers RotatingFileHandler level INFO formatter simple filename logs info log maxBytes 10485760 10MB backupCount 20 encoding utf8 delay True error file handler class logging handlers RotatingFileHandler level ERROR formatter simple filename logs errors log maxBytes 10485760 10MB backupCount 20 encoding utf8 delay True loggers anyconfig level WARNING handlers console info file handler error file handler propagate no kedro io level INFO handlers console info file handler error file handler propagate no kedro pipeline level INFO handlers console info file handler error file handler propagate no root level INFO handlers console info file handler error file handler ``` And then `kedro package` is run to package the project However the logging yml is in the conf so it is not included in the packaged wheel file The conf is copied to the instance running the DAG and its path is specified to the conf source as `def execute self context ` `36 Starting execution of KedroOperator ` `37 configure project self package name ` `38 with KedroSession create self package name ` `39 self project path ` `40 env=self env extra params=self extra params ` `41 conf source= home ec2 user airflow ml conf models as session ` `42 session run self pipeline name node names= self node name ` `43 Execution of KedroOperator completed ` Now the log gives the error ` Found local files ` ` home ec2 user airflow ml logs dag id=rre value de run id=manual 2023 10 27T08 21 46 405139+00 00 task id=define project parameters attempt=2 log` ` 2023 10 27 08 26 57 UTC taskinstance py 1157 INFO Dependencies all met for dep context=non requeueable deps ti=` ` 2023 10 27 08 26 57 UTC taskinstance py 1157 INFO Dependencies all met for dep context=requeueable deps ti=` ` 2023 10 27 08 26 57 UTC taskinstance py 1359 INFO Starting attempt 2 of 2` ` 2023 10 27 08 26 57 UTC taskinstance py 1380 INFO Executing on 2023 10 27 08 21 46 405139+00 00` ` 2023 10 27 08 26 57 UTC standard task runner py 57 INFO Started process 12975 to run task` ` 2023 10 27 08 26 57 UTC standard task runner py 84 INFO Running airflow tasks run rre value de define project parameters manual 2023 10 27T08 21 46 405139+00 00 job id 259 raw subdir DAGS FOLDER rre value de dag py cfg path tmp tmp8w8z 1aw ` ` 2023 10 27 08 26 57 UTC standard task runner py 85 INFO Job 259 Subtask define project parameters` ` 2023 10 27 08 26 57 UTC task command py 415 INFO Running on host ip 172 31 14 127 eu west 1 compute internal` ` 2023 10 27 08 26 58 UTC taskinstance py 1660 INFO Exporting env vars AIRFLOW CTX DAG OWNER= airflow AIRFLOW CTX DAG ID= rre value de AIRFLOW CTX TASK ID= define project parameters AIRFLOW CTX EXECUTION DATE= 2023 10 27T08 21 46 405139+00 00 AIRFLOW CTX TRY NUMBER= 2 AIRFLOW CTX DAG RUN ID= manual 2023 10 27T08 21 46 405139+00 00 ` ` 2023 10 27 08 26 58 UTC rre value de dag py 36 INFO Starting execution of KedroOperator` ` 2023 10 27 08 26 58 UTC logging mixin py 151 INFO rre value` ` 2023 10 27 08 26 58 UTC logging mixin py 151 INFO Model version 20231027 112147` ` 2023 10 27 08 26 58 UTC logging mixin py 151 INFO Project rre value` ` 2023 10 27 08 26 58 UTC logging mixin py 151 INFO rre value` ` 2023 10 27 08 26 58 UTC local task job runner py 228 INFO Task exited with return code 6` I wonder what does the error code 6 mean here

meharji arumilli · Answer

< Nok Lam Chan> could you kindly give comments to try anything else if we are in same page

Nok Lam Chan · Answer

< meharji arumilli> Which version of Kedro are you using

meharji arumilli · Answer

kedro version 0 18 14

Nok Lam Chan · Answer

The logging file you show is quite old

Nok Lam Chan · Answer

<https github com kedro org kedro starters blob main astro airflow iris 7B 7B 20cookiecutter repo name 20 7D 7D conf base logging yml> It should be more close to this

Nok Lam Chan · Answer

gt I wonder what does the error code 6 mean here I have no idea this seem to be something come from Airflow instead of Kedro gt Now the log gives the error ` Found local files ` It s all INFO logging can you point me to the error