Hello all I have packaged the kedro project as `kedro packag

Question

Hello all I have packaged the kedro project as `kedro package` and created dag `kedro airflow create` This created the whl and dags Then using the below docker file i have built the docker image for the kedro project `FROM apache airflow 2 6 3 python3 8` ` install project requirements` `WORKDIR opt test fi ` `COPY src requirements txt ` `USER root` `RUN chmod R a+rwx opt test fi ` ` Install necessary packages` `RUN sudo apt get update amp amp apt get install y wget gnupg2 libgomp1 amp amp apt get y install git` `USER airflow` `COPY data data ` `COPY conf conf ` `COPY logs logs ` `COPY src src ` `COPY output output ` `COPY dist dist ` `COPY pyproject toml ` `RUN mount=type=bind src= env dst=conf env conf env amp amp python m pip install upgrade pip amp amp python m pip install r requirements txt amp amp python m pip install dist test fi 0 1 py3 none any whl` `EXPOSE 8888` `CMD kedro run ` The docker image is built as `docker build t test fi ` Then i have installed airflow using docker compose yml file in EC2 instance And attached the docker image to the worker and scheduler services Tested the docker image test fi by docker exec into the container and ran the command `kedro run`and the project runs as expected However with the airflow when the dag is triggered i get the below error in airflow UI without much information in logs to debug The below log is using `logging level = DEBUG` ` Found local files ` ` opt airflow logs dag id=test fi run id=scheduled 2023 06 27T14 37 54 602904+00 00 task id=define project parameters attempt=1 log` ` 2023 07 27 14 37 56 UTC taskinstance py 1037 DEBUG previous execution date was called` ` 2023 07 27 14 37 56 UTC init py 51 DEBUG Loading core task runner StandardTaskRunner` ` 2023 07 27 14 37 56 UTC taskinstance py 1037 DEBUG previous execution date was called` ` 2023 07 27 14 37 56 UTC base task runner py 68 DEBUG Planning to run as the user` ` 2023 07 27 14 37 56 UTC taskinstance py 789 DEBUG Refreshing TaskInstance lt TaskInstance test fi define project parameters scheduled 2023 06 27T14 37 54 602904+00 00 queued gt from DB` ` 2023 07 27 14 37 56 UTC taskinstance py 1112 DEBUG lt TaskInstance test fi define project parameters scheduled 2023 06 27T14 37 54 602904+00 00 queued gt dependency Trigger Rule PASSED True The task instance did not have any upstream tasks ` ` 2023 07 27 14 37 56 UTC taskinstance py 1112 DEBUG lt TaskInstance test fi define project parameters scheduled 2023 06 27T14 37 54 602904+00 00 queued gt dependency Not In Retry Period PASSED True The task instance was not marked for retrying ` ` 2023 07 27 14 37 56 UTC taskinstance py 1112 DEBUG lt TaskInstance test fi define project parameters scheduled 2023 06 27T14 37 54 602904+00 00 queued gt dependency Previous Dagrun State PASSED True The task did not have depends on past set ` ` 2023 07 27 14 37 56 UTC taskinstance py 1112 DEBUG lt TaskInstance test fi define project parameters scheduled 2023 06 27T14 37 54 602904+00 00 queued gt dependency Task Instance State PASSED True Task state queued was valid ` ` 2023 07 27 14 37 56 UTC taskinstance py 1112 DEBUG lt TaskInstance test fi define project parameters scheduled 2023 06 27T14 37 54 602904+00 00 queued gt dependency Task Instance Not Running PASSED True Task is not in running state ` ` 2023 07 27 14 37 56 UTC taskinstance py 1103 INFO Dependencies all met for dep context=non requeueable deps ti= lt TaskInstance test fi define project parameters scheduled 2023 06 27T14 37 54 602904+00 00 queued gt ` ` 2023 07 27 14 37 56 UTC taskinstance py 1112 DEBUG lt TaskInstance test fi define project parameters scheduled 2023 06 27T14 37 54 602904+00 00 queued gt dependency Trigger Rule PASSED True The task instance did not have any upstream tasks ` ` 2023 07 27 14 37 56 UTC taskinstance py 1112 DEBUG lt TaskInstance test fi define project parameters scheduled 2023 06 27T14 37 54 602904+00 00 queued gt dependency Task Concurrency PASSED True Task concurrency is not set ` ` 2023 07 27 14 37 56 UTC taskinstance py 1112 DEBUG lt TaskInstance test fi define project parameters scheduled 2023 06 27T14 37 54 602904+00 00 queued gt dependency Not In Retry Period PASSED True The task instance was not marked for retrying ` ` 2023 07 27 14 37 56 UTC taskinstance py 1112 DEBUG lt TaskInstance test fi define project parameters scheduled 2023 06 27T14 37 54 602904+00 00 queued gt dependency Previous Dagrun State PASSED True The task did not have depends on past set ` ` 2023 07 27 14 37 56 UTC taskinstance py 1112 DEBUG lt TaskInstance test fi define project parameters scheduled 2023 06 27T14 37 54 602904+00 00 queued gt dependency Pool Slots Available PASSED True There are enough open slots in default pool to execute the task` ` 2023 07 27 14 37 56 UTC taskinstance py 1103 INFO Dependencies all met for dep context=requeueable deps ti= lt TaskInstance test fi define project parameters scheduled 2023 06 27T14 37 54 602904+00 00 queued gt ` ` 2023 07 27 14 37 56 UTC taskinstance py 1308 INFO Starting attempt 1 of 2` ` 2023 07 27 14 37 56 UTC taskinstance py 1327 INFO Executing lt Task KedroOperator define project parameters gt on 2023 06 27 14 37 54 602904+00 00` ` 2023 07 27 14 37 56 UTC standard task runner py 57 INFO Started process 85 to run task` ` 2023 07 27 14 37 56 UTC standard task runner py 84 INFO Running tasks run test fi define project parameters scheduled 2023 06 27T14 37 54 602904+00 00 job id 884 raw subdir DAGS FOLDER test fi dag py cfg path tmp tmpu1fp72mc ` ` 2023 07 27 14 37 56 UTC standard task runner py 85 INFO Job 884 Subtask define project parameters` ` 2023 07 27 14 37 56 UTC cli action loggers py 65 DEBUG Calling callbacks lt function default action log at 0x7f4f6b6038b0 gt ` ` 2023 07 27 14 37 56 UTC taskinstance py 1037 DEBUG previous execution date was called` ` 2023 07 27 14 37 56 UTC task command py 410 INFO Running lt TaskInstance test fi define project parameters scheduled 2023 06 27T14 37 54 602904+00 00 running gt on host e1be34e2e4d4` ` 2023 07 27 14 37 56 UTC settings py 353 DEBUG Disposing DB connection pool PID 85 ` ` 2023 07 27 14 37 56 UTC settings py 212 DEBUG Setting up DB connection pool PID 85 ` ` 2023 07 27 14 37 56 UTC settings py 285 DEBUG settings prepare engine args Using NullPool` ` 2023 07 27 14 37 56 UTC taskinstance py 789 DEBUG Refreshing TaskInstance lt TaskInstance test fi define project parameters scheduled 2023 06 27T14 37 54 602904+00 00 running gt from DB` ` 2023 07 27 14 37 56 UTC taskinstance py 1037 DEBUG previous execution date was called` ` 2023 07 27 14 37 56 UTC taskinstance py 868 DEBUG Clearing XCom data` ` 2023 07 27 14 37 56 UTC retries py 80 DEBUG Running RenderedTaskInstanceFields write with retries Try 1 of 3` ` 2023 07 27 14 37 56 UTC retries py 80 DEBUG Running RenderedTaskInstanceFields do delete old records with retries Try 1 of 3` ` 2023 07 27 14 37 56 UTC taskinstance py 1545 INFO Exporting env vars AIRFLOW CTX DAG OWNER= AIRFLOW CTX DAG ID= test fi AIRFLOW CTX TASK ID= define project parameters AIRFLOW CTX EXECUTION DATE= 2023 06 27T14 37 54 602904+00 00 AIRFLOW CTX TRY NUMBER= 1 AIRFLOW CTX DAG RUN ID= scheduled 2023 06 27T14 37 54 602904+00 00 ` ` 2023 07 27 14 37 56 UTC init py 117 DEBUG Preparing lineage inlets and outlets` ` 2023 07 27 14 37 56 UTC init py 158 DEBUG inlets outlets ` ` 2023 07 27 14 37 57 UTC store py 32 INFO `read ` not implemented for `BaseSessionStore` Assuming empty store ` ` 2023 07 27 14 37 57 UTC session py 50 WARNING Unable to git describe opt test fi` ` 2023 07 27 14 37 57 UTC logging mixin py 150 INFO Model version 20230727 143757` ` 2023 07 27 14 37 57 UTC common py 123 DEBUG Loading config file opt test fi conf base logging yml ` ` 2023 07 27 14 37 57 UTC local task job runner py 225 INFO Task exited with return code Negsignal SIGABRT` Can anyone offer help to fix this It seems to be related to the line ``DEBUG Loading config file opt test fi conf base logging yml ``

Nok Lam Chan · Answer

Which version of Kedro are you using

Nok Lam Chan · Answer

The `Negsignal SIG` error looks very familiar You may try this to see if its resolve the problem <https github com kedro org kedro plugins issues 13 issuecomment 1088710755>

Nok Lam Chan · Answer

gt 2023 07 27 14 37 57 UTC store py 32 INFO `read ` not implemented for `BaseSessionStore` Assuming empty store I see this error message which suggests you are using relatively old kedro version if it s possible I ll advise upgrade the Kedro version

meharji arumilli · Answer

< Nok Lam Chan> im using kedro version 0 18 1

Nok Lam Chan · Answer

I haven t used Airflow for a while but please let me know if the above fix works for you

meharji arumilli · Answer

im trying as suggested in the thread by setting ` disable existing loggers True` in logging yml file I will update if it solves

meharji arumilli · Answer

< Nok Lam Chan> it did worked However there are new issues 1 For each node or task in the dag it picks new model version Normally it has one model version for the whole project ```from kedro config import TemplatedConfigLoader class MyTemplatedConfigLoader TemplatedConfigLoader def init self conf source env runtime params self params = os environ self update model version to parameters self update project parameters super init conf source=conf source env=env runtime params=runtime params globals dict=self params def update model version to parameters self if model version not in self params keys version = datetime now strftime Y m d H M S self params model version = version print f Model version version ``` 2 I do not see the errors logs as the pipeline failed at some node as it now only shows exited with error code 1

Nok Lam Chan · Answer

gt Normally it has one model version for the whole project Do you mean when you do `kedro run` locally

Nok Lam Chan · Answer

I am not sure how these model version are update One important thing to note is that when you run Kedro on orchestrator they are not longer running in the same process By default `kedro airflow` map each node to a separate task but you will be responsible to make the judgement how it should be organised

Nok Lam Chan · Answer

gt 2 I do not see the errors logs as the pipeline failed at some node as it now only shows exited with error code 1 If you go into the Airflow UI it doesn t show any log

meharji arumilli · Answer

no it is not showing any log of the kedro processing steps which it normally does It shows the same as before but instead of the line `INFO Task exited with return code Negsignal SIGABRT` it shows ` 2023 07 27T17 41 31 694+0000 common py 123 DEBUG Loading config file opt test fi conf base logging yml ` ` 2023 07 27T17 41 36 018+0000 taskinstance py 789 DEBUG Refreshing TaskInstance lt TaskInstance test fi train model manual 2023 07 27T17 38 28 224721+00 00 running gt from DB` ` 2023 07 27T17 41 36 039+0000 job py 210 DEBUG heartbeat ` ` 2023 07 27T17 41 37 392+0000 local task job runner py 225 INFO Task exited with return code 1`

meharji arumilli · Answer

Regarding the first issue when it is run as `kedro run` in the container it has only 1 model version for the whole project

Nok Lam Chan · Answer

Regarding to model version I think this is related <https kedro org slack com archives C03RKP2LW64 p1690447843976229 thread ts=1690447843 976229 amp cid=C03RKP2LW64|https kedro org slack com archives C03RKP2LW64 p1690447843976229 thread ts=1690447843 976229 amp cid=C03RKP2LW64> It is because when you run it distributedly they no longer share the same timestamp

Nok Lam Chan · Answer

Each node is run in a separate process thus they won t share the same timestamp

Nok Lam Chan · Answer

Regard to the airflow issue I am afraid I cannot be much help here

meharji arumilli · Answer

Some insights when i have logging yml file as below it returns the error `Task exited with return code Negsignal SIGABRT` And setting `disable existing loggers True` runs the project but is not capturing the logs in airflow UI and it returns only `Task exited with return code 1` or `Task exited with return code 0` ```version 1 disable existing loggers False formatters simple format asctime s name s levelname s message s json formatter format asctime s name s levelname s message s class pythonjsonlogger jsonlogger JsonFormatter handlers console class logging StreamHandler level INFO formatter simple stream info file handler class logging handlers RotatingFileHandler level INFO formatter simple filename logs info log maxBytes 10485760 10MB backupCount 20 encoding utf8 delay True error file handler class logging handlers RotatingFileHandler level ERROR formatter simple filename logs errors log maxBytes 10485760 10MB backupCount 20 encoding utf8 delay True loggers anyconfig level WARNING handlers console info file handler error file handler propagate no level INFO handlers console info file handler error file handler propagate no kedro pipeline level INFO handlers console info file handler error file handler propagate no root level INFO handlers console info file handler error file handler ``` When the logging yml is changed as below even with `disable existing loggers False` it does not give `Task exited with return code Negsignal SIGABRT` error unlike the above version of loggin yml This still gives the error codes`Task exited with return code 1` or `Task exited with return code 0` in airflow UI ```version 1 disable existing loggers False formatters simple format asctime s name s levelname s message s handlers console class logging StreamHandler level INFO formatter simple stream info file handler class logging handlers RotatingFileHandler level INFO formatter simple filename logs info log maxBytes 10485760 10MB backupCount 20 encoding utf8 delay True error file handler class logging handlers RotatingFileHandler level ERROR formatter simple filename logs errors log maxBytes 10485760 10MB backupCount 20 encoding utf8 delay True loggers kedro level INFO test fi level INFO root handlers info file handler error file handler ``` The actual logs are visible inside the kedro project logs info log and logs errors log files but not in the airflow UI Does this help any to troubleshoot

meharji arumilli · Answer

```root handlers console info file handler error file handler ``` and adding console to the root again gives `Task exited with return code Negsignal SIGABRT` it has something to do with logging yml file