Is there a way in kedro in which I can overwrite the info lo Kedro #questions

Is there a way in kedro in which I can overwrite t...

tom kurian

08/30/2023, 2:56 PM

Is there a way in kedro in which I can overwrite the info.log file every time it executes. Currently it is appending new run logs.

Nok Lam Chan

08/30/2023, 2:57 PM

you will need to change the type of logger it is using.

tom kurian

08/30/2023, 2:57 PM

can you give me an example if possible;

Nok Lam Chan

08/30/2023, 2:58 PM

Copy code

info_file_handler:
    class: logging.handlers.RotatingFileHandler
    level: INFO
    formatter: simple
    filename: info.log
    maxBytes: 10485760 # 10MB
    backupCount: 20
    encoding: utf8
    delay: True

This is the default

Nok Lam Chan

08/30/2023, 2:58 PM

Do you need to overwrite the file completely? Or you want to keep the record having

info1.log

info2.log

etc?

datajoely

08/30/2023, 2:58 PM

or you could use an

after_pipeline_run

hook to just truncate the file

datajoely

08/30/2023, 2:58 PM

ugly but would work

tom kurian

08/30/2023, 2:58 PM

I need to overwrite everytime

Nok Lam Chan

08/30/2023, 3:01 PM

https://www.datasnips.com/188/overwriting-a-python-logging-file-instead-of-appending/

Nok Lam Chan

08/30/2023, 3:01 PM

I guess it is as simple as changing

mode: w

, I haven’t tested though

tom kurian

08/30/2023, 3:01 PM

let me try 🙂

tom kurian

08/30/2023, 3:06 PM

Copy code

info_file_handler:
    class: logging.handlers.RotatingFileHandler
    level: INFO
    formatter: simple
    filename: logs/info.log
    maxBytes: 10485760 # 10MB
    backupCount: 20
    encoding: utf8
    delay: True
    mode: w

i tried this, but still it is appending the logs.

Nok Lam Chan

08/30/2023, 3:09 PM

No Kedro magic is involved, all pure python

logging

. I just tried and it works. Thanks for asking the question I didn’t know this before too 🙂 The simple solution is don’t use

RotatingFileHandler

, use the basic one.

Copy code

info_file_handler:
    class: logging.FileHandler
    level: INFO
    formatter: simple
    filename: info.log
    delay: True
    encoding: utf8
    mode: w

🚀 2

💪 1

2 Views

Open in Slack

Previous Next