I'm trying to get logging working and was hoping someone could point me in the right direction. when you load kedro and run it out of the box, kedro automatically writes nodes and pipeline details to the console, I'd like to keep that as it is. What I'm not figuring out is how to use logger inside a module to write log entries to a file and not the console. I have a large json object I want to print to file so I can look at it. I tried setting up my logging.yml file but I'm not understanding something.

logging.yml
handlers:
  ...other built-in kedro handlers...
  debug_file_handler:
    class: logging.handlers.RotatingFileHandler
    level: DEBUG
    formatter: simple
    filename: logs/debug.log
    maxBytes: 10485760 # 10MB
    backupCount: 20
    encoding: utf8
    delay: True

loggers:
  kedro:
    level: INFO

  kedro_workbench:
    level: INFO

  DataSets:
    level: DEBUG
    handlers: [debug_file_handler]

root:
  handlers: [rich, info_file_handler, error_file_handler]

in my module I used:
import logging
logger = logging.getLogger('DataSets')
logger.debug(output)

but when I run the pipeline, the contents of output are still written to the console. What am I missing here? thanks kindly!

you need to add

debug_file_handler

in

root

instead.

thanks @Nok Lam Chan are you saying I need to remove the

handlers: [debug_file_handler]

from my 'DataSets' entry?

yes, remove handlers from

DataSets:

and append

debug_file_handler

to

root

crap @Nok Lam Chan, that didn't work. I updated my code as you recommended but the console still receives the logger.debug() contents. Can you recommend anything else?

Does it at least show up in the file that you wish to write?

yup!

is this DataSet in

kedro_workbench

?

yup everything is in my project

hm.. I am unsure. Can you try comment out the

kedro_workbench

?

arg, I removed the reference entirely but the console still gets the debug details....thanks for trying!

I just tested it and it works as expected.

# logging.yml

...
  debug_file_handler:
    class: logging.handlers.RotatingFileHandler
    level: DEBUG
    formatter: simple
    filename: debug.log
    maxBytes: 10485760 # 10MB
    backupCount: 20
    encoding: utf8
    delay: True
loggers:
  kedro:
    level: INFO

  debug_logging:
    level: INFO
root:
  handlers: [rich, info_file_handler, debug_file_handler]

thank you kindly, I'm not sure I understand how to implement it? if I remove "DataSets" from loggers, how do I instantiate the logger in my module? just use the default?

logger = logging.getLogger(__name__)

In your example you used,

my_logger = logging.getLogger("DEBUG_ONLY")

but there is no 'DEBUG_ONLY' entry in the loggers section.

``` debug_file_handler:

class: logging.handlers.RotatingFileHandler

level: DEBUG

formatter: simple

filename: logs/debug.log

maxBytes: 10485760 # 10MB

backupCount: 20

encoding: utf8

delay: True

loggers:

kedro:

level: INFO

kedro_workbench:

level: INFO

root:

handlers: [rich, info_file_handler, error_file_handler]```

Does this config works?

@Nok Lam Chan thank you for following up. I tried the config above, but it doesn't achieve what I'm trying to do. No matter what I do, the DEBUG gets written to the console... 😞