Hi all. I'm trying to add some verbosity to a specific node in my pipeline, to help monitor any potential errors that could arise. I manage several packages, but for this, the two that I care about are Package1 (a common library) and Package2 ( where Kedro projects live). Package1 contains a predefined logger that is set exactly how we want our logs to be output. Normally, I would just import that logger directly into the script I'm running and boom, verbose logging. However, when I import it into the node in Project2 that I want to keep track of, nothing changes in the stdout. Is there a specific thing I must do (maybe in the conf) to allow this logger to work? I suppose it's worth noting that the node itself may also be importing functions/classes from other packages that we maintain, Package3, Package4, etc etc. I don't know if that makes a difference in this case

In your conf there is a logging.yml. In general it's not a good idea to import a logger from a separate module directly. How is the original logger configured?

So I'd happily just use the logging file. Can't seem to get the setup correct tho. Maybe you can help me get it to be as verbose as possible.

version: 1

disable_existing_loggers: False

formatters:
  simple:
    format: "%(asctime)s - %(name)s - %(levelname)s - %(message)s"

handlers:
  console:
    class: logging.StreamHandler
    level: INFO
    formatter: simple
    stream: <ext://sys.stdout>

  info_file_handler:
    class: logging.handlers.RotatingFileHandler
    level: INFO
    formatter: simple
    filename: info.log
    maxBytes: 10485760 # 10MB
    backupCount: 20
    encoding: utf8
    delay: True

  rich:
    class: kedro.logging.RichHandler
    rich_tracebacks: True
    tracebacks_show_locals: True

loggers:
  kedro:
    level: INFO

  my_pipeline:
    level: INFO

  package1:
    level: INFO

  package3:
    level: INFO

  package4:
    level: INFO

root:
  handlers: [rich, info_file_handler]

The goal is that all

<http://logging.info|logging.info>()

statements in packages 1->4 are all printed

So with this settings log are not showing up?

So the logs I see are like the following

[09/19/23 11:46:11] INFO     Running node: node_function: node_function(None) -> [NODE.start_date,NODE.end_date]                                                                                       node.py:331
                    INFO     Saving data to 'NODE.start_date' (MemoryDataset)...                                                                                                                                                   data_catalog.py:531
                    INFO     Saving data to 'NODE.end_date' (MemoryDataset)...                                                                                                                                                     data_catalog.py:531
                    INFO     Completed 1 out of 31 tasks                                                                                                                                                                                     sequential_runner.py:85
                    INFO     Loading data from 'params:PARAMS.trader_initials' (MemoryDataset)...

But I would want verbose logging for what I see inside

node_function

, if you get me

What do you mean by verbose? How are you logging information within your node?

I mean the node does several calculations. Each calculation comes from one of our preexisting packages (package1->package4 from the example that I posted). Each calculation also has it's own logging messages,

<http://logger.info|logger.info>("I'm calculating this now"),logger.warning("High standard deviation found .."), etc etc

. I need these internal logging statements printed out in the stdout for the entire Kedro pipeline basically

how did you create

logger

?