Hello everyone I'm using kedro inside a notebook ...
# questions
j
Hello everyone I'm using kedro inside a notebook on Databricks (unfortunately I need to use notebooks and I don't have authorization to use the new kedro-databricks plugin yet). It turns out that I'm having a very bad experience with the logs. The notebook cell is constantly blinking and freezing, so I can't view the logs. Below is my log configuration file. Could anyone suggest a better log configuration for Databricks notebooks?
Copy code
version: 1

disable_existing_loggers: False

formatters:
  simple:
    format: "%(asctime)s - %(name)s - %(levelname)s - %(message)s"

handlers:
  console:
    class: logging.StreamHandler
    level: INFO
    formatter: simple
    stream: <ext://sys.stdout>

  info_file_handler:
    class: logging.handlers.RotatingFileHandler
    level: INFO
    formatter: simple
    filename: info.log
    maxBytes: 10485760 # 10MB
    backupCount: 20
    encoding: utf8
    delay: True

  rich:
    class: kedro.logging.RichHandler
    rich_tracebacks: True

loggers:
  kedro:
    level: INFO

  grp_global:
    level: INFO

root:
  handlers: [console]
👀 1
l
Does changing the logging levels to WARNING improve the blinking/freezing issue? I'm thinking maybe the logging messages being sent too fast after each other might be contributing to this problem.
j
Yes, I think so. But I'd like to keep the info logs too 🙁
print() messages also works well
l
I'm thinking if there's any way to stagger the logging messages without having to write a custom handler. Does disabling rich logging improve anything? https://docs.kedro.org/en/stable/logging/index.html#how-to-use-logging-without-the-rich-library
j
Is it using rich logging? The handler is set to console only
l
Right, it shouldn't.
j
I have just found out that the KEDRO_LOGGING_CONFIG variable was not defined for my project. I will try to define it and evaluate again. Thanks, Laura.
👍🏼 1
l
Let me know how it goes!
j
That worked. Now this is actually using the console handler
But it's a bit weird. Do you know how to remove the [dark_orange] flag? Eg: INFO 2024-10-29 173512,027: Loading data from [dark_orange]params:HMC[/dark_orange] (MemoryDataset)... - kedro.io.data_catalog
l
What version of Kedro are you using? This issue was solved on 0.19.8.
j
I'm using 0.19.3
I will upgrade it. Thanks, Laura!
y
unfortunately I need to use notebooks
I feel you...
❤️ 1