https://kedro.org/ logo
#questions
Title
# questions
g

Gary McCormack

04/17/2023, 3:56 PM
Hi All, I want my logs directory to have sub directories inside that segment them by date. So logs that are generated by daily pipeline runs will be in their own individual folder. I assume from what I read that this is not available using Kedro's
conf/base/logging.yaml
by default (please correct me if I'm wrong), so I would have thought that I would need to create my own custom logger. If I remove the logging conf file however then this raises an exception (
ValueError: dictionary doesn't specify a version
). Any suggestions or advice on how to get setup a custom logger to run on Kedro would be great!
a

Antony Milne

04/18/2023, 10:05 AM
Hi Gary, you’re right that this is not available using Kedro’s logging configuration by default. What you’ll need to do is write a custom
conf/base/logging.yml
to achieve this. The format of this file is actually nothing to do with Kedro - it’s just Python’s logging dictionary schema. The reason for the
ValueError
is that you’re missing the
version
key, which is compulsory.
So basically you need to work out what are the right logging handlers that might be able to achieve what you need. If there’s nothing already available to achieve this then I’ll warn you that it can be quite fiddly to write your own. In which case you might be able to achieve this more easily by bypassing Python’s logging system and instead using Kedro hooks.
At a glance, I think
TimedRotatingFileHandler
might do what you need though 🙂 https://docs.python.org/3/library/logging.handlers.html#timedrotatingfilehandler
g

Gary McCormack

04/18/2023, 10:10 AM
Thanks for the reply! Ok, this is good, at least I know I'm on the right track. I can try see if there are any handlers that works for my use cases, but yes I agree these things do become quite tricky and messy. I haven't considered hooks for this job, but if all else fails with the previous steps I'll definitely try these. Thanks again! 🙂
👍 1
🥳 1
3 Views