Hi All I want my logs directory to have sub directories insi

Question

Hi All I want my logs directory to have sub directories inside that segment them by date So logs that are generated by daily pipeline runs will be in their own individual folder I assume from what I read that this is not available using Kedro s `conf base logging yaml` by default please correct me if I m wrong so I would have thought that I would need to create my own custom logger If I remove the logging conf file however then this raises an exception `ValueError dictionary doesn t specify a version` Any suggestions or advice on how to get setup a custom logger to run on Kedro would be great

Antony Milne · Answer

Hi Gary you re right that this is not available using Kedro s logging configuration by default What you ll need to do is write a custom `conf base logging yml` to achieve this The format of this file is actually nothing to do with Kedro it s just<https docs python org 3 library logging config html dictionary schema details| Python s logging dictionary schema> The reason for the `ValueError` is that you re missing the `version` key which is compulsory

Antony Milne · Answer

So basically you need to work out what are the right <https docs python org 3 library logging handlers html module logging handlers|logging handlers> that might be able to achieve what you need If there s nothing already available to achieve this then I ll warn you that it can be quite fiddly to write your own In which case you might be able to achieve this more easily by bypassing Python s logging system and instead using Kedro hooks

Antony Milne · Answer

At a glance I think `TimedRotatingFileHandler` might do what you need though slightly smiling face <https docs python org 3 library logging handlers html timedrotatingfilehandler>

Gary McCormack · Answer

Thanks for the reply Ok this is good at least I know I m on the right track I can try see if there are any handlers that works for my use cases but yes I agree these things do become quite tricky and messy I haven t considered hooks for this job but if all else fails with the previous steps I ll definitely try these Thanks again slightly smiling face