Hi everyone, I hope you’ve all had a nice week-end...
# questions
Hi everyone, I hope you’ve all had a nice week-end 🙂 I’m sure this is doable, but I haven’t found much “leads” in the documentation. How could I have a “*per-run log file*” that would be save as an artifact of the pipeline in the data dir ? Thx in advance Regards, Marc
🤔 1
That’s a nice question!
I would like to see the answer 🤔
Oh… I must confess that I’m flattered that you find the question interesting 🙂 Let’s see if someone has a straightforward suggestion / solution !
I think this is easiest with a
hook, you have access to the
Thx @datajoely for you answer and suggestion. But I remain a little bit confused. If I remember correctly,
is a datetime like
, correct ? If so… What am I supposed to do with it to access to the run’s log ? Thx in advance for you time, advice and patience 🙏🏼 🙂
oh we’re talking about the logging config 🤔
@datajoely can it be done via logging config ? 🙂 I’ve looked into the doc but haven’t found anything. And I confess that I didn’t have time to dive into rich’s docs… I was hoping that someone more advance would have a “quick & easy” solution 😜
so I was wondering about this then got sucked into something else
I wonder if you can set up the logging formatter to do this
but from a lifecycle point of view I think we initialise the logging config before the session is initialised
I really don’t know here
it’s a thin team today will see if others have any ideas tomorrow
🙏🏼 🙂
Hi @datajoely 🙂 I was going through my old conversation / threads and saw that this one was still “pending”. Any updates / insights ? Thx in advance, M
Good question - let me raise this
this technically is similar to log rotation right? https://docs.python.org/3/library/logging.handlers.html#rotatingfilehandler but not quite, because we'd want to have 1 file per run, and log rollover when the size of the log goes over a certain threshold...
one can force manual rollover https://stackoverflow.com/a/44636428/554319 but I don't think this is possible just tweaking the logging configuration 🤔
@Juan Luis are
resolvers available in
? Could you do something like
Copy code
    class: logging.handlers.RotatingFileHandler
    level: INFO
    formatter: simple
    filename: info_${oc.env:KEDRO_SESSION_ID}.log
💡 1
this is a clever trick - but then if the filename uses the session id, there's no need to do manual rotation right? and also,
wouldn't normally be set, or am I missing something?
yeah you’d have to set it in an
and I’m not sure if that would even work tbh
continuing with the resolver idea, could there be a
resolver that got it from the session?
this would be very very useful
especially if we also make the session_id configurable
man I’m so excited by what OmegaConf is able to do
🍿 meow popcorn Thanks guys for considering this question / request 🙂
how would such a resolver be defined though? resolvers are added to omegaconf on
, it's unclear to me how to fetch the global Kedro session or context 🤔
omegaconf could fetch a ENV var that doesn’t exist yet
Not sure if I missed anything - the standard library
should support this and you just need to defined the filepath to save in
instead of the root directory.
how do you define the filepath dynamically for each run?
So you also need to match
? Otherwise there are easy way to rotate the files (pretty sure it was asked a few months ago). Using
may not be ideal because it is decoupled from config loader in 0.19, it will be just a simple
@Marc Gris could you please share why you want to do this? And what log do you want to keep, the entire log from the logger or certain logs? What is the use case you are solving?
👍 1
Hi @Ivan Danov Thanks for your message. Actually, I had posted the question 2 month ago. Therefore, I hope you won’t mind if I can’t remember the exact use case… 😅 However, I can remember the “spirit”: Striving to apply MLOps best practices, it seemed to me relevant to keep track of as many thing as possible regarding my experiments etc… The log of a run being information rich, it naturally seemed to me worth persisting & versioning “individually”… Thanks again & Have a nice week-end, M.
if we added the session id to the actually log lines and switched to JSON logging, perhaps thats a good way of doing this? It would work nicely in ELK for example
yeah I think it's pretty reasonable to desire 1 log file per run 🤔 and so far seems like the best solution in Kedro is
kedro run > log_$(date).txt
https://stackoverflow.com/questions/44635896/how-to-create-a-new-log-file-every-time-the-application-runs One can achieve that by doing what is suggested in this stackoverflow post ☝️. Best option is creating a custom logging handler, extending the built-in
, which will always create a new file. The path and the name of that file can be configured through
. Making that file have the same name as the session id is not that easy indeed, but doable with a few hacks. I think the work on this will make it much easier once it's done: https://github.com/kedro-org/kedro/issues/1731
🙏🏼 2