what is the best way to access the catalog as defined in

catalog.yml

from a node? I'd like to add a dynamically defined dataset to it.

hey @Gauthier Pierard let me double check this and will get back to you shortly

We typically don’t encourage that, nodes shouldn’t be aware of IO

Ok. I need to do this because the dataset save path depends on previous nodes outputs (e.g. detected date of the last run)

You can search for examples of how to do dynamic pipelines, typically we encourage this to work via hooks that can access the catalog

Hey, @Gauthier Pierard, as pointed out above we do not encourage this to be done. But technically you can do it via the after_node_run hook. But for that, you’ll have to modify private datasets collection for the catalog as we do not p[provide a public API for that.

thanks for the pointers. basically why don't you encourage this? I'd think that allowing such a feedback from the nodes to the catalog must be a relatively common use case. but perhaps I'm wrong