Is there a way in kedro to define or register para...
# questions
m
Is there a way in kedro to define or register parameters, like a logger, that all parameters and nodes can get access to, s.a. for the datasets from the catalog file?
m
Could you rephrase/elaborate more on this question? What is the intent here?
m
Certainly, in my case, I initialise a weights and biases experiment tracker:
Copy code
import wandb
run = wandb.init(project="visualize-sklearn")
I then want to pass around this initialised “run” parameter to the nodes without having to initialise it again. Would the simplest method be to define it in as a argument in
conf/base
and add a custom resolver for OmegaConf, or is there a more convenient method?
m
You have at least 3 options: 1. Initialize the
run
in the first node, then just pass it as an explicit input to every other node that would need to do sth with W&B 2. Initialize the
run
in the hook (e.g.
before_pipeline_run
) and pass it implicitly to every node with
before_node_run
implemented in the same hook class https://docs.kedro.org/en/stable/hooks/introduction.html). 3. Initialize the
run
in the hook (again,
before_pipeline_run
) but do not pass it to nodes and just use
wandb.run
as it’s a global singleton https://docs.wandb.ai/guides/runs#create-a-run There are probably a few other options too
As for the hooks, it’s worth mentioning that e.g. in the
after_context_created
you can capture references to catalog/parameters/config loader, which would be a good place to get some configurations that can be used in the
before_pipeline_run
to initialize wandb
run