Hi Team I m working on a use case wherein I need to make cer

Question

Hi Team I m working on a use case wherein I need to make certain values from a `cache` made available inside the ` load` method of multiple Kedro Datasets How to go about it Can we use hooks or anything simpler

marrrcin · Answer

What kind of cache

datajoely · Answer

And is CachedDataSet not helpful here

Afaque Ahmad · Answer

I ve to use SparkDataSet Its a `redis` cache having metadata updated dynamically based on which data is loaded

datajoely · Answer

so hooks + custom dataset feel like the right approach

datajoely · Answer

it sort of goes against the reproducibility centred patterns we want to encourage but I m increasingly of the opinion that we need to think about the right way of doing dynamic conditional pipelines

Afaque Ahmad · Answer

Gotcha okay what are the repercussions around having dynamic conditional pipelines the metadata will have history so the same behaviour could be reproduced

Afaque Ahmad · Answer

If we were to implement using hooks what would be the best place to inject the values

marrrcin · Answer

`before dataset loaded` hook most probably

datajoely · Answer

or `before pipeline run` since you have access to everything

Afaque Ahmad · Answer

In case I use `before pipeline run` the datasets would be loaded by that time I need to inject the values before loading the dataset seems I ve to use `before dataset loaded` ~`before dataset loaded` is called each time before loading the dataset or only once before all datasets are loaded ~

Afaque Ahmad · Answer

Is there a way I could fetch the values from the cache only once in Kedro s lifecycle and avoid doing it with every dataset load

Nok Lam Chan · Answer

Before pipeline happens before this I believe

Nok Lam Chan · Answer

Pulling out some slightly outdated docs wait a min

Nok Lam Chan · Answer

There will be a more updated version and explanation of the lifecycle coming I will work on it sweat smile sweat smile sweat smile

Afaque Ahmad · Answer

Great I ll use `before pipeline run` Where can I find all the inputs to the hook `before pipeline run` In the example I see only `run params`

datajoely · Answer

datajoely · Answer

you want to look here not at that example pluggy will smartly provide only relevant inputs

datajoely · Answer

the `pipeline inputs` attr will be helpful here

Afaque Ahmad · Answer

Awesome thanks so much everyone