Hey team, I have a question regarding

Dynamic Pipelines

. Namely, I would like to load a kedro dataset which then is used as the variants. For example I have a python list with stock tickers saved as a pickle and I would like to load that list and use it for the variants of the pipelines. I was thinking of intializing a kedro-session and simply load the list there and then pass it over to the variants. Wdyt? @marrrcin

Is your intent is to have a single "shared" dataset that will be re-used in all variants?

So far I have seen and used the variants as something that is hard-coded. For example

DYNAMIC_PIPELINES_MAPPING = {
    "reviews_predictor": ["base", "test1"],
    "price_predictor": ["base", "candidate1", "candidate2", "candidate3"],
}

But if we would now assume that we do not have 3

candidate

but 300, this approach of listing everything here would become somewhat infeasible. Particularly if

candidate

would needed to be created: e.g. a function that creates a list of

[candidate1, candidate11, candidate204]

The whole intent and Kedro assumption is to have this explicitly listed. You can generate

DYNAMIC_PIPELINES_MAPPING

via code, but you shouldn't load this from dataset/parameters as it is not supported by Kedro. You can hack your way around it (duh!) but then some other pieces of the framework might not work as expected / plugins might break. Sounds like something like https://github.com/takikadiri/kedro-boot/tree/main might help you with that.

Understood! Yes I was also thinking of manually inserting a dataframe, though that seemed a bit hacky