Hey team, I have a question regarding `Dynamic Pip...
# questions
Hey team, I have a question regarding
Dynamic Pipelines
. Namely, I would like to load a kedro dataset which then is used as the variants. For example I have a python list with stock tickers saved as a pickle and I would like to load that list and use it for the variants of the pipelines. I was thinking of intializing a kedro-session and simply load the list there and then pass it over to the variants. Wdyt? @marrrcin
Is your intent is to have a single "shared" dataset that will be re-used in all variants?
So far I have seen and used the variants as something that is hard-coded. For example
Copy code
    "reviews_predictor": ["base", "test1"],
    "price_predictor": ["base", "candidate1", "candidate2", "candidate3"],
But if we would now assume that we do not have 3
but 300, this approach of listing everything here would become somewhat infeasible. Particularly if
would needed to be created: e.g. a function that creates a list of
[candidate1, candidate11, candidate204]
Hence I am looking for a way to create/ load the
list of variants and then pass it if that makes sense.
The whole intent and Kedro assumption is to have this explicitly listed. You can generate
via code, but you shouldn't load this from dataset/parameters as it is not supported by Kedro. You can hack your way around it (duh!) but then some other pieces of the framework might not work as expected / plugins might break. Sounds like something like https://github.com/takikadiri/kedro-boot/tree/main might help you with that.
Understood! Yes I was also thinking of manually inserting a dataframe, though that seemed a bit hacky