Hi everyone I m coming back with a question that I think has

Question

Hi everyone I m coming back with a question that I think has been floating around here a few times Sorry for that DISCLAIMER warning This is about facilitating dynamic pipelines bad smiling imp I know When creating a dynamic pipeline it is useful even vital to access the currently running context session ex to do things like accessing the parameters after they ve been overwritten via `kedro run params` etc Somewhere in a github issues I had found something that looked more or less like this ```from kedro framework session import get current session current session = get current session current context = session load context current params = context catalog load parameters ``` However this is the old api and does not work anymore confused How can we access the current session at runtime in `pipeline py` For now I m condemned to either hard code things manually or to do this ugly and not really useful hack ```from kedro framework session import KedroSession project path = paths PROJECT ROOT DIR as posix static session = KedroSession create project path=project path ``` If someone has a solution please let me know Many thanks M P S I remember a mention being made about using the `after context created` hook but from the depth of my ignorance I have no clue how this could be of help in what I want to achieve slightly frowning face

Lodewic van Twillert · Answer

Hello again I too sometimes want dynamic pipelines though I do try to avoid them as much as I can in the final product they are very nice for experimenting if you have a lot of parameter options to test I will share my Hooks to solve this Wonder what people think 1 Create a ~`after context created`~ `after catalog created` hook 2 Replace the `register pipelines ` function with a custom `register dynamic pipelines catalog DataCatalog ` function a If you can pass the `catalog` to create pipelines you can access datasets and parameters to dynamically build your pipeline 3 Use `create pipeline catalog DataCatalog ` functions to create your pipelines Ill share my approach to create the pipelines Of course you would need to extend this example if you also want to dynamically create DataCatalog entries to save the outputs of your dynamic pipeline But im leaving that out here to save space Here are my steps `ProjectHooks` in `hooks py` Make sure to add it to your `settings py` on the commented line saying `HOOKS= ProjectHooks ` ```from kedro framework context import KedroContext from kedro framework hooks import hook impl from kedro framework project import pipelines from import DataCatalog from yourpackage import pipeline registry class ProjectHooks hook impl def after context created self context KedroContext gt None context catalog hook impl def after catalog created self catalog DataCatalog conf catalog gt None Hook to fill in and extend templated pipelines pipeline registry register pipelines = pipeline registry register dynamic pipelines catalog pipelines configure yourpackage pipeline registry ``` Notice that we overwrite the `pipeline registry register pipelines` method with `pipeline registry register dynamic pipelines ` in the hook In your `pipeline registry py` use the following ```from import DataCatalog from kedro pipeline import Pipeline def register pipelines Method that will be assigned to the callable returned by register dynamic pipelines by a Hook raise NotImplementedError register pipelines is expected to be overwritten by ProjectHooks Make sure the hooks is found in hooks py and enabled in settings py def register dynamic pipelines catalog DataCatalog gt dict str Pipeline Register the project s pipelines depending on the catalog Create pipelines dynamically based on parameters and datasets defined in the catalog The function must return a callable without any arguments that will replace the `register pipelines ` method in this same module using an `after catalog created hook` Args catalog The DataCatalog loaded from the KedroContext Returns A callable that returns a mapping from pipeline names to ``Pipeline`` objects create pipelines with access to catalog my pipeline = create pipeline catalog=catalog def register pipelines Register the project s pipelines Returns A mapping from pipeline names to ``Pipeline`` objects pipelines = default my pipeline return pipelines return register pipelines``` This uses a `create pipeline catalog=catalog ` method for example something like this ```from import DataCatalog def create pipeline catalog DataCatalog parameters = catalog datasets parameters load models = parameters models a dictionary with the keys being the model labels dynamic nodes = for model parameters in models items model nodes = node func=fit model inputs=dict X= X train y= y train model type=f params model class fit kwargs=f params model fit kwargs outputs=f model fitted node func=score model inputs=dict X= X test y= y test model=f model fitted outputs=f model score dynamic nodes extend dynamic nodes output pipeline = pipeline dynamic nodes return output pipeline ```

Lodewic van Twillert · Answer

Hope you don t consider this spam because it s quite verbose But by using namespaced pipelines and data factories you could even make this a lot cleaner Instead of copy pasting more code maybe it s handier to attach the py files thinking face Same Hook as above But instead of creating pipelines by changing the parameter and dataset names directly we create `namespace`d pipelines Along with dataset factories <https docs kedro org en stable data data catalog html example 3 generalise datasets using namespaces into one dataset factory> have a look at the catalog yml This way not only is the pipeline dynamic but also your catalog

Marc Gris · Answer

< Lodewic van Twillert> Spam Are you kidding wink Thank you so much for all this I ll explore play around and let you know how it goes Thank you so much for sharing pray skin tone 3 sunny slightly smiling face Regards M

Lodewic van Twillert · Answer

Haha ok great then enjoy Hope to hear your thoughts

Lodewic van Twillert · Answer

For completeness I made some quick typos in the `catalog yml` should have been this ```models linear regression class sklearn linear model LinearRegression fit kwargs fit intercept true decision tree class sklearn tree DecisionTreeRegressor fit kwargs criterion squared error quantile regression class sklearn linear model QuantileRegressor fit kwargs fit intercept true quantile 0 5```

Nok Lam Chan · Answer

Thanks a lot < Lodewic van Twillert> very creative solution We have an open issue to track Dynamic Pipeline While we may not have an immediate solution it is still beneficial to have workaround

Nok Lam Chan · Answer

I ll cross post your solution there <https github com kedro org kedro issues 2627>

Nok Lam Chan · Answer

And this is gem not spams

Lodewic van Twillert · Answer

Cool thanks I didnt realize there was already a Dynamic Pipeline issue ill keep an eye on that too muscle

Nok Lam Chan · Answer

The solution is quite comprehensive it would be great if you can share a demo repository but this is awesome work slightly smiling face

Lodewic van Twillert · Answer

Yeah sure happy to I can make a working version with the iris starter and link a repository

Nok Lam Chan · Answer

I ve linked your response on the issue now Let me know when you have a working repository and I can update it accordingly

Marc Gris · Answer

Hi < Nok Lam Chan> Good to see you here slightly smiling face I m both really appreciative amp grateful by what < Lodewic van Twillert> has shared But just to try to really understand leaving aside the general question of dynamic pipelines do you confirm that there is no simple amp straight forward way to achieve what the old api allowed to do with `from kedro framework session import get current session` Thx M

Nok Lam Chan · Answer

< Marc Gris> The direct replacement of it is hook either `after context created` or `after catalog created` What do you need to do Do you just need `params` to create your pipeline

Marc Gris · Answer

< Nok Lam Chan> here s an example of the dirty dynamic things that I try to do ```def create pipeline kwargs gt Pipeline all models = model a model b model c all pipelines = for model in all models model pipeline = pipeline MODEL INPUT PREP node func=functions prepare model input inputs= featurized users featurized items featurized interactions f params models model name f params models model params input prep outputs=f model input name=f model input preparation node tags= model input model other nodes all pipelines append model pipeline return sum all pipelines ``` What would be an elegant proper way to do this avoiding the hardcoding and ideally accessing ` params models ` to iterate over I do hear and understand Kedro s stance for static data centric pipelines But one must reckon that if one has dozens of models creating their pipelines manually would be both tedious and so redundant boilerplaty Thx slightly smiling face P S May be what was shared by Lodevic completely fits my bill and solves my problem But I m stuck at work on something and did have time to dive into his suggestion

Nok Lam Chan · Answer

dynamic pipeline is not so dirty IMO I have a lot of empathy on that It should be avoid when it s not necessary and there are cases where it is necessary slightly smiling face

Nok Lam Chan · Answer

I am trying to work out an example

Lodewic van Twillert · Answer

If you are interested in trying my approach or just testing it I have finished an example in this repository <https github com Lodewic kedro dynamic pipeline hook example>

Marc Gris · Answer

Wow Awesome You re blazingly fast Will dive into it with GREAT interest ASAP slightly smiling face Thx again

Nok Lam Chan · Answer

that s an awesome README

Lodewic van Twillert · Answer

Thank you It has the Kedro badge I saw being mentioned haha

Nok Lam Chan · Answer

Ya that is neat smile

Iñigo Hidalgo · Answer

< Lodewic van Twillert> this is amazing We re in the process of migrating away from kedro 0 17 1 where we make use of the old session API Marc was trying to use and this repo shows such a clever use of hooks to replace the use of the session Thanks for sharing

Iñigo Hidalgo · Answer

What is the reason for this hook ``` def after context created self context KedroContext gt None context catalog```

Iñigo Hidalgo · Answer

it doesn t save it as a class attribute or anything does calling the catalog force it to instantiate before it normally would or smth

Lodewic van Twillert · Answer

Hi glad you like it hope it helps you That hook I don t remember exactly when we started to include that we had an issue to which that was a solution but maybe it was unrelated to the dynamic pipelines Might have been something do to with the lazy loading in the catalog as you also suspected Could try commenting it out and see if it all works the same am also curious tbh thinking face

Iñigo Hidalgo · Answer

will give it a try once I get one of our projects set up and I ll update you slightly smiling face

Lodewic van Twillert · Answer

< Iñigo Hidalgo> If you remove that hook then `kedro run` breaks but `kedro viz` will be fine but here is why The reason is that when you call `context catalog` then the `after catalog created ` hooks are triggered Normally the catalog seems to be created After `register pipelines ` is required So when our `register pipelines ` function is not replaced then it will raise an error So calling `context catalog` after the context is created we force the `after catalog created ` to be triggered earlier than usual gt therefore overwriting the `register pipelines ` function before it is required Here is the ` get catalog ` method from kedro where you can see that the `after catalog created ` hook is triggered +1 I think `kedro viz` also creates the catalog earlier than `kedro run` would so for viz it is not required to

Lodewic van Twillert · Answer

In short you were right the first time