https://kedro.org/ logo
#plugins-integrations
Title
# plugins-integrations
a

Adrien

02/08/2024, 5:46 PM
Hello guys, is it possible to use a custom runner with kedro Vertex ai ? I know the plugin has his own runner but maybe their is a way to use both ?
d

Deepyaman Datta

02/08/2024, 5:50 PM
The
VertexAIPipelinesRunner
looks quite simple, so you could extend it or include it's functionality in your custom runner or use that? There would be no native way to combine runners.
a

Adrien

02/08/2024, 5:53 PM
Thanks for your answer ! I wanted to do that but when vertexai compile the pipeline, on each node he is calling the default Vertexai runner. So if I do that, it will not work on vertexai cmoud
d

Deepyaman Datta

02/08/2024, 5:54 PM
I have not used the plugin, but I would say you should be able to patch the command to use your runner.
a

Adrien

02/08/2024, 5:55 PM
By forking vertexai plugin ? It was my last solution ...
d

Deepyaman Datta

02/08/2024, 5:56 PM
You can override Click commands in your project. It's not 100% straightforward, but doable
a

Adrien

02/08/2024, 5:58 PM
Mmmh I'll think about it thanks :) not really used to click command
d

Deepyaman Datta

02/08/2024, 5:58 PM
If you want to override a Click group, you need to import that group, and then you can write new commands to overwrite it. Most likely, you will import stuff from the VertexAI plugin, so you're not rewriting (much) code.
@marrrcin please comment if this is unnecessarily complicated, and I'm missing a very straightforward way to do it. ๐Ÿ˜„
a

Adrien

02/08/2024, 6:16 PM
Yes, if there is an easier way to do it would be perfect ๐Ÿ‘Œ Seems a little bit complicated
I juste checked and i can't do what i want with cli. The runner is hardcoded in vertexai
m

marrrcin

02/09/2024, 12:13 AM
We're keen to accept a PR for that feature ๐Ÿ˜‰ What use case you want to cover by overriding the runner ?
a

Adrien

02/09/2024, 11:34 AM
I want to implement pytorch cluster computing with vertexai. I need to make it work fast and then i'll make a PR similar to kedro-azureml
m

marrrcin

02/09/2024, 12:33 PM
If you are really in a hurry thenโ€ฆ ๐Ÿ™ˆ
Copy code
from unittest.mock import patch
with patch('kedro_vertexai.vertex_ai.runner.VertexAIPipelinesRunner', new=YourRunner):
    # do magic here :D
๐Ÿ”ฅ 1
๐Ÿ‘ 1
a

Adrien

02/09/2024, 1:07 PM
Wow your line seems magic :0 I run this in the cli.py ?
Yes i do have a work but i wanted to keep having updates
Fork*