https://kedro.org/ logo
#questions
Title
# questions
i

Idris Benkhelil

06/21/2023, 6:02 AM
Hello, Thank you for this great library. I am a DS working in France. I have a question, I want to make my pipeline dynamic, ie: Pipeline:
Copy code
[etape 1] > [etape 2] > [if score_etape2 < X ] > [etape4]
				      > [if score_etape2 >= X ] > [etape5]
Do you have any indication of how I can do this? Or an example of code already implemented? Thanks in advance. Idris
m

Marc Gris

06/21/2023, 9:20 AM
Hi Idris Salutation d’un compatriote, DS à l’île Maurice 🙂 I’m definitely no kedro expert at all… not remotely… so.. I hope you won’t mind my “jumping in”… but what comes to mind is: Couldn’t you simply have a single _etape_3_ with 2 branches testing for _score_etape2_ ? Could you may be give a some more details about the purpose / functions of your different etapes ? Wishing you a nice day Marc
i

Idris Benkhelil

06/21/2023, 10:29 AM
Hello Marc, Salutation également cher confrere 🙂 merci pour ta réponse. Je suis nouveau sur Kedro. My goal is to create a mother pipeline that will conditionally launch daughter pipelines
Copy code
Daughter pipeline1: Model Evolution 
	Cross validation of a model, returns data to tell if the model is valid or not. 

Daughter pipeline2: Train
	Train and store the model.

Daughter pipeline3: Tuning parameters
	Returns new model parameters.
Copy code
Pipeline mere: 
Model Evaluation >
	If Model Evaluation > NOK Then Tuning + Train
	If Model Evaluation > OK Then Train
m

Marc Gris

06/21/2023, 2:01 PM
oupssy… fired the message while I was still typing it 😅
😅 1
Just to check that I’m getting things correctly: You have some pre-trained model that you want to update / train on new data, with an optional tuning step if the model has “drifted” and its performance has dropped below some benchmark. Correct ?
i

Idris Benkhelil

06/22/2023, 5:02 AM
Hi Marc, it's an application that runs every two weeks with a new batch of data. Every two weeks, I re-train the model: • with the model parameters in "production" • with a cross validation on 5 folds • with the new data. And on these folds I take the average of the metric. If this indicator (x of the metric) exceeds a threshold I've set based on my experience of the subject. You can see the beginnings of a model drift. So I'd like to add a little hyperparemeters tuning step in the case of a drift, to train the future model with these new parameters. Make drift management automatic. And be able to implement this on Kedro 🙂 with different pipelines.
👍 1
j

Juan Luis

06/22/2023, 11:01 AM
hi @Idris Benkhelil, @Nok Lam Chan gave an approximate answer to this here https://github.com/kedro-org/kedro/issues/2627#issuecomment-1602435185 long story short, there's not a native way to achieve this behavior in Kedro at the moment, unfortunately. please upvote the issue so it has more flexibility
thankyou 2
👍 2
i

Idris Benkhelil

06/22/2023, 11:53 AM
I will try the first solution, ty guys!
m

Marc Gris

06/22/2023, 3:04 PM
Not knowing that the exact behavior you asked for is not (yet) available, I was going to suggest that “simple-is-often-good-enough” 🙂 you could indeed have the condition in a step-2 “get hyper-params”, that would, on the basis of score threshold, either return the “old” hyper-params or some freshly tuned hyper-params… have a nice end-of-day/week
13 Views