Hi @Nok Lam Chan a coworker just asked me about https://github.com/noklam/kedro-softfail-runner Do you ever use it in your daily work? or was it more of a proof-of-concept?

👋 hey, I created this for a client originally, so it has been used at least in a project. There were few people interested in this or extending runner, so I make this pip installable for people to try out

Saw that part in the readme. It isn't super important for my coworker either so I'll pass that on. Thank you 🙂

https://kedro-org.slack.com/archives/C03RKP2LW64/p1708296676033049?thread_ts=1708120844.771949&cid=C03RKP2LW64 I guess I hadn't read your message in detail 😆

Let me know if it works for you, we would like to revisit runner at some point I think there are more to it if we start to think about how to re-run pipeline from failure (adding memory to a kedro run) So far it's just idea but not much implemented

For us we want to run multiple "strategies" in parallel given input data, the outputs are independent of each other but the inputs overlap, so we built it all as "one big pipeline". It's the same thing I said in the thread I linked. We probably should've been structuring this code in a more modular way, so each strategy was its own pipeline, so one pipeline failing wouldn't affect the rest but 🤷

Interesting, in this case the soft fail runner would work but probably do more than you want. You will like the strategy stops as soon as error happens. It's something that is easy to do if you do it in a procedural way but kedro resolving dependencies itself

each "strategy" is a single node. so we have a subset of the pipeline which calculates the "current market state" then we run multiple nodes in parallel which produce different trades and then we combine those trades into a single file which goes to market. it's mostly a problem of design. the data preparation steps are quite coupled with the strategy steps. ideally we would have one pipeline which computes the current market state, and then shoot off multiple different pipelines all of which do their own thing independently, and then have a final step which aggregates the outputs

lol