https://kedro.org/ logo
#random
Title
# random
n

Nok Lam Chan

03/07/2024, 2:33 PM
KMini Kedro challenge: Let say the node name is
create_derived_features
, how can you filter the highlighted pipeline that shown in
kedro-viz
? (with Python API or
kedro run
CLI) https://demo.kedro.org/?pipeline_id=__default__&selected_id=04ba733a
🧩 2
K 2
m

marrrcin

03/07/2024, 3:46 PM
WDYM 🤔 ?
n

Nok Lam Chan

03/07/2024, 4:10 PM
@marrrcin sorry maybe it's not clear, I mean how can you achieve this filtering either via the CLI options or the pipeline API
i

Iñigo Hidalgo

03/07/2024, 4:47 PM
using the pipeline API: probably do some slicing of to-nodes create_derived_features and from-nodes create_derived_features and add the pipelines together, exploiting the deduplication which happens when adding pipelines
n

Nok Lam Chan

03/07/2024, 7:25 PM
@Iñigo Hidalgo correct, basically a union of pipeline.filter with to nodes and from nodes. I feel like this can be done more easily without think about set.
It’s also not possible to select multi pipeline, you will have to register the pipeline in advance. Would that be useful or it’s not a problem because you only need to register once so it’s not worth it?
i

Iñigo Hidalgo

03/07/2024, 9:45 PM
Personally I don’t use the pipeline slicing feature too often other than debugging. I tend do build up from smaller pipelines which I progressively sum together
n

Nok Lam Chan

03/07/2024, 11:39 PM
@Iñigo Hidalgo that make sense. There are many alternatives, for example always use a bigger pipeline and then use tags to filter.
https://docs.getdbt.com/reference/node-selection/graph-operators dbt have something like "+mymodel+", which mean running upstream & downstream of a node. As you notice this is not possible to do in Kedro CLI. I guess ML pipeline is different from data pipeline and rarely need these advance filters
7 Views