:kedro:Mini Kedro challenge:
Let say the node name...
# randomn
Nok Lam Chan
03/07/2024, 2:33 PMKMini Kedro challenge:
Let say the node name is
create_derived_featureskedro-vizkedro run🧩 2
K 2
m
marrrcin
03/07/2024, 3:46 PMWDYM 🤔 ?
n
Nok Lam Chan
03/07/2024, 4:10 PM@marrrcin sorry maybe it's not clear, I mean how can you achieve this filtering either via the CLI options or the pipeline API
i
Iñigo Hidalgo
03/07/2024, 4:47 PMusing the pipeline API:
probably do some slicing of to-nodes create_derived_features and from-nodes create_derived_features and add the pipelines together, exploiting the deduplication which happens when adding pipelines
n
Nok Lam Chan
03/07/2024, 7:25 PM@Iñigo Hidalgo correct, basically a union of pipeline.filter with to nodes and from nodes. I feel like this can be done more easily without think about set.
Nok Lam Chan
03/07/2024, 7:26 PMIt’s also not possible to select multi pipeline, you will have to register the pipeline in advance. Would that be useful or it’s not a problem because you only need to register once so it’s not worth it?
i
Iñigo Hidalgo
03/07/2024, 9:45 PMPersonally I don’t use the pipeline slicing feature too often other than debugging. I tend do build up from smaller pipelines which I progressively sum together
n
Nok Lam Chan
03/07/2024, 11:39 PM@Iñigo Hidalgo that make sense. There are many alternatives, for example always use a bigger pipeline and then use tags to filter.
Nok Lam Chan
03/07/2024, 11:41 PMhttps://docs.getdbt.com/reference/node-selection/graph-operators
dbt have something like "+mymodel+", which mean running upstream & downstream of a node. As you notice this is not possible to do in Kedro CLI. I guess ML pipeline is different from data pipeline and rarely need these advance filters
10 Views