Hi Team 🦜, How can I limit the loaded config for native latest kedro in 2 dimensions, eg. `kedro run --pipeline dc_xyz --env dev`: 1. by env (

conf/dev/

) 2. by pipeline (

conf/base/data_connectors/xyz/

) Is there a simple way to achieve this double filter without much hacking?

do the two approaches not work in combination?

I think I’ve just never done either, if I understand correctly, the default is that we pick up every config object in conf if we don’t specify an environment, right? But that will also mean scanning through all the other entries and pipelines we don’t want to know about. We are building a setup where we only need two things, the conf for that particular pipeline, and the conf for that particular env

so it will do an AND and it’s not easy to configure it differently

but that still means one initiation, one spark session, etc?

I think it’s two parallel initiations, performance is cluster specific

Slightly confused here, is the question about running two separate pipelines? Or is it because Kedro loading config that aren’t necessary in this case?

@Nok Lam Chan definitely don’t want to run two separate pipelines. We have dev and prod environments, and we want to avoid duplicating catalogs or params if possible, and we want to avoid loading configs we don’t need. So if we have pipelines A, B and C, and environment X and Y, we want to have • catalogs and params that are pipeline-dependent but not env dependent (which for now we put into

base/

), and • params that are env dependent (which we for now put into

dev/

and

prod/

) And if we are running pipeline A in environment X, we should process the catalog/param related to A, and the params related to X, but no B, C or Y. Does that make sense?