Can anyone explain to me why Kedro attempts to load all catalog definitions, even if running only a specific Pipeline that uses a subset of the catalog? For example, let's say I have a catalog with

input

,

output

and

wrong

entries.

wrong

has a configuration problem (e.g. no credentials could be found), but I'm running a pipeline

mypipeline

which only uses

input

and

output

. Why does

kedro run --pipeline mypipeline

fail if

wrong

is configured improperly in this case? I get that you usually want to be able to view the entire catalog, but is

--pipeline <...>

not enough information to let Kedro know that I potentially don't want that?

Catalog entires are lazily evaluated wherever we can, however datasets which require remote execution like SQL or Spark need to be eagerly evaluated. What you can do is use different configuration environments and swap out the badly configured catalog entries for dummy MemoryDataSets