Hey team 👋, Quick design question on a Kedro pipeline that ingests Kafka messages into a bronze layer (read from Kafka → write raw data to storage). The pipeline logic is simple; most of the config is really which topics to read and where to write. We package the project as a wheel and run it on Databricks. Today : • One generic pipeline for all ingestions • One shared Kedro environment • Source and sink in the catalog are generic — the actual topics and output path/table are passed at runtime via CLI or Databricks job params This works, but it feels a bit unnatural for Kedro: the catalog doesn’t really describe what’s being ingested, and its meaning depends on the run rather than the environment. Three options : • A - Runtime params (what we have now) One pipeline, one env, topics/table passed at each run. • B - Explicit catalog + registry One env, all source/sink pairs listed in the catalog. Same pipeline code, but one named pipeline per flow in the registry. • C - One env per flow Separate envs for each ingestion (local + Databricks). Each has a fixed source/sink in the catalog; same generic pipeline everywhere. Questions : 1. Which would you pick: A, B, or C? 2. Any better pattern we’re missing? 3. Do we care more about a single catalog overview or i*solating each flow in its own env*? We’re talking a small number of flows (not hundreds). One flow can also read from multiple topics into one bronze table. Thanks!

I'd be leaning towards B. Check out pipeline reuse by namespacing as well, it helps in these "same pipeline logic but different data" situations: https://docs.kedro.org/en/stable/build/namespaces/

Thank you Laura !

Alternatively you can use the data factory pattern, it does almost the same thing, but more interesting ad you can write to a catalog entry through a regex.

Thanks, Laurens. This will definitely help me. 🙏