Hey fellas, I have 100 pipeline with their own namespace created using loop , each namespace has its own input which is stored in a common table in MySQL. How do I pass those input effeciently to my pipelines.

If what you’re looking for is effectively “how to pass the same data catalog entry to each namespaced pipeline” then when using:

from kedro.pipeline.modular_pipeline import pipeline

pipeline(
   inputs={"<name of the dataset within the modular pipeline": "name of the dataset from the *root* data catalog"}, 
   namespace="your_ns",
   # ... other pipeline params
)

Note that

"<name of the dataset within the modular pipeline"

is WITHOUT the namespace

No, every input is different, I am creating a bscktesting pipeline. Each pipeline will have different input

So just create entries in the data catalog with namespaces, like this:

your_ns.dataset_name:
   type: xyz

Yes that would work, but I have to create 100 files for 100 input :/ instead of directly fetching it from single DB source

Is there some kind of a pattern in the data catalog?

Pipeline would roughly like this, except there would 100 of them with each having their own nampespace

How do I change month every time in each pipeline input?

You’re generating the pipeline via code, so I guess that you have those

Mar

,

Jan

, `Feb`… etc in some kind of a list

Thankss mate

Does that solve your problem?

I can use this as an workaround, i will use month as a parameter to pass in my SQL query, that will help me to get different input every time.