Hey fellas,
I have 100 pipeline with their own na...
# questionsc
Chandan Malla
09/27/2023, 7:05 AMHey fellas,
I have 100 pipeline with their own namespace created using loop , each namespace has its own input which is stored in a common table in MySQL. How do I pass those input effeciently to my pipelines.
m
marrrcin
09/27/2023, 7:14 AMIf what you’re looking for is effectively “how to pass the same data catalog entry to each namespaced pipeline” then when using:
Copy code
from kedro.pipeline.modular_pipeline import pipeline
pipeline(
inputs={"<name of the dataset within the modular pipeline": "name of the dataset from the *root* data catalog"},
namespace="your_ns",
# ... other pipeline params
)"<name of the dataset within the modular pipeline"c
Chandan Malla
09/27/2023, 7:15 AMNo, every input is different, I am creating a bscktesting pipeline. Each pipeline will have different input
m
marrrcin
09/27/2023, 7:16 AMSo just create entries in the data catalog with namespaces, like this:
Copy code
your_ns.dataset_name:
type: xyzc
Chandan Malla
09/27/2023, 7:17 AMYes that would work, but I have to create 100 files for 100 input :/ instead of directly fetching it from single DB source
m
marrrcin
09/27/2023, 7:17 AMIs there some kind of a pattern in the data catalog?
marrrcin
09/27/2023, 7:18 AMIf there is, then you can use dataset factories if you’re on
Kedro >= 0.18.12 Copy code
"{namespace}.dataset_name":
type: xyz
filepath: "path/to/data/{namespace}/file.csv"c
Chandan Malla
09/27/2023, 7:18 AMPipeline would roughly like this, except there would 100 of them with each having their own nampespace
m
marrrcin
09/27/2023, 7:19 AMmarrrcin
09/27/2023, 7:19 AMYou can do this with custom names too, like:
Copy code
"{namespace}.data_from_{month}":
type: xyz
filepath: "path/to/data/{namespace}/{month}.csv"👍 1
marrrcin
09/27/2023, 7:20 AMWill that work for you?
c
Chandan Malla
09/27/2023, 7:20 AMHow do I change month every time in each pipeline input?
Chandan Malla
09/27/2023, 7:21 AMOh got it
m
marrrcin
09/27/2023, 7:21 AMYou’re generating the pipeline via code, so I guess that you have those
MarJanc
Chandan Malla
09/27/2023, 7:22 AMThankss mate
m
marrrcin
09/27/2023, 7:22 AMDoes that solve your problem?
c
Chandan Malla
09/27/2023, 7:25 AMI can use this as an workaround, i will use month as a parameter to pass in my SQL query, that will help me to get different input every time.
👍 1
Chandan Malla
09/27/2023, 8:39 AMHow do I add {namespace} in parameter.yml also?
2 Views