Matěj Pekár
11/16/2023, 8:22 PMTakieddine Kadiri
11/17/2023, 8:18 AMNok Lam Chan
11/17/2023, 8:51 AMOur current approach involves parametrizing the filepath in the data catalog and passing it as a CLI argument. This is then executed by a wrapper script which applies the same pipeline to a whole folder by simply calling kedro with different arguments. Unfortunately, this method is very inefficient and seems suboptimal.Is that a second-order pipeline? That is you use a kedro pipeline to generate another kedro pipeline? I agree with @Takieddine Kadiri
PartitionedDataset
and additionally dataset factory may help.
docs.kedro.orgMatěj Pekár
11/17/2023, 9:58 PMTile Pipeline
that we want to run either independently or in the Image Pipeline
(not quite sure how to do this properly). The second part is being able to execute the "Image Pipeline" for set of images. Currently, we achieve this by parameterizing the data catalog entry and executing Kedro with the file name as CLI parameter. However, this is very inefficient. We have multiple pipelines like this so we are seeking for a general solution.
I don't see how PartitionedDataset
could help us. We are likely dealing with second-order pipelines, but I can't find any information about that in the docs.Nok Lam Chan
11/18/2023, 5:23 AMcreate N tiles
Step2: Run individual “tile pipeline” and concat.
Currently, we achieve this by parameterizing the data catalog entry and executing Kedro with the file name as CLI parameter. However, this is very inefficient. We have multiple pipelines like this so we are seeking for a general solution.Would you be able to give some example how do you do this exactly?
Takieddine Kadiri
11/18/2023, 10:12 AMLukas Innig
11/18/2023, 4:08 PM