I m inexperienced so this is basic question I m trying to ad

Question

I m inexperienced so this is basic question I m trying to add datasets programmatically I ve made a catalog py file that contains `from import DataCatalog` `from import PartitionedDataSet` `from kedro extras datasets pandas import CSVDataSet` `from kedro config import ConfigLoader` `conf paths = conf base conf local ` `conf loader = ConfigLoader conf paths ` `atlas regions = conf loader get atlas regions A yml file consisting of regions with names` `catalog dictionary = ` `for region in atlas regions regions ` `name = region name ` ` catalog dictionary f name data right = PartitionedDataSet path = ClinicalDTI R VIM ` ` dataset = nifti NIfTIDataSet filename suffix = f seedmasks name R T1 nii gz ` `catalog dictionary f name data right = CSVDataSet filepath = data 01 raw iris csv ` ` catalog dictionary f name data right output = CSVDataSet filepath = data 01 raw iris csv ` `io = DataCatalog catalog dictionary ` `print io list ` Kedro version 0 17 7 Running catalog py prints the expected list of datasets But what do I need to do to be able to use these datasets in a pipeline

datajoely · Answer

Hi < Anjali Datta> since you say you re new to Kedro I d highly recommend you follow the tutorials since this approach isn t the recommended approach

Deepyaman Datta · Answer

< datajoely> I think this use case isn t covered by Spaceflights because the data layout is complex configurable set of regions `PartitionedDataSet` for each region based on the region name presumably want to run a pipeline for each region to create output for that region

Deepyaman Datta · Answer

< Anjali Datta> I ve created a quick and dirty example of how you can have a dynamic catalog + pipelines using Jinja see This is the diff on top of just creating a new project named Jinja Example Cons of this approach I ve defined `regions` in two places because you can t use something like `globals yml` inside Jinja In Kedro 0 19 I think a new OmegaConfLoader will be the preferred way to go and I don t think that support Jinja I m not 100 sure how this use case would be best handled there Too much Jinja makes pipelines confusing I think this use case for reused modular pipelines is fair though If you aren t familiar with namespacing reuse of modular pipelines see I can try and add an example of keeping the pipeline definition in Python and using with pipelines as an alternative even though i don t think it s well documented P S I used Kedro 0 18 6 which includes some stuff like pipeline autodiscovery if you try to replicate with 0 17 7 you will need to add data processing explicitly

Anjali Datta · Answer

Thank you so much < datajoely> and < Deepyaman Datta>