Hey y all A tough one for me at least say my data is a month

Question

Hey y all A tough one for me at least say my data is a monthly time series I want to train one model per month I can do it easily with a for loop but that won t allow me to run in parallel Is there a kedro esque way to do this using maybe modular pipelines I think I know how to do it if there was a fixed number of months but that is not the case

Ian Whalen · Answer

You can parallelize the for loop with something like joblib If you wanted to use kedro to parallelize with `ParallelRunner` you d probably have to use some black magic which wouldn t be too nice Maybe this is helpful <https stackoverflow com questions 29589327 train multiple models in parallel with sklearn>

Ricardo Araújo · Answer

I d rather use kedro itself I am already doing paralell training of different models but on same data The difficulty is training each model with different data a grid of parallel training

Ricardo Araújo · Answer

I don t think this is something supported at all I could load the data inside the pipeline and create many different pipelines in there but it breaks completely Kedro s semantics

Ricardo Araújo · Answer

Thanks for the pointers though < Ian Whalen>

Deepyaman Datta · Answer

The pipeline structure needs to be determined before you run it So if you want to run something based on the structure of the data you ll need something that reads the data e g to determine the set of months in some file and uses that to dynamically construct a Kedro pipeline and then execute it

Ricardo Araújo · Answer

Yep results don t look good maintenance becomes a pain

Ricardo Araújo · Answer

Maybe that s a nice feature if someone can fully specify Basically there would have to be a way for pipelines to be dynamically instantiated based on the output of a node Say we have a pipeline `consumer` that gets a single number and do something with it We also have `generator` which generates a list of numbers of unknown size It should be possible to connect these two by saying that `generator` creates a sequence of outputs that `consumer` can use and Kedro under the hood would take care of this by instantiating multiple `consumer` pipelines each being passed an element provided by `generator`

Deepyaman Datta · Answer

Do you know of other pipelining frameworks by chance that provide this kind of behavior I know lot of orchestrators allow loops conditional branches etc but not familiar with dynamic generation of pipelines partway through

Ricardo Araújo · Answer

I m not familiar with any other framework that provides this

Ricardo Araújo · Answer

As a fun note I asked ChatGPT how to solve this and it hallucinated a `kedro runner TaskRunner` that kinda would do the trick It allowed for a pipeline name to be passed and instantiated inside a node with remapped inputs

Ricardo Araújo · Answer

This would solve the problem of not being able to import a pipeline inside the nodes module because of circular references

Ricardo Araújo · Answer

But now I wonder if it would be possible to import a separate pipeline module altogether hm

Ricardo Araújo · Answer

Tried that turns out that importing a separate pipeline module inside a node from another pipeline breaks something and Kedro is unable to figure out the graph

Ricardo Araújo · Answer

That would have done it I think I d instantiate many modular pipelines inside a node

Ricardo Araújo · Answer

I m thinking a solution would be to read the dataset in the catalog from the pipeline definition so that I can instantiate as many pipelines as the data needs Is that possible at all < Deepyaman Datta>

Deepyaman Datta · Answer

That s similar to what I said in <https kedro org slack com archives C03RKP2LW64 p1678489685247699 thread ts=1678476079 170489 amp cid=C03RKP2LW64> However as mentioned there you need to know the data before you construct the pipeline

Ricardo Araújo · Answer

Yes But how to read the data in the pipeline py file

Ricardo Araújo · Answer

For this to be minimally organized data would have to come from the catalog

Ricardo Araújo · Answer

I m trying to use KedroSession but so far no luck

Deepyaman Datta · Answer

What you re saying should work What problem are you running into

Ricardo Araújo · Answer

`kedro framework session get current session` seems to have been deprecated

Ricardo Araújo · Answer

In 1 18

Ricardo Araújo · Answer

Not sure how to load the current session

Deepyaman Datta · Answer

I ll look into it tomorrow morning if I have some time else I ll loop somebody in I don t know off the top of my head without trying it quickly

Ricardo Araújo · Answer

I think I managed to do it ```from kedro framework project import configure project from kedro framework session import KedroSession with KedroSession create package name= lt project name gt as session context = session load context catalog = context catalog```

Ricardo Araújo · Answer

Then `catalog load data ` will work

Deepyaman Datta · Answer

Ah I was literally about to post this ``` with KedroSession create as session context = session load context catalog = context catalog```

Deepyaman Datta · Answer

So seems we wrote pretty much the exact same thing independently slightly smiling face

Ricardo Araújo · Answer

Yes lol Thanks for the help Still a bit hacky but will do for the moment

Deepyaman Datta · Answer

I don t think you need to pass `package name` or import `configure project` or at least I didn t

Ricardo Araújo · Answer

That makes sense I don t think it is needed

Ricardo Araújo · Answer

I still have to register catalog items programmatically as well otherwise it won t store the output of those many pipelines

Ricardo Araújo · Answer

But I think that can be done using catalog api

Deepyaman Datta · Answer

Yeah Sometimes I think `PartitionedDataSet` can be a good way to be able to define once in a structure for each dataset but that requires writing the pipeline in a different way and has it s own limitations so I think what you say is good for now

Deepyaman Datta · Answer

FWIW I noticed if I ran into an error in that catalog loading piece Kedro swallowed it ``` diamonds deepyaman Deepyamans MacBook Air spaceflights kedro run 03 13 23 07 29 03 INFO Kedro project spaceflights session py 340 03 13 23 07 29 04 WARNING opt miniconda3 envs diamonds lib python3 10 site packages kedro framework project init py 359 UserWarning An error occurred while importing the warnings py 109 spaceflights pipelines data science module Nothing defined therein will be returned by find pipelines Traceback most recent call last File opt miniconda3 envs diamonds lib python3 10 site packages kedro framework project init py line 357 in find pipelines pipeline module = importlib import module pipeline module name File opt miniconda3 envs diamonds lib python3 10 importlib init py line 126 in import module return bootstrap gcd import name level package level File lt frozen importlib bootstrap gt line 1050 in gcd import File lt frozen importlib bootstrap gt line 1027 in find and load File lt frozen importlib bootstrap gt line 1006 in find and load unlocked File lt frozen importlib bootstrap gt line 688 in load unlocked File lt frozen importlib bootstrap external gt line 883 in exec module File lt frozen importlib bootstrap gt line 241 in call with frames removed File Users deepyaman github kedro org kedro spaceflights src spaceflights pipelines data science init py line 3 in lt module gt from pipeline import create pipeline NOQA File Users deepyaman github kedro org kedro spaceflights src spaceflights pipelines data science pipeline py line 10 in lt module gt context = session context AttributeError KedroSession object has no attribute context warnings warn INFO Loading data from companies CSVDataSet data catalog py 343 INFO Running node preprocess companies node preprocess companies companies gt preprocessed companies node py 327 03 13 23 07 29 05 INFO Saving data to preprocessed companies ParquetDataSet data catalog py 382 INFO Completed 1 out of 3 tasks sequential runner py 85 INFO Loading data from shuttles ExcelDataSet data catalog py 34``` So maybe just something to be aware of I can also raise an issue on this later