https://kedro.org/ logo
#questions
Title
# questions
s

saeed choobani

08/11/2023, 7:51 AM
Hi Kedro team! I have a scenario where I need to load a file from multiple environments and aggregate them. What is the best practice for doing so? Should I create a custom dataset or are there better ways of doing it?
d

datajoely

08/11/2023, 8:39 AM
When you say multiple environments want do you mean?
s

saeed choobani

08/11/2023, 8:53 AM
So i have multiple data catalogs and I have multiple
envs
to access them.
d

datajoely

08/11/2023, 8:53 AM
ah and you want to bring them together?
👍 1
it’s not straight forward
s

saeed choobani

08/11/2023, 8:54 AM
a handpicked list of them & aggregate them in to a single input to my node.
d

datajoely

08/11/2023, 8:54 AM
intentionally so - it’s been designed in a way where they are isolated in accordance with the 12factor app principles
I think creating an
aggregate
environment with that sort of thing is the best way to do it
If you want to do this dynamically I think the
after_context_created
hook you could dynamically play around with things before the catalog is created with the
env
logic applied, but it’s a bit complicated
s

saeed choobani

08/11/2023, 8:57 AM
Thanks for your suggestions. I'll look into them. 🙌