Recently I've been dealing with a huge number of namespaced pipelines. A really common operation I need to do is "get all datasets with the pattern `{ns1}.{ns2}.{ns3}.dataset`" Where I specifically need to know what

{ns1}

,

{ns2}

, and

{ns3}

are to make some decisions in a reporting node. Specifically, how axes are named in some plots. Is there any built in way to do this?

Th way I used to do this is leverage settings to define the ns1, ns2, ns3 range, and use a dynamic pipeline to spawn each of the pipeline instances, then collect a final reporting node with all their outputs

Yeah that's more or less where I am I just have so many namespaces that its cumbersome haha Maybe I'm just holding it wrong at this point 😁

Nice blog post! I'm already doing this actually My final namespaced datasets look like

{ns1}.{ns2}.{seed}.{ns3}.{ns4}.{ns5}.{ns6}.{ns7}.synthetic.data

(anonymized of course) So even just constructing the right pattern in the right order is my issue I have these namedtuples floating around that help me a little bit with keeping track of them I feel like maybe I should do something smarter in the dataset naming like

ns1={ns1}

to keep things straight

Why so many namespaces?

its a big experiment that I'm running some namespaces are names of models, some datasets lots of levers that I want to be able to split out with tags / pipeline names (also tough) this step that i'm working on is essentially what you're suggesting - merging them all into one dataset that easier to work with