hi team, just a quick question. let's say I have output O1 from node1 with the associated catalog configured so the content of O1 will be saved to CSV. node2 will use O1 as input. The current behaviour is that node2 will reload the data from O1 file instead of from memory (this is expected I assume, due to the catalog configuration). Is there any way I could still have O1 saved as CSV (easier for business people to check data quality) while having O1 loaded to node2 through memory (faster and no need to deal with csv save/load tricks), Thanks
02/08/2023, 2:20 AM
You could define two output datasets from node1. One of the two datasets would be saved as a CSV (i.e. there is a catalog entry for CSV), while the other is simply referenced by node2 (i.e. no catalog entry, and therefore it's an "in-memory" dataset).
Let me know if that makes sense.
(I'm guessing it would require minimal changes to work with 0.18.x, but nobody's asked; if this solves your problem and you'd use it, I'm happy to try and find some time to update it, or of course happy to accept PRs)
02/08/2023, 3:20 AM
Cool, thanks all, I will start with option 1 (logically, it should work without issue) and move to a more programmatic solution (accelerator) if get extra time.
Good to know the accelerator was built so I know I was not asking for a meaningless use case 🙂