Hey, I have a question about

databricks.ManagedTableDataset

. I have a

PartitionedDataset

in which about 30 files are stored. In a node I read this

PartitionedDataset

, process it and now I want to create a new table in Databricks for each original file. So the output of the node should be a

PartitionedDataset

of the type

databricks.ManagedTableDataset

. Unfortunately, I can't do this because ManagedTableDataset has no filname etc. Does anyone have an idea how I can still realize this?

the PartitionedDataSet is designed around a filepath

ah okay i was afraid of that :D But thank you anyway for the quick response

I think we may actually have a better answer for you - this has kicked off a discussion on the dev channel

Oh exciting🤩, I can't access the dev channel, can I?

Hi @Simon Wolf - we’ve discussed this internally and we recommend to use the approach with using a generator in the node to

yield

your tables (alongside with their names / other properties that you need to save them) and implementing a dataset inherited from the original

databricks.ManagedTableDataset

to handle that. Pseudo code:

# node:
def node(partitioned_dataset):
    for partition, loader in partitioned_dataset.items():
        # do your thing
        yield (partition, <put your data here>)

# dataset:
class YourDataSet(databricks.ManagedTableDataset):
    def _save(self, data_to_save: tuple):
        partition, data, other_properties = data_to_save
        # handle data saving

Also @Simon Wolf if you want to join the Technical Steering Comittee you can be part of the dev channel 😛

@datajoely Ah haha I see 😂Thank you very much for discussing this. I hadn't seen the possibility of using nodes with a generator before but that looks super promising! At the moment the solution idea is not 100% clear to me but I will figure it out👍

Let me know if you need further assistance 😉