Gary McCormack
04/11/2023, 2:08 PMconf/base/parameters.yaml
file
param_1: foo
param_2: bar
my_nice_new_params:
If I have the following toy code block:
@hook_impl
def before_node_run(..args.., catalog: DataCatalog, ..more_args..)
print(catalog._get_dataset('params:my_nice_new_params'))
new_param1, new_param2 = run_some_super_cool_logic()
catalog.add_feed_dict(
{
'params:my_nice_new_params':[new_param1, new_param2]
}, replace=True
)
print(catalog._get_dataset('params:my_nice_new_params'))
Then the printed stdout will be something like
MemoryDataSet(data=<NoneType>)
MemoryDataSet(data=<list>)
which is what I would have hoped for. However, when the node itself is run and accesses the 'params:my_nice_new_params'
, the original None
value remains. Is there a step that I'm missing that saves the most recent instance of the catalog?Antony Milne
04/11/2023, 2:39 PMbefore_node_run
hook to return a dictionary {"params:my_nice_new_params": [new_params1, new_params2}
JSONDataSet
for this sort of thing if you want to persist it to disk.