Alexandre Ouellet
06/14/2023, 7:07 PMNok Lam Chan
06/14/2023, 9:29 PMAlexandre Ouellet
06/15/2023, 2:08 AMIñigo Hidalgo
06/15/2023, 9:25 AMpipeline = node_a -> versioned_dataset -> node_b
pipeline_1 => node_a_1 (slow) -> versioned_dataset_1 !!! versioned_dataset_2 -> node_b_1
pipeline_2 => node_a_2 (fast) -> versioned_dataset_2 -> node_b_2
pipeline_2 starts slightly after pipeline_1, but node_a_2 finishes before node_a_1 which leads to node_b_1 getting versioned_dataset_2 as an input?Alexandre Ouellet
06/15/2023, 1:29 PMIñigo Hidalgo
06/15/2023, 1:51 PMlatest/
which in both cases should be the correct version, as node_b_2
would start as soon as versioned_dataset_2
was generated by node_a_2
, and node_b_1
would start right after node_a_1
generates versioned_dataset_1
.
if instead of forcing the pause through debugging you make node_a
sleep
according to a certain parameter, and make that parameter bigger in one of the two runs, does the behavior still occur?Alexandre Ouellet
06/15/2023, 1:53 PMNok Lam Chan
06/15/2023, 2:25 PMAlexandre Ouellet
06/15/2023, 2:26 PMNok Lam Chan
06/15/2023, 2:27 PMAlexandre Ouellet
06/15/2023, 2:27 PMNok Lam Chan
06/15/2023, 2:28 PMAlexandre Ouellet
06/15/2023, 2:28 PMNok Lam Chan
06/15/2023, 2:28 PMAlexandre Ouellet
06/15/2023, 3:40 PMNok Lam Chan
06/15/2023, 3:54 PMAlexandre Ouellet
06/15/2023, 7:27 PMNok Lam Chan
06/16/2023, 11:41 AMWARNING /Users/Nok_Lam_Chan/GitHub/kedro/kedro/io/core.py:632 warnings.py:109
: UserWarning: Save version
'2023-06-16T11.33.37.506Z' did not match load version
'2023-06-16T11.33.52.306Z' for
CSVDataSet(filepath=/Users/Nok_Lam_Chan/dev/test/kedr
o-versionned-dataset-bug/data/01_raw/versionned_datas
et.csv, load_args={}, protocol=file,
save_args={'index': False},
version=Version(load=None,
save='2023-06-16T11.33.37.506Z')). This is strongly
discouraged due to inconsistencies it may cause
between 'save' and 'load' operations. Please refrain
from setting exact load version for intermediate data
sets where possible to avoid this warning.