Hi Kedro team...We ran a forecasting pipeline(runs...
# questionsb
Balachandran Ponnusamy
04/03/2023, 10:37 PMHi Kedro team...We ran a forecasting pipeline(runs 20 hours) having around 231nodes and it failed around 198 th node...now I want to run only the remaining failed 33 nodes , but then the error log/info logs doesn't provide a list of remaining nodes to rerun...Can you pls help on this...we definitely not want to run this forecasting pipeline again for 20 hours
d
Deepyaman Datta
04/03/2023, 11:34 PMAre any of the upstream nodes reading from all persisted inputs? I can't tell, because you seem to be using a very custom setup with some `IPythonXXDataSet`s and using
papermillMemoryDataSetb
Bernardo Sandi
04/03/2023, 11:57 PMHey @Deepyaman Datta,
The custom datasets created are inheriting AbstractDataSet.
We know that is not the problem, as we experience the same issue when running on the command line and using CSV and Parquet datasets
d
Deepyaman Datta
04/04/2023, 12:01 AMAre you dynamically modifying your catalog?
Deepyaman Datta
04/04/2023, 12:13 AMI tried making a node fail in Spaceflights, and I'm getting the expected behavior, so I don't think it's something that's regressed across the board. So probably need a minimal example or some more context if going to investigate,
b
Bernardo Sandi
04/04/2023, 12:42 AMYes, the catalog is being modified dynamically.
The forecasting steps are happening for 21 different groups, so the catalog is updated on a with a hook to alter the datasets
d
Deepyaman Datta
04/04/2023, 1:50 AMIf you look at the code for how to determine command to rerun, it depends on looking at the catalog to find ancestors nodes with persisted inputs. If you mess with the catalog (depending on what you do), it could be inaccurate.
👍 1
Deepyaman Datta
04/04/2023, 4:53 AMFor reference: https://github.com/kedro-org/kedro/blob/fa8c56fa2e510e6a449f5ac7356f76c167be978a/kedro/runner/runner.py#L215-L280
So, if you want to debug this, since you can replicate in a local run, see what the state of catalog is + what persistent ancestors gives you with a small pipeline with an error introduced (and with whatever catalog-modifying hook you've written).
👍 2
🙏 1
j
Juan Luis
04/04/2023, 1:41 PMhi folks, this is kind of a parallel conversation but I'm curious to know more about your setup. are you triggering Papermill from Kedro nodes to run Jupyter notebooks, or is it the other way around?
b
Balachandran Ponnusamy
04/04/2023, 1:57 PMFrom Notebooks we have a Kedro session.run() triggers the pipeline run.
j
Juan Luis
04/04/2023, 2:01 PMoh I see, so you use Papermill to launch a Notebook that triggers a Kedro pipeline ✔️
b
Bernardo Sandi
04/04/2023, 4:17 PMThanks, @Deepyaman Datta
@Logan Rupert @Rahul Agarwal for reference ^^
👍 1
Bernardo Sandi
04/10/2023, 5:40 PM@Dotun O ^^
👍 1
4 Views