Hi Team 🦜
I have a quite complex Kedro pipeline that spends several minutes getting through configloaders, when it's starting to run. In itself this is fine, but I'm struggling to spin up the kedro kernel in a jupyter notebook or jupyterlab, because they all time out. Is there a way to increase timeout in CLI or a config file I missed - also, is maybe my assumption wrong that this could cause timeout errors? (I'm guessing that because other pipelines with less of a configloader leadtime can spin up their kernels in an otherwise identical environment)
04/25/2023, 4:55 PM
I've run into these issues myself in the past, and I think the best thing to do is to run a profiler; it's possible it will highlight something that may be inefficient by accident (e.g. some regex substitution used to eat up the majority of the time for config loading when you had a lot of entries).
Few minutes on config loaders sounds really excessive, so would love to understand where this time is being spent (as well as the rough size of the pipeline)!
05/02/2023, 1:47 PM
Yes, the team did profile it and naturally it turned out the configloader ran multiple times 🤡 - thanks for the suggestion 🦜