Hello ! 🙂 When I run a Kedro pipeline remotely on a Dataproc cluster, I get the following error:

ValueError: Failed to find the pipeline named 'XXXXXX'. It needs to be generated and returned by the 'register_pipelines' function.

However, when I run

kedro run --pipeline <pipeline>

locally on my machine, the pipeline is correctly detected and executed. Just to clarify, I do have an

__init__.py

file in the pipeline directory, and my

register_pipelines()

function uses

find_pipelines()

as shown below:

from kedro.framework.project import find_pipelines
from kedro.pipeline import Pipeline

def register_pipelines() -> dict[str, Pipeline]:
    pipelines = find_pipelines()
    pipelines["__default__"] = sum(pipelines.values())
    return pipelines

Do you have any idea what could be causing this issue on the cluster? Any insights or suggestions would be greatly appreciated. Thank you in advance!

Hey, is it possible that there are some dependencies missing on the cluster which is making the pipeline not be discovered?

Hello @Ankita Katiyar 🙂 Yes, that’s what I had thought too, but other pipelines are running fine with the same dependencies and configuration. It’s only this newly added cluster that fails to be identified… 😞

Can you show me the full log for this error? The actual reason for failures usually is sandwiched in the logs 😅

Indeed, after digging a bit deeper into the logs, I first focused on the last few lines. But by scrolling further up, I realized the issue was actually related to a module that the pipeline could not find. The logs are quite noisy. There is an initial part that shows the real root cause, followed by a section generated by Kedro, and finally a message saying the pipeline was not found. It is definitely not easy to read... Anyway, I really appreciate your help @Ankita Katiyar 🙂 Thank you again!

yeah, the logs are indeed quite noisy and it has been reported before. I’ll dig up the issue and make a note of it 🙂