Hello everyone, Is there a way to force a pipeline to run its nodes in the order they are declared within the pipeline ? I know that I can create a dummy output just to force one node to run after another but I’d like to know if there is a better way to accomplish this Thanks

Why do you need ordering other than topologically sorted one?

I have 2 nodes in my pipelines: • Node 1: Does some processing and saves a parquet file on S3 • Node 2: has a sql query as input. The sql query creates an external table in snowflake that uses that saved parquet file So, node 2 depends on node 1 but not in an explicit way (the input of node 2 is not dependent on the output of node 1)

It would be safer to return some metadata (e.g. a path to the parquet file saved on s3) from node 1 and use it as an input to the node 2

I think that is what I’m going to do 👍

Yeah fake ‘pass through’ nodes are the way to do this if you really need, but topological should get you most of the way

Hi, I am super late on this but it triggered interest. I am not totally sure about the suggestion. The problem above is a regular one. I don't see how passing the file path is compatible with the idea of the catalog. The catalog allows for defining the datasets. If you pass the file path, it means that you either define the dataset within the node (not using the genuine load mechanism) or you do not use a dataset but genuine library functionalities like read_sql in pandas. Am I missing something?

You can pass anything from the node, just for the sake of telling Kedro that there’s a dependency between nodes. It does not have to be path or anything.

Ah OK! My apologies, I misunderstood the reply. So it's actually like a dummy output. Like a boolean has_finished has output/input in the sequence.

Yup

Thanks! And sorry for the entropy created.