I found a bug in the kedro-aws-batch deployment.
When you run a pipeline other than your default pipeline and this contains nodes that are not in your default pipeline you will get an error saying that the node cannot be found when the container runs of batch,
This is because in the session when it looks for the nodes it only looks for nodes in the default pipeline and not all the nodes.
my quick and dirty fix was to add the non-default pipeline to my default pipeline.
Now i wanted to rely on the find_pipelines and sum(pipelines), but that wasnt possible because i reuse pipelines to create subset for training/inference and testing, so there are duplicate nodes.
There are two parts to solution:
1. let the session look for all the nodes in all the pipelines, so it will be found.
2. also send the pipeline (or tag, etc) args to batch. Basically, now the runner on batch is running kedro run --env=aws_batch --runner=AWSBatchRunner --node=NameOfNode, but that will only look in the default pipeline. if the tags and pipeline and other args are passed on this issue will be resolved