In kedro kubeflow, how do I get a run-once pipeline to use a new PVC? Every time I rerun a run-once, it tries to use the PVC from the original run which is not there any longer, and it hangs with:

This step is in Pending state with this message: Unschedulable: 0/5 nodes are available: 5 persistentvolumeclaim "spacetut-on-kfp-qcjxs-default-data-volume" not found. preemption: 0/5 nodes are available: 5 Preemption is not helpful for scheduling.

I tried using a new experiment name inside kubeflow.yaml but the new experiment still produces the same error…

I recall I had similar issues due to kubeflow caching turned on - have a look at this thread and try to disable it https://www.kubeflow.org/docs/components/pipelines/v1/overview/caching/

Thanks for the pointer. There is also

max_cache_staleness: P0D

in kubeflow.yaml. But it looks like this is a bit of a moving target: https://github.com/kubeflow/pipelines/issues/4857