Is there a way to have files in the Kedro Catalog, that are missing? I have a kedro pipeline which generates a file that is used again for the next run of that same pipeline. However, when the pipeline runs for the first time, that file does not exist, and it is handled in a node in the pipeline. Kedro throws an missing file error here at this time. Is there a way this can be handled through Kedro? Maybe add an catalog parameter missing=True or optional=True, and Kedro can safely ignore the file? How I currently implemented the solution was to create an empty...

wow this is hard. I tried for some time to use hooks for this but I failed https://github.com/kedro-org/kedro/issues/2690#issuecomment-1607746840 I ended up replying "no, this is not possible" https://stackoverflow.com/a/76558253/554319 cc @datajoely (since you also 👀 the question)

I think this is the right solution

but doing a

catalog.add(..., replace=True)

inside the hook didn't affect the

catalog.load

call outside the hook 😢

I reckon if you did

after_context_created

it may work

same result 👎🏼

yeah agreed!