Hello all! I have a pipeline A which depends on a dataset in another pipeline B. For now, this is a MemoryDataset, so when I run my pipeline with

kedro run --pipeline pipeline_a

, it is obviously not working since the dataset only exists at runtime. This dataset, named

text_processor

, is in fact an instance of a custom Python class. So when I run pipeline A, I get the error

ValueError: Pipeline input(s) {'text_processor'} not found in the DataCatalog

. I should therefore put this dataset in the DataCatalog. I'm not sure how I can do this. I have never saved an instance of a class before, but my guess would be to save it with pickle.

yes - see if a pickle works (try different engines too)