https://kedro.org/ logo
#questions
Title
# questions
f

Francis Duval

01/10/2024, 2:13 PM
Hello all! I have a pipeline A which depends on a dataset in another pipeline B. For now, this is a MemoryDataset, so when I run my pipeline with
kedro run --pipeline pipeline_a
, it is obviously not working since the dataset only exists at runtime. This dataset, named
text_processor
, is in fact an instance of a custom Python class. So when I run pipeline A, I get the error
ValueError: Pipeline input(s) {'text_processor'} not found in the DataCatalog
. I should therefore put this dataset in the DataCatalog. I'm not sure how I can do this. I have never saved an instance of a class before, but my guess would be to save it with pickle.
d

datajoely

01/10/2024, 2:16 PM
yes - see if a pickle works (try different engines too)
❤️ 1