Hello! I was wondering if any of you know or has seen using external none python programs inside a Kedro pipeline. For context, I'm dealing with large raster files and using python workarounds at dataset level using rioxarray or even rasterio is quite challenging and problematic to get it right and not memory hungry. On the other hand there is the command line program

gdal

, which is a CLI for a c++ lib (the one used under the hood by rasterio), that can handle the large files without problem because it does all the streaming and all sorts of nice things under the hood. So I want to integrate this processing in my existing pipeline. Is it a bad idea to have a custom dataset that calls an external program via

subprocess

or something similar ? have you ever seen a pattern like this before? Will God kill a kitten if I go with this approach? Thank you!

Hi @Biel Stela! No kittens will be harmed 🐈 You can absolutely integrate *n*on-Python external programs into a Kedro pipeline. Kedro doesn’t care what’s inside your node/dataset - only that inputs and outputs are well-defined and reproducible. A custom dataset or custom node that calls

subprocess.run(["gdal", ...])

is fine and often practical for large data or memory-intensive processing. If you want Kedro to treat the GDAL step as a dataset I/O operation (part of the data catalog), make a custom dataset or alternatively, just make a Kedro node that calls the CLI.