Anybody using some kind of config file to define t...
# questions
b
Anybody using some kind of config file to define their pipelines they want to share?
n
I know team define nodes and pipelines. @datajoely has an idea to took this a bit further. Basically pipeline that is serialisable.
d
Yeah that prototype is here https://github.com/AlpAribal/kedro-inspect, but we’ve also observed YAML pipelines to scale poorly since you end up losing all help from the IDE / code analysis tooling
this book by the Google SRE team is also gold and informs why we ended up going all in with OmegaConf https://sre.google/workbook/configuration-specifics/
b
@datajoely I have that same issue with datasets and parameters being strings, how do you work around that?
d
So @Nok Lam Chan is working on
kedro-lsp
which should provide in-IDE validation - also a good question if this can be used as a static analysis tool too 🤔
b
oh that's very exciting