Jens Peder Meldgaard
07/02/2025, 6:37 AMkedro-databricks
and I am uncertain of how to resolve it - anyone who can help me figure out what to do here? 🙏
https://github.com/JenspederM/kedro-databricks/issues/135
A bit of explanation:
The issue occurs when using namespaces
for pipelines, as it prepends the namespace
to any input and output resulting in, e.g., ValueError: Pipeline input(s) {'active_modelling_pipeline.X_train', 'active_modelling_pipeline.y_train'} not found in the DataCatalog
when using a namespace
called active_modelling_pipeline
.
When nodes are executed in Databricks, each node is executed in a workflow task with a command similar to kedro run --nodes <node-name> --conf-source <some-path> --env <some-env>
. Do I need to add the --namespace <some-namespace>
option to the invocation to get it to correctly resolve the catalog paths?