Hi:) I have 3 different pipelines and 2 different ...
# questions
n
Hi:) I have 3 different pipelines and 2 different catalogs: catalog.yaml for the first and second pipeline and catalog_p3.yaml for the third pipeline. Is there any way to access catalog filename from e.g. KedroContext durng the pipeline run? E.g. if i run third pipeline i want to access filename "catalog_p3.yaml" to be able to log this file into mlflow. Is there any way to force pipeline p3 to use only catalog_p3.yaml instead of merging it with catalog.yaml, which is used by 1 and 2 pipeline?
a
Hey Nikola, The way that the config loader works is, it loads all the catalog files that follows a certain naming pattern, in this case
catalog*
and then merges all the keys together.
KedroContext
and the
OmegaConfigLoader
do not maintain a record of which config comes from which file. What is your usecase for the separate catalogs? Do they contain duplicate keys that you want different pipelines to use different configs?
n
If you choose to have a separate config file per pipeline strictly, maybe consider adding a namespace to it so there is a clean separation?