Hi everyone `local catalog yml` does not override the `base

Question

Hi everyone `local catalog yml` does not override the `base catalog yml` Any idea what could cause this behavior Thx M

Nok Lam Chan · Answer

Which ConfigLoader you are using And can you show the relevant config for both `yml` files p s I know I still owe some answer to other questions you asked will get back to it

Marc Gris · Answer

image png

Marc Gris · Answer

oh I think I now see what s wrong sweat smile Let me check that

Marc Gris · Answer

Ok It s a bit stupid but putting my ego aside I ll share what was wrong because I d love to have you re advice on how to best handle this

Marc Gris · Answer

image png

Marc Gris · Answer

image png

Marc Gris · Answer

My assumption was that commenting out a dataset in the local catalog would result in this dataset not being created But of course comments being not evaluated the ` table db columns` in `base` does get created in the catalog

Marc Gris · Answer

Nota bene This dataset only concerns one of many other pipelines Let s assume that I m working offline Since commenting out in `local` does not impact `base` is it possible to work on run the other pipelines in my project without commenting things out in `base` and without having to create a local db Thanks in advance for your advice suggestions Regards M P S of course it is not out of lazyness that I m reluctant to comment in `base` sweat smile It s just that messing with `base` to be able to work locally feels off

Nok Lam Chan · Answer

I see what you mean but would your pipeline still works if you remove this datasets

Marc Gris · Answer

Yes 3 modular pipelines out 4 do not rely on this dataset

Nok Lam Chan · Answer

`local` are mean to override in this case when you commented it out it simply means nothing to override

Nok Lam Chan · Answer

Does it affect you whether or not this dataset is inside your catalog If you are not using it

Marc Gris · Answer

I m not sure to understand your question I m personally not yet concerned with the pipeline that depends on that depends I work on the pipelines downstream from it But my colleagues are hence its presence in `base` slightly smiling face Don t you think that this edgy case might be pointing to the utility of having a mechanism that would allow from `local` to ignore things in `base` i e something like `conf local catalog yml` ```dataset type to ignore ``` What do you think slightly smiling face Would that deserve a feature request

Nok Lam Chan · Answer

gt and without having to create a local db I guess I am not clear about this yet why do you need to create a local db From my understanding this dataset doesn t concern your pipeline so whether or not it exists or not shouldn t affect your pipeline

Marc Gris · Answer

= gt If I m working offline and do not mess around with base then when I run `kedro run p my pipeline without db` would raise an error since the creation of the catalog is modular pipeline agnostic

Marc Gris · Answer

Even if I actually do not need the dataset for the pipeline I m working on kedro will still try to query the db

Nok Lam Chan · Answer

I think the more important question here is the connection is created when the Dataset is created could we delay it until it query If you can open an issue about this on GH it would be great To your point that using `local` to remove certain entries I understand the idea of not touching `base` and use local as the only source of change But if the above problem is fixed I don t see a strong need for this If you think this is important feel free to open an issue for this too but I suggest do them separately

Marc Gris · Answer

Thanks for your comments Good point regarding the lazy creation of datasets +1 skin tone 3 Will create a feature request on GH Thx again M

Nok Lam Chan · Answer

For some reason I thought there are changes related to the connection but I cannot find it Cc < Iñigo Hidalgo> For the Ibis dataset how is the connection handled

Nok Lam Chan · Answer

I found a old PR actually change it to singleton connection <https github com kedro org kedro pull 1163>

Nok Lam Chan · Answer

Feel free to create the issue I will link the issues properly and discuss with team

Marc Gris · Answer

+1 skin tone 3 slightly smiling face

Iñigo Hidalgo · Answer

For the Ibis dataset the connection is created on catalog initialization when the first instance of the dataset is initialized Subsequent instances will use the same connection That s only my current implementation I never evaluated the time it takes to generate the connection but I do think that creating a new connection on load would be kinda heavy

Iñigo Hidalgo · Answer

I haven t followed the whole discussion but could you create a dummy dataset type to override the type locally which just doesn t do anything

Iñigo Hidalgo · Answer

Obviously it would break if you try to run a pipeline with it

Iñigo Hidalgo · Answer

But if you just need a placeholder to run some other part of your code it should work

Iñigo Hidalgo · Answer

For Ibis I guess I could also delay the connection creation to the first time an instance loads and then still reuse that connection thinking face Seems less clean to me for some reason though and I wonder what the true value would be

Marc Gris · Answer

Hi < Iñigo Hidalgo> Thx for your messages and suggestions +1 skin tone 3 slightly smiling face One of my use case was when working offline and not having a local db to query My reasoning lazyness sweat smile is that it would be really nice amp neat if only `pipeline c` depends on a sql datasets that `kedro run p pipeline a pipeline b` would work without any problem and without having to muddle with anything

Nok Lam Chan · Answer

This is a reasonable ask Nudging for a github issue so it get discussed by the team

Marc Gris · Answer

Yes Will take time to do this today +1 skin tone 3