Hi all I am running multiple pipelines using thread runner H

Question

Hi all I am running multiple pipelines using thread runner How to increase concurrent connection to db while declaring credentials for MySQL in credentials yml To do it in python I can do it using passing extra argument in create engine Or how to close connection after loading the data That way I won t face the error

Nok Lam Chan · Answer

Great question I don t have an immediate answer Can you share how would you do it in pure Python

Chandan Malla · Answer

Nok Lam Chan · Answer

You can override the `create connection` method by subclass the `SQLDataSet`

Nok Lam Chan · Answer

Do you want to open a PR to add this I think it s reasonable to support extra args to connection <https github com kedro org kedro plugins blob 7b3ac6c7b88c44143a26fe556475cebb9012ac08 kedro datasets kedro datasets pandas sql dataset py L472 L486|https github com kedro org kedro plugins blob 7b3ac6c7b88c44143a26fe556475cebb9012ac08 kedro datasets kedro datasets pandas sql dataset py L4 86> Essentially the args need to be passed here

Chandan Malla · Answer

I will but I have never worked on an open source project If you can guide me on how do I verify if the changes are done are working or not Or any article which will guide me

Nok Lam Chan · Answer

This will be a good first PR

Nok Lam Chan · Answer

there are some guides in and

Chandan Malla · Answer

Sure I will on the weekend in the meantime is there any possibility of a workaround in this

Nok Lam Chan · Answer

gt You can override the `create connection` method by subclass the `SQLDataSet` This will be the workaround

Nok Lam Chan · Answer

So essentially ```class CustomSQLDataSet SQLDataSet def create connection self lt extra args gt ```

Chandan Malla · Answer

In which file this change will go

Nok Lam Chan · Answer

This is a workaround and you need to create a CustomDataset

Nok Lam Chan · Answer

I am in a meeting now but you can search it in the doc with custom dataset