https://kedro.org/ logo
#questions
Title
# questions
c

Chandan Malla

09/28/2023, 6:56 AM
Hi all, I am running multiple pipelines using thread runner How to increase concurrent connection to db while declaring credentials for MySQL in credentials.yml? To do it in python I can do it using passing extra argument in create_engine Or how to close connection, after loading the data ? That way I won't face the error .
n

Nok Lam Chan

09/28/2023, 8:27 AM
Great question, I don't have an immediate answer. Can you share how would you do it in pure Python?
n

Nok Lam Chan

09/28/2023, 10:56 AM
You can override the
create_connection
method by subclass the
SQLDataSet
.
Do you want to open a PR to add this? I think it’s reasonable to support extra args to connection. https://github.com/kedro-org/kedro-plugins/blob/7b3ac6c7b88c44143a26fe556475cebb9012ac08/kedro-datasets/kedro_datasets/pandas/sql_dataset.py#L4[…]86 Essentially the args need to be passed here
c

Chandan Malla

09/28/2023, 3:51 PM
I will, but I have never worked on an open source project. If you can guide me on how do I verify if the changes are done are working or not? Or any article which will guide me.
n

Nok Lam Chan

09/28/2023, 4:05 PM
This will be a good first PR :)
c

Chandan Malla

09/29/2023, 6:29 AM
Sure I will on the weekend, in the meantime is there any possibility of a workaround in this?
n

Nok Lam Chan

09/29/2023, 10:13 AM
You can override the
create_connection
method by subclass the
SQLDataSet
.
This will be the workaround
So essentially
Copy code
class CustomSQLDataSet(SQLDataSet):
     def create_connection(self, <extra_args>:
         ...
c

Chandan Malla

09/29/2023, 10:26 AM
In which file this change will go?
n

Nok Lam Chan

09/29/2023, 10:31 AM
This is a workaround and you need to create a CustomDataset
I am in a meeting now but you can search it in the doc with “custom dataset”