Hi all,
I am running multiple pipelines using thre...
# questionsc
Chandan Malla
09/28/2023, 6:56 AMHi all,
I am running multiple pipelines using thread runner
How to increase concurrent connection to db while declaring credentials for MySQL in credentials.yml?
To do it in python I can do it using passing extra argument in create_engine
Or how to close connection, after loading the data ? That way I won't face the error .
n
Nok Lam Chan
09/28/2023, 8:27 AMGreat question, I don't have an immediate answer. Can you share how would you do it in pure Python?
c
Chandan Malla
09/28/2023, 10:09 AMn
Nok Lam Chan
09/28/2023, 10:56 AMYou can override the
create_connectionSQLDataSetNok Lam Chan
09/28/2023, 10:57 AMDo you want to open a PR to add this? I think it’s reasonable to support extra args to connection.
https://github.com/kedro-org/kedro-plugins/blob/7b3ac6c7b88c44143a26fe556475cebb9012ac08/kedro-datasets/kedro_datasets/pandas/sql_dataset.py#L4[…]86
Essentially the args need to be passed here
c
Chandan Malla
09/28/2023, 3:51 PMI will, but I have never worked on an open source project. If you can guide me on how do I verify if the changes are done are working or not? Or any article which will guide me.
n
Nok Lam Chan
09/28/2023, 4:05 PMThis will be a good first PR :)
Nok Lam Chan
09/28/2023, 4:05 PMc
Chandan Malla
09/29/2023, 6:29 AMSure I will on the weekend, in the meantime is there any possibility of a workaround in this?
n
Nok Lam Chan
09/29/2023, 10:13 AMYou can override theThis will be the workaroundmethod by subclass thecreate_connection.SQLDataSet
Nok Lam Chan
09/29/2023, 10:14 AMSo essentially
Copy code
class CustomSQLDataSet(SQLDataSet):
def create_connection(self, <extra_args>:
...c
Chandan Malla
09/29/2023, 10:26 AMIn which file this change will go?
n
Nok Lam Chan
09/29/2023, 10:31 AMThis is a workaround and you need to create a CustomDataset
Nok Lam Chan
09/29/2023, 10:31 AMI am in a meeting now but you can search it in the doc with “custom dataset”