Aleksander Jaworski
07/24/2023, 11:22 AMhello_world.py
will take several seconds to run when the configuration is large enough, as first you will see all the logs and all the setup will be done for the data catalog etc, none of which would actually end up being used in a hello_world.py
2. When setting up the project for someone, it is impossible to provide a credentials file with just the required credentials. In kedro all of them need to be filled right now as it is all validated at once. In a sort of lazy version, only the dependencies that follow from the pipeline would need to be evaluated.
Are there any solutions or modifications I could use to improve my approaches here? Thanks in advance! :)Nok Lam Chan
07/24/2023, 11:34 AMIt feels a bit cumbersome that even a simpleI suspect this is related. How big is your catalog (number of entries?) Can you try removing any SQLDataset and see if it is speed up? Most dataset are lazily initiated, it should not have any impact on loading up the catalog. https://github.com/kedro-org/kedro/issues/2829will take several seconds to run when the configuration is large enough, as first you will see all the logs and all the setup will be done for the data catalog etc, none of which would actually end up being used in ahello_world.py
hello_world.py
When setting up the project for someone, it is impossible to provide a credentials file with just the required credentials. In kedro all of them need to be filled right now as it is all validated at once. In a sort of lazy version, only the dependencies that follow from the pipeline would need to be evaluated.Is this also related to SQLDataset which requires db connection (and credentials)?
Aleksander Jaworski
07/24/2023, 11:35 AMparameters.yml
and catalog .yml
and ive been using the TemplatedConfigLoader
to interpolate elements of configuration from env varsNok Lam Chan
07/24/2023, 11:46 AMAleksander Jaworski
07/24/2023, 11:57 AM~21 secs
for kedro catalog list
~6 secs
after commenting out the SQL and GBQ datasetsNok Lam Chan
07/24/2023, 12:20 PMimport <your_module>
to see that how long does it takes, this doesn’t load up any kedro thing and can help you isolate how much overhead is added by Kedro.
• Connections - Do you have other connections setup other than SQL?
Normally reading config should be very fast, 200 -800 LOC shouldn’t create any significant overhead.Aleksander Jaworski
07/25/2023, 8:56 AM