https://kedro.org/ logo
Powered by
#questions
Title
# questions
e

Eluard Camota

10/28/2023, 3:05 AM
Hi another question, is there a way to directly store the connection string in the catalog.yml This is the current config: 
catalog.yaml
Copy code
shuttles_table_dataset:
  type: pandas.SQLTableDataset
  credentials: db_credentials
  table_name: shuttles
credentials.yml
Copy code
db_credentials:
  con: <postgresql://scott:tiger@localhost/test>
This is what I need: 
catalog.yml
Copy code
shuttles_table_dataset:
  type: pandas.SQLTableDataset
  con: <postgresql://scott:tiger@localhost/test>
  table_name: shuttles
Solved: It should look like this: 
catalog.yml
Copy code
shuttles_table_dataset:
  type: pandas.SQLTableDataset
  credentials:
      con: <postgresql://scott:tiger@localhost/test>
  table_name: shuttles
👍 1
n

Nok Lam Chan

10/28/2023, 11:22 PM
Did the 
credentials.yml
not work? note that now you are putting your key/secret directly and it may get comitted.
e

Eluard Camota

10/29/2023, 1:30 AM
it's works but when I build it as a docker image and run it on prefect it cannot locate the credentials.yml
Also this is what I'm facing right now while running the docker image on prefect
Copy code
2023-10-29 00:22:35 16:22:35.563 | INFO    | Flow run 'exotic-chimpanzee' - Downloading flow code from storage at '/home/kedro_docker'
2023-10-29 00:22:35 16:22:35.566 | ERROR   | Flow run 'exotic-chimpanzee' - Flow could not be retrieved from deployment.
2023-10-29 00:22:35 Traceback (most recent call last):
2023-10-29 00:22:35   File "/usr/local/lib/python3.8/site-packages/prefect/engine.py", line 405, in retrieve_flow_then_begin_flow_run
2023-10-29 00:22:35     flow = await load_flow_from_flow_run(flow_run, client=client)
2023-10-29 00:22:35   File "/usr/local/lib/python3.8/site-packages/prefect/client/utilities.py", line 51, in with_injected_client
2023-10-29 00:22:35     return await fn(*args, **kwargs)
2023-10-29 00:22:35   File "/usr/local/lib/python3.8/site-packages/prefect/deployments/deployments.py", line 230, in load_flow_from_flow_run
2023-10-29 00:22:35     await storage_block.get_directory(from_path=from_path, local_path=".")
2023-10-29 00:22:35   File "/usr/local/lib/python3.8/site-packages/prefect/filesystems.py", line 162, in get_directory
2023-10-29 00:22:35     copytree(from_path, local_path, dirs_exist_ok=True, ignore=ignore_func)
2023-10-29 00:22:35   File "/usr/local/lib/python3.8/shutil.py", line 555, in copytree
2023-10-29 00:22:35     with os.scandir(src) as itr:
2023-10-29 00:22:35 FileNotFoundError: [Errno 2] No such file or directory: '/home/kedro_docker'
n

Nok Lam Chan

10/29/2023, 2:02 PM
It is because credentials are not packaged into the container, it will be more suitable to use environment variables for productions
👍🏼 1
4 Views
Powered by