Hi another question is there a way to directly store the con Kedro #questions

Hi another question, is there a way to directly st...

Eluard Camota

10/28/2023, 3:05 AM

Hi another question, is there a way to directly store the connection string in the catalog.yml This is the current config:

catalog.yaml

Copy code

shuttles_table_dataset:
  type: pandas.SQLTableDataset
  credentials: db_credentials
  table_name: shuttles

credentials.yml

Copy code

db_credentials:
  con: <postgresql://scott:tiger@localhost/test>

This is what I need:

catalog.yml

Copy code

shuttles_table_dataset:
  type: pandas.SQLTableDataset
  con: <postgresql://scott:tiger@localhost/test>
  table_name: shuttles

Eluard Camota

10/28/2023, 3:09 AM

Solved: It should look like this:

catalog.yml

Copy code

shuttles_table_dataset:
  type: pandas.SQLTableDataset
  credentials:
      con: <postgresql://scott:tiger@localhost/test>
  table_name: shuttles

👍 1

Nok Lam Chan

10/28/2023, 11:22 PM

Did the

credentials.yml

not work? note that now you are putting your key/secret directly and it may get comitted.

Eluard Camota

10/29/2023, 1:30 AM

it's works but when I build it as a docker image and run it on prefect it cannot locate the credentials.yml

Eluard Camota

10/29/2023, 1:31 AM

Also this is what I'm facing right now while running the docker image on prefect

Copy code

2023-10-29 00:22:35 16:22:35.563 | INFO    | Flow run 'exotic-chimpanzee' - Downloading flow code from storage at '/home/kedro_docker'
2023-10-29 00:22:35 16:22:35.566 | ERROR   | Flow run 'exotic-chimpanzee' - Flow could not be retrieved from deployment.
2023-10-29 00:22:35 Traceback (most recent call last):
2023-10-29 00:22:35   File "/usr/local/lib/python3.8/site-packages/prefect/engine.py", line 405, in retrieve_flow_then_begin_flow_run
2023-10-29 00:22:35     flow = await load_flow_from_flow_run(flow_run, client=client)
2023-10-29 00:22:35   File "/usr/local/lib/python3.8/site-packages/prefect/client/utilities.py", line 51, in with_injected_client
2023-10-29 00:22:35     return await fn(*args, **kwargs)
2023-10-29 00:22:35   File "/usr/local/lib/python3.8/site-packages/prefect/deployments/deployments.py", line 230, in load_flow_from_flow_run
2023-10-29 00:22:35     await storage_block.get_directory(from_path=from_path, local_path=".")
2023-10-29 00:22:35   File "/usr/local/lib/python3.8/site-packages/prefect/filesystems.py", line 162, in get_directory
2023-10-29 00:22:35     copytree(from_path, local_path, dirs_exist_ok=True, ignore=ignore_func)
2023-10-29 00:22:35   File "/usr/local/lib/python3.8/shutil.py", line 555, in copytree
2023-10-29 00:22:35     with os.scandir(src) as itr:
2023-10-29 00:22:35 FileNotFoundError: [Errno 2] No such file or directory: '/home/kedro_docker'

Nok Lam Chan

10/29/2023, 2:02 PM

It is because credentials are not packaged into the container, it will be more suitable to use environment variables for productions

👍🏼 1

6 Views

Open in Slack

Previous Next