https://kedro.org/ logo
#questions
Title
# questions
a

Andreas Hofer

03/13/2024, 8:20 AM
Hey, does anyone know if a dataset factory should also work in an sql query? Following catalog entry:
Copy code
"{date}#some_query":
  type: kedro_datasets.pandas.SQLQueryDataset
  credentials: sql_tb_credentials
  sql: >
    SELECT *
    FROM  table
    WHERE Date(time) >= '{date}'
and the pipeline:
Copy code
def create_pipeline(**kwargs) -> Pipeline:
    return pipeline(
        [
            node(
                func=load_data,
                inputs=f"{get_latest_date()}#some_query",
                outputs="some",
                name="some_query",
            ),
        ]
    )

The `kedro run` seems to work, but the `kedro catalog resolve` throws KeyError: 'filepath'
m

Merel

03/13/2024, 11:12 AM
Hi @Andreas Hofer, could you share the full error stacktrace?
j

Juan Luis

03/13/2024, 11:38 AM
a

Andreas Hofer

03/13/2024, 11:59 AM
For me it's not a partitioned dataset. Thank you for leading me to the issue.
Copy code
Traceback (most recent call last):
  File "/home/adhofer/workspace/projects/kedro/.venv/bin/kedro", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/home/adhofer/workspace/projects/kedro/.venv/lib/python3.11/site-packages/kedro/framework/cli/cli.py", line 197, in main
    cli_collection()
  File "/home/adhofer/workspace/projects/kedro/.venv/lib/python3.11/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/adhofer/workspace/projects/kedro/.venv/lib/python3.11/site-packages/kedro/framework/cli/cli.py", line 125, in main
    super().main(
  File "/home/adhofer/workspace/projects/kedro/.venv/lib/python3.11/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/home/adhofer/workspace/projects/kedro/.venv/lib/python3.11/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/adhofer/workspace/projects/kedro/.venv/lib/python3.11/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/adhofer/workspace/projects/kedro/.venv/lib/python3.11/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/adhofer/workspace/projects/kedro/.venv/lib/python3.11/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/adhofer/workspace/projects/kedro/.venv/lib/python3.11/site-packages/click/decorators.py", line 45, in new_func
    return f(get_current_context().obj, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/adhofer/workspace/projects/kedro/.venv/lib/python3.11/site-packages/kedro/framework/cli/catalog.py", line 264, in resolve_patterns
    str(context.project_path) + "/", ds_config["filepath"]
                                     ~~~~~~~~~^^^^^^^^^^^^
KeyError: 'filepath'
m

Merel

03/13/2024, 12:01 PM
hmm I think it might be that the
SQLQueryDataset
doesn't have the filepath/path key at all.
👍 1
Which Kedro version are you using @Andreas Hofer?
a

Andreas Hofer

03/13/2024, 12:02 PM
It's 0.19.1
m

Merel

03/13/2024, 12:04 PM
Could you try upgrading to the latest version? I think this issue has been resolved in
0.19.3
a

Andreas Hofer

03/13/2024, 12:09 PM
Oh sorry I thought I already had the latest version. I pinned the version by mistake. Now it can no longer find my credentials in 'credentials.yml'. I try to check this.
🤔 1
Ok, strange... I killed my terminal and reinstalled kedro. The credentials are still in place... i have no idea:
Copy code
Traceback (most recent call last):
  File "/home/adhofer/workspace/projects/-kedro/.venv/lib/python3.11/site-packages/kedro/io/data_catalog.py", line 51, in _get_credentials
    return credentials[credentials_name]
           ~~~~~~~~~~~^^^^^^^^^^^^^^^^^^
KeyError: 'sql_tb_credentials'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/adhofer/workspace/projects/-kedro/.venv/bin/kedro", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/home/adhofer/workspace/projects/-kedro/.venv/lib/python3.11/site-packages/kedro/framework/cli/cli.py", line 198, in main
    cli_collection()
  File "/home/adhofer/workspace/projects/-kedro/.venv/lib/python3.11/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/adhofer/workspace/projects/-kedro/.venv/lib/python3.11/site-packages/kedro/framework/cli/cli.py", line 127, in main
    super().main(
  File "/home/adhofer/workspace/projects/-kedro/.venv/lib/python3.11/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/home/adhofer/workspace/projects/-kedro/.venv/lib/python3.11/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/adhofer/workspace/projects/-kedro/.venv/lib/python3.11/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/adhofer/workspace/projects/-kedro/.venv/lib/python3.11/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/adhofer/workspace/projects/-kedro/.venv/lib/python3.11/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/adhofer/workspace/projects/-kedro/.venv/lib/python3.11/site-packages/click/decorators.py", line 45, in new_func
    return f(get_current_context().obj, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/adhofer/workspace/projects/-kedro/.venv/lib/python3.11/site-packages/kedro/framework/cli/catalog.py", line 236, in resolve_patterns
    data_catalog = DataCatalog.from_config(catalog_config)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/adhofer/workspace/projects/-kedro/.venv/lib/python3.11/site-packages/kedro/io/data_catalog.py", line 286, in from_config
    ds_config = _resolve_credentials(  # noqa: PLW2901
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/adhofer/workspace/projects/-kedro/.venv/lib/python3.11/site-packages/kedro/io/data_catalog.py", line 83, in _resolve_credentials
    return {k: _map_value(k, v) for k, v in config.items()}
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/adhofer/workspace/projects/-kedro/.venv/lib/python3.11/site-packages/kedro/io/data_catalog.py", line 83, in <dictcomp>
    return {k: _map_value(k, v) for k, v in config.items()}
               ^^^^^^^^^^^^^^^^
  File "/home/adhofer/workspace/projects/-kedro/.venv/lib/python3.11/site-packages/kedro/io/data_catalog.py", line 78, in _map_value
    return _get_credentials(value, credentials)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/adhofer/workspace/projects/-kedro/.venv/lib/python3.11/site-packages/kedro/io/data_catalog.py", line 53, in _get_credentials
    raise KeyError(
KeyError: "Unable to find credentials 'sql_tb_credentials': check your data catalog and credentials configuration. See <https://kedro.readthedocs.io/en/stable/kedro.io.DataCatalog.html> for an example."
m

Merel

03/13/2024, 5:03 PM
I'll try and reproduce this. Not sure on top of my head what's going on here.
Just to double check this only happens when doing
kedro catalog resolve
or also with
kedro run
?
FYI: I've found the issue and will open a PR to fix it.
🥳 1
a

Andreas Hofer

03/14/2024, 10:44 AM
Thank you very much 🙂
3 Views