https://kedro.org/ logo
Join Slack
Powered by
Hey, does anyone know if a dataset factory should...
# questions
a
Hey, does anyone know if a dataset factory should also work in an sql query? Following catalog entry:
Copy code
"{date}#some_query":
  type: kedro_datasets.pandas.SQLQueryDataset
  credentials: sql_tb_credentials
  sql: >
    SELECT *
    FROM  table
    WHERE Date(time) >= '{date}'
and the pipeline:
Copy code
def create_pipeline(**kwargs) -> Pipeline:
    return pipeline(
        [
            node(
                func=load_data,
                inputs=f"{get_latest_date()}#some_query",
                outputs="some",
                name="some_query",
            ),
        ]
    )

The `kedro run` seems to work, but the `kedro catalog resolve` throws KeyError: 'filepath'
m
Hi @Andreas Hofer, could you share the full error stacktrace?
j
a
For me it's not a partitioned dataset. Thank you for leading me to the issue.
Copy code
Traceback (most recent call last):
  File "/home/adhofer/workspace/projects/kedro/.venv/bin/kedro", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/home/adhofer/workspace/projects/kedro/.venv/lib/python3.11/site-packages/kedro/framework/cli/cli.py", line 197, in main
    cli_collection()
  File "/home/adhofer/workspace/projects/kedro/.venv/lib/python3.11/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/adhofer/workspace/projects/kedro/.venv/lib/python3.11/site-packages/kedro/framework/cli/cli.py", line 125, in main
    super().main(
  File "/home/adhofer/workspace/projects/kedro/.venv/lib/python3.11/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/home/adhofer/workspace/projects/kedro/.venv/lib/python3.11/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/adhofer/workspace/projects/kedro/.venv/lib/python3.11/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/adhofer/workspace/projects/kedro/.venv/lib/python3.11/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/adhofer/workspace/projects/kedro/.venv/lib/python3.11/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/adhofer/workspace/projects/kedro/.venv/lib/python3.11/site-packages/click/decorators.py", line 45, in new_func
    return f(get_current_context().obj, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/adhofer/workspace/projects/kedro/.venv/lib/python3.11/site-packages/kedro/framework/cli/catalog.py", line 264, in resolve_patterns
    str(context.project_path) + "/", ds_config["filepath"]
                                     ~~~~~~~~~^^^^^^^^^^^^
KeyError: 'filepath'
m
hmm I think it might be that the 
SQLQueryDataset
doesn't have the filepath/path key at all.
👍 1
Which Kedro version are you using @Andreas Hofer?
a
It's 0.19.1
m
Could you try upgrading to the latest version? I think this issue has been resolved in 
0.19.3
a
Oh sorry I thought I already had the latest version. I pinned the version by mistake. Now it can no longer find my credentials in 'credentials.yml'. I try to check this.
🤔 1
Ok, strange... I killed my terminal and reinstalled kedro. The credentials are still in place... i have no idea:
Copy code
Traceback (most recent call last):
  File "/home/adhofer/workspace/projects/-kedro/.venv/lib/python3.11/site-packages/kedro/io/data_catalog.py", line 51, in _get_credentials
    return credentials[credentials_name]
           ~~~~~~~~~~~^^^^^^^^^^^^^^^^^^
KeyError: 'sql_tb_credentials'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/adhofer/workspace/projects/-kedro/.venv/bin/kedro", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/home/adhofer/workspace/projects/-kedro/.venv/lib/python3.11/site-packages/kedro/framework/cli/cli.py", line 198, in main
    cli_collection()
  File "/home/adhofer/workspace/projects/-kedro/.venv/lib/python3.11/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/adhofer/workspace/projects/-kedro/.venv/lib/python3.11/site-packages/kedro/framework/cli/cli.py", line 127, in main
    super().main(
  File "/home/adhofer/workspace/projects/-kedro/.venv/lib/python3.11/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/home/adhofer/workspace/projects/-kedro/.venv/lib/python3.11/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/adhofer/workspace/projects/-kedro/.venv/lib/python3.11/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/adhofer/workspace/projects/-kedro/.venv/lib/python3.11/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/adhofer/workspace/projects/-kedro/.venv/lib/python3.11/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/adhofer/workspace/projects/-kedro/.venv/lib/python3.11/site-packages/click/decorators.py", line 45, in new_func
    return f(get_current_context().obj, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/adhofer/workspace/projects/-kedro/.venv/lib/python3.11/site-packages/kedro/framework/cli/catalog.py", line 236, in resolve_patterns
    data_catalog = DataCatalog.from_config(catalog_config)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/adhofer/workspace/projects/-kedro/.venv/lib/python3.11/site-packages/kedro/io/data_catalog.py", line 286, in from_config
    ds_config = _resolve_credentials(  # noqa: PLW2901
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/adhofer/workspace/projects/-kedro/.venv/lib/python3.11/site-packages/kedro/io/data_catalog.py", line 83, in _resolve_credentials
    return {k: _map_value(k, v) for k, v in config.items()}
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/adhofer/workspace/projects/-kedro/.venv/lib/python3.11/site-packages/kedro/io/data_catalog.py", line 83, in <dictcomp>
    return {k: _map_value(k, v) for k, v in config.items()}
               ^^^^^^^^^^^^^^^^
  File "/home/adhofer/workspace/projects/-kedro/.venv/lib/python3.11/site-packages/kedro/io/data_catalog.py", line 78, in _map_value
    return _get_credentials(value, credentials)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/adhofer/workspace/projects/-kedro/.venv/lib/python3.11/site-packages/kedro/io/data_catalog.py", line 53, in _get_credentials
    raise KeyError(
KeyError: "Unable to find credentials 'sql_tb_credentials': check your data catalog and credentials configuration. See <https://kedro.readthedocs.io/en/stable/kedro.io.DataCatalog.html> for an example."
m
I'll try and reproduce this. Not sure on top of my head what's going on here.
Just to double check this only happens when doing 
kedro catalog resolve
or also with 
kedro run
?
FYI: I've found the issue and will open a PR to fix it.
🥳 1
a
Thank you very much 🙂
4 Views
Open in Slack
PreviousNext
Powered by