https://kedro.org/ logo
#questions
Title
# questions
e

Elias

11/25/2022, 10:13 AM
Copy code
catalog.yml:

inspection_output:
  type: pandas.SQLQueryDataSet
  credentials: postgresql_credentials
  table_name: shuttles
  layer: model_output
  save_args:
    index: true
d

datajoely

11/25/2022, 10:16 AM
You’re using SQLQueryDataSet not SQLTableDataSet
e

Elias

11/25/2022, 10:17 AM
ah, thx!
d

datajoely

11/25/2022, 10:17 AM
no worries that’s a subtle one!
e

Elias

11/25/2022, 10:20 AM
can you also version these datasets? or is it best practice to do so by a custom created column?
d

datajoely

11/25/2022, 10:24 AM
We don’t support this on the SQL side unfortunately. You could use append only mode and add a timestamp column although this may get very tall over time.
You may also consider a simple subclass that does this automatically
e

Elias

11/25/2022, 2:11 PM
okay nice, is there any way to attain the timestamp kedro uses to version datasets during runtime? I want to give the same timestamp to multiple tables, which are written in different nodess.
d

datajoely

11/28/2022, 9:56 AM
it’s created in the session object, if you really want to you could create a hook that sets this as an environment variable and then you can reuse that. In the future this will be easier to configure.
4 Views