Sergey S
04/17/2024, 5:15 PM...
├── 2024-04-17-test-run2
│ ├── 01_raw <-- Raw immutable data
│ ├── 02_intermediate <-- Typed data
│ ├── 03_primary <-- Domain model data
│ ├── 04_feature <-- Model features
│ ├── 05_model_input <-- Often called 'master tables'
│ ├── 06_models <-- Serialised models
│ ├── 07_model_output <-- Data generated by model runs
│ ├── 08_reporting <-- Ad hoc descriptive cuts
├── 2023-03-01-test-run1
│ ├── 01_raw <-- Raw immutable data
│ ├── 02_intermediate <-- Typed data
│ ├── 03_primary <-- Domain model data
│ ├── 04_feature <-- Model features
│ ├── 05_model_input <-- Often called 'master tables'
│ ├── 06_models <-- Serialised models
│ ├── 07_model_output <-- Data generated by model runs
│ ├── 08_reporting <-- Ad hoc descriptive cuts
...
Artur Dobrogowski
04/17/2024, 8:00 PMin catalog:
entry:
filepath: data/run-${timestamp:${globals:run_timestamp}}/02_intermediate/...
in globals:
run_timestamp: ""
in settings:
@cached
def get_timestamp(stamp):
if stamp:
return stamp
return datetime.now().isoformat()
CONFIG_LOADER_ARGS = {
"custom_resolvers": {
"timestamp": get_timestamp,
}
}
Artur Dobrogowski
04/17/2024, 8:04 PM