I like the idea of expanding 08 into more granular post-model layers

I also find myself not really needing the level of granularity in the 'data engineering' layers

Aha, I didn't come upon this before https://towardsdatascience.com/the-importance-of-layered-thinking-in-data-engineering-a09f685edc71

So I guess my question wrt to that stated philosophy would be: does this kedro layer thinking change at all if one manages data transformations completely outside of Kedro? (for example, using dbt or a feature store)

Like maybe use

features

to stage data pulled from a feature store

and

model_input

to stage data constructed from features and ready for fitting

But also, since the granularity of the

01_

-

04_

kedro layers were designed from a primarily DE perspective, I wonder what additional granularity of layers would logically arise from rethinking from a DS perspective? For example (and there will be some overlap here): • benchmark datasetes (from papers, etc) • data before train/test split • training data • test data • "inference" data that has been transformed/scored/augmented by model inferences (

07_

) I could also see granular expansion of

08_reporting

: • model_diagnostics data • model_validation data • model_comparison data (experiment tracking)

After some thought, here's a possible "DS perspective-driven" version of the kedro layers:

├── data
│   ├── 01_source         <-- Datasets cached from external sources
│   ├── 02_staging        <-- Transformed datasets
│   ├── 03_benchmark      <-- Benchmark datasets & metrics
│   ├── 04_training       <-- Direct inputs to model fitting
│   ├── 05_testing        <-- Direct inputs to model inference
│   ├── 06_models         <-- Serialised models
│   ├── 07_inference      <-- Data generated from models
│   ├── 08_diagnostic     <-- Model diagnostic data
│   ├── 09_evaluation     <-- Model evaluation data