Hi, I want to introduce datatests into my Kedro pipeline. There are two types of datatest gerneric and custom. Generic datatests cover f.e. constraints like not_null,unique,between, is_foreign_key. Custom datatests are like are_there_no_overlapping_timestlots. I covered the generic tests by using the schema of pandera and parsing the yaml. eg.

type:package.datasets.snowpark_dataset.SnowparkTableDataset
    schema: SERVING
    database: MY_FAVORIT_DB
    name: MY_FAVORIT_TABLE
    columns:
    - name: CUSTOMER_ID
        type: int
        data_tests:
            - name: not_null
            - name: unique

Which generates the sql

select count(CUSTOMER_ID) from MY_FAVORIT_DB.SERVING.MY_FAVORIT_TABLE where CUSTOMER_ID notnull;

I know the frameworks like great expectation and pander and I would like to implement the solution similar to sql mesh and dot according the Write-Audit-Publish pattern but I struggle with the scope. I see different solutions: 1. I treat a custom datatest as a node and there is a custom dataset like datatestresult. I run the datates nodes after the business transforamtion node. The diffilcuty is that I have already faulty data in the budinesstransformation node and I have a hustle reversing the data if I figure out that a datatest failed. I assume that the scope is totally wrong because one node should represent one transformation and the datset the storage. 2. I introduce datatest's in the save methode. In the save methode I create a release_candidate which represents a temporary table with the applied business logic, apply the tests to the release_candidate and if they succeede I swap the candidate with the target table. I know to little about kedro achitecture to make a good suggestion. Any inputs? I'm having the feeling that I'm already quite bending the framework