Hi Guys, How can i change the default dataset from MemoryDataset to a kedro-mlflow dataset? or provide some rule to map such datasets without having to maintain two sources of datasets? Currently, i am using a modular pipeline to create namespaces for different experiments i want to run in a single session. For example, i am testing the accuracy of several prediction methods in two ways: 1. a random train-test split 2. a date-based train-test split, this is to check performance on the latest data and detect drift. Now i can very easily create multiple pipelines by remapping some inputs and outputs using the modular pipeline concept, however, i want to cache some of the training steps since these are very big (and costly) multi-modal models. I use kedro-mlflow to log the artefact and metrics to mlflow and s3, however this requires such datasets to be described in the catalog.yml. I used the templatedconfig and the Jinja2 syntax to create a list of datasets, however now i have to maintain these lists in two different places, which is begging for bugs. My prefered solution would be to have a single parameters file, where i specify all the parameters for which i want to run them in a grid (ParameterGrid). This could look like:

parameters.yml

ParameterGrid:
  name_of_parameter:
    version_1:
      - value1
      - value2
    version_2:
      - value1
etc.

and now i could run through these options with the namespace. However, now i need to have dataset entries in the catalog.yml which match these

version_1

and

version_2

names. Since i dont want these to be stored in memory and than destroyed. Instead i want to use the kedro_mlflow datasets. so for example for the parquet files i would use something like:

X_test_{{ split_crit }}:
  type: kedro_mlflow.io.artifacts.MlflowArtifactDataSet
  data_set:
      type: pandas.ParquetDataSet
      filepath: <s3://sagemaker-vertex/data/05_model_input/X_test_{{> split_crit }}.parquet

and for the metrics:

my_model_metrics_{{ split_crit }}:
    type: kedro_mlflow.io.metrics.MlflowMetricDataSet
    key: accuracy

and for the models

multi_modal_model:
  type: kedro_mlflow.io.models.MlflowModelLoggerDataSet
  flavor: mlflow.pyfunc
  pyfunc_workflow: python_model
  save_args:
    conda_env:
        python: "3.9.10"
        dependencies:
            - "mlflow==1.27.0"

However, in kedro these output datasets cannot be shared (even though in mlflow this would be fine)

so there are two questions here: 1. How to change the default dataset 2. How to do a parameter sweep and multiple runs

yeah, i though about doing a run.yml with the different cli commands