Just had a little idea for a potential new plugin :

kedro-linter

... NOT for linting Python code, but rather for linting the kedro project's config files and general file tree for adherence to best practices. For example, it could report the number of implicit in-memory datasets not present in

caatalog.yml

, report presence of non-notebook files in

notebooks

or other file types out of place (independent of

.gitignore

), report "feature coverage" of various kedro features (parameters, named datasets, etc) across pipelines, etc etc etc.

What is your experience with combining Kedro and BentoML? I’ve used Kedro-MLFlow quite a bit, but find its built-in model serving quite limiting, especially when it comes to the performance and stability of batch inference for transformer models on the GPU. However, I do really like the concept of packaging basically the evaluation part of my pipeline as a KedroPipeline and serving that. To establish the same in BentoML, do you package/pickle every node and store it (in bento) and then create a runnable out of that, or do you have you found a more sophisticated approach like serving the KedroPipeline, and maybe extracting only the framework specific parts to leverage bento’s integrations? For integration with BentoML, im currently using the following:

import bentoml
from pathlib import Path


def _find_kedro_project(current_dir):  # pragma: no cover
    from kedro.framework.startup import _is_project

    while current_dir != current_dir.parent:
        if _is_project(current_dir):
            return current_dir
        current_dir = current_dir.parent

    return None


def retrieve_kedro_context(env="local"):
    from kedro.framework.session import KedroSession
    from kedro.framework.startup import bootstrap_project

    project_path = _find_kedro_project(Path.cwd())
    metadata = bootstrap_project(project_path)

    with KedroSession.create(
        package_name=metadata.package_name,
        project_path=metadata.project_path,
        env=env,
    ) as kedro_session:
        return kedro_session.load_context()


def download_model(name: str) -> bentoml.Model:
    try:
        return bentoml.transformers.get(name)

    except bentoml.exceptions.NotFound:
        catalog = retrieve_kedro_context().catalog
        pipeline = catalog.load(name)
        return bentoml.transformers.save_model(name, pipeline)


def get_runner(name: str, init_local: bool = False):
    runner = download_model(name).to_runner()
    if init_local:
        runner.init_local(quiet=True)
    return runner

This allows one to use any arbitrary catalog entry (so you store your model as a pickle on s3, or in mlflow), however integration with KedroPipeline seems very complicated as bento needs to be aware of the model framework. In additation, CI/CD now needs access the kedro context, while id prefer to simply link the mlflow storage to bentoml and maybe use some adapters for pre and post-processing. Additionaly, Bento uses its own storage location, which for as far as i know i can;t move to the cloud (which is quite catastrophic when using Llama=70b, since it will instantly fill up your local storage), have you found any ways around this? Do you make storing bentos part of your kedro pipelines? in the above code, i look in the bento storage, and if not found in the kedro catalog. But this obvisously only works when you actively manage this (otherwise you’d be pulling old models). any thoughts or preferences? Also, do you use pip, conda or poetry? im looking to use different dependancy groups to separate model deps from dev/training deps. also because there have been quite a few breaking changes lately when upgrading packages. Is there any special tricks you employ wrt staggered updating of deps combined with tests? Do you link the poetry deps with mlflow, or do you use bentos inver_packages? Also, what are your opinions when it comes to deploying the packaged models to k8s? Do you simply deploy docker containers direcly, or use something like Seldon or Kserve? Or even Bento’s Yatai? Im curious!

about @Hugo Evers question on inference: upon reading Hopsworks FTI architecture these days I was thinking that, if the output of a Kedro training pipeline is a model that can be serialized (ONNX or your format of choice), and also given that latency might be sensitive in some cases, is it worth doing the inference from Kedro? this might seem like I don't want you to use Kedro for inference pipelines while of course I want you to use Kedro for everything, but I'm seeking to understand how others do it.

about "Also, do you use pip, conda or poetry?": mostly pip. I only use conda micromamba if (1) I don't want to compile my dependencies from source, (2) I want to avoid PyTorch or tensorflow 800 megabyte wheels, (3) or I want to use fine-grained feature selection beyond what pip supports, such as GPU stuff. workflow tools (like Poetry, Hatch, PDM, Rye) are for development and not for production, IMHO

Hi Team! I’m trying to upload via Kedro MLFLow some models as MLFlow artifacts to a pod, and I’m struggling to set the

PipelineML

pipeline. Does anyone have experience with this or suggestions on how to implement it? Thanks in advance! Slack conversation

Re Kedro SageMaker plugin: In my daily workflow (Apple silicon), I use

docker buildx build --platform=linux/amd64

to build the images I eventually push to ECR. is there a way to specify what platform target

kedro sagemaker

should use?

We’ve recently released version 0.6.0 of

kedro-azureml

with GREAT help from external contributors 🎉 Keep it going guys! @Florian d, @Tomas Van Pottelbergh https://www.linkedin.com/posts/getindata_getindata-kedro-azure-activity-7117156892325388289-C-bt?utm_source=share&utm_medium=member_desktop

Hi team! Which visualization libraries allow to input parameters and run a kedro pipeline from the very interface? I was able to do it using

streamlit

, but I could not do so with

dash

or

vizro

... Do you know how to do that or if there are any other libraries to do so? Thank you!

hey everyone, I've been working some time on using kedro with Argo and since the current documentation is outdated in the webpage, I have implemented kind of a script using Hera Workflows to convert kedro pipelines to Argo Workflows DAG. The following code converts your kedro nodes to Kubernetes Pods and creates Argo DAGs based on your pipeline dependencies. I hope this is useful to some people. We can also try to put this into the page if the developers find it useful.

import logging
from pathlib import Path

from hera.auth import ArgoCLITokenGenerator
from hera.shared import global_config
from hera.workflows import (
    DAG,
    Container,
    Env,
    Parameter,
    RetryPolicy,
    RetryStrategy,
    Workflow,
)
from kedro.framework.project import pipelines
from kedro.framework.startup import bootstrap_project
from kedro.pipeline import Pipeline


# More info on hera docs
global_config.host = "<https://argo-workflows.io>"    # Put YOUR OWN INSTANCE PROFILE
global_config.token = ArgoCLITokenGenerator
global_config.namespace = "NAMESPACE OF THE ARGO WORKFLOWS IN K8s"
global_config.service_account_name = "SERVICE ACCOUNT OF THE ARGO WORKFLOWS IN K8s"


IMAGE_REGISTRY = "TO BE FILLED"
IMAGE = "IMAGE NAME TO ADD HERE"

WORKFLOW_DEFAULTS = {}

PROJECT_DIR = Path(__file__)

logger = logging.getLogger()


def convert_camel_case_to_kebab_case(name: str):
    return "".join(["-" + c.lower() if c.isupper() else c for c in name]).lstrip("-")


def get_container(image_tag: str, envs: list[Env] = None) -> Container:
    return Container(
        name="k8s-pod",
        inputs=[
            Parameter(name="cmd"),
            Parameter(name="memory"),
            Parameter(name="cpu"),
        ],
        retry_strategy=RetryStrategy(limit="5", retry_policy=RetryPolicy.on_error),
        image=f"{IMAGE_REGISTRY}/{IMAGE}:{image_tag}",
        termination_message_policy="FallbackToLogsOnError",
        image_pull_policy="Always",
        # In order to define pod resources for each task, use podspecpatch
        pod_spec_patch="""
            {
                "containers":[
                    {
                        "name":"main",
                        "resources":{
                            "limits":{
                                "cpu": "{{inputs.parameters.cpu}}",
                                "memory": "{{inputs.parameters.memory}}"
                            },
                            "requests":{
                                "cpu": "{{inputs.parameters.cpu}}",
                                "memory": "{{inputs.parameters.memory}}"
                            }
                        }
                    }
                ]
            }
        """,
        env=envs,  # Add user specified envs
        command=["bash"],
        args=["-c", "{{inputs.parameters.cmd}}"],
    )


def get_pipeline(pipeline_name: str = None) -> Pipeline:
    metadata = bootstrap_project(Path(PROJECT_DIR))

    <http://logger.info|logger.info>("Project name: %s", metadata.project_name)

    <http://logger.info|logger.info>("Initializing Kedro...")

    pipeline_name = pipeline_name or "__default__"

    pipeline = pipelines.get(pipeline_name)

    return pipeline


def create_workflow(
    image_tag: str,
    envs: list[Env] = None,
    generate_name="kedro-wf-",
    kedro_env: str = "staging",
    kedro_pipeline_name: str = None,
    **extra_params
) -> Workflow:
    """Create a workflow"""

    if envs is None:
        envs = []

    with Workflow(
        generate_name=generate_name,
        entrypoint="main",
        **WORKFLOW_DEFAULTS,
    ) as w:
        k8s_pod = get_container(image_tag=image_tag, envs=envs)

        with DAG(name="main"):
            for node, deps in get_pipeline(
                kedro_pipeline_name
            ).node_dependencies.items():
                # node.name, [d.name for d in deps]
                kedro_cmd = (
                    f"kedro "
                    f"run "
                    f"--env "
                    f"{kedro_env} "
                    f"--nodes "
                    f"{node.name} "
                )
                if extra_params is not None:
                    kedro_cmd = kedro_cmd + f"--params {extra_params}"
                
                # Use can add tags to kedro nodes for compute heavy task to automatically
                # assign more CPU resources. The code below can be changed depending on 
                # user need. Similar logic can be implemented for memory as well
                if "ComputeHeavyTask" in node.tags:
                    cpu = "6"
                else:
                    cpu = "1"

                memory = "5Gi"

                k8s_pod(
                    name=convert_camel_case_to_kebab_case(f"{node.name}"),
                    arguments={
                        "cmd": kedro_cmd,
                        "cpu": cpu,
                        "memory": memory,
                    },
                    dependencies=[
                        convert_camel_case_to_kebab_case(d.name) for d in deps
                    ],
                )

    return w

Quick question about creating a custom Kedro starter. How do you pass the credential you enter at prompt (kedro new --starter=mystarter) to conf/local/credentials.yml ?

An idea that has been in the back of my mind is to reduce documentation effort: a confluence documentation plugin. A Kedro project could be a one-stop-shop for a developer, but it is inaccessible to others. There is a need to have automatically-generated documentation. A lot of organizations use Confluence, which provides information to a variety of different organizational members. If we could find a way to (1) do all of our detailed documentation (EDA decisions included) in Kedro, and then (2) have an easy migration tool to Confluence that would be pretty awesome. For example, we are putting all of our SQL queries and references to data files in the Kedro catalog anyway, so generating the data sources to a confluence page should be automatable. This type of documentation can provide other members of an organization information about where exactly the information came from, providing automatic data lineage for stakeholders. Have not read deep enough into the API to know the feasibility of getting triggered updates to the Confluence page when a plugin "confluence-generate" type of command is run. Just throwing it out there as something that would be DRY (don't repeat yourself) and satisfy a SSOT (single source of truth) principle for the current config of the project. Update one thing and the other thing updates. Anyway, just musing out loud. • Confluence module — Atlassian Python API 3.41.2 documentation (atlassian-python-api.readthedocs.io • Publish Python Project Documentation on Confluence/HTML using Sphinx | by Shravankumar Hiregoudar | Towards Data Science

Ignore that ☝️ - fat fingered 🙂 - real message inbound.

Hey all! - I am working with

kedro kubeflow

with a repo that has multiple pipelines and lots and lots of parameters in aggregate across pipelines, and I'm receiving an error when trying to upload a compiled manifest :

The pipeline spec is invalid.: Invalid input error: The input parameter length exceed maximum size of 10000.

This looks to be a limitation of K8s, specifically : https://github.com/kubeflow/pipelines/issues/2286 I've tried modifying some of the kedro-kubeflow plugin code to account for passing in specific pipelines via

kedro kubeflow compile --pipeline <blah> -o <pipeline_blah>.yml

and the manifest looks clean but I'm getting another error now :

kedro: error: argument --configure_logging: invalid <lambda> value: 'config.yaml'

and I was just curious if folks, including @marrrcin, had seen this previously, or not. 🙂

Hello everyone👋, I am glad to announce the release of Kedro Boot 🚀 a new community plugin that streamlines the integration between kedro projects and any generic applications ! It will help you inject application's data into kedro projects and dynamically orchestrating multiples pipeline runs 🎉. If you are intested, you can : • 📦 Download the package with

pip install kedro-boot

• G Visit Kedro Boot github repo, give feedback and ⭐ if you like it • 🧪 Try & Learn with Kedro Boot Examples and give feedback ! ✨ This enable using Kedro pipelines in a wide range of online and low latency use cases, including model serving, data apps (streamlit, dash), statistical simulations, paralell processing of unstructured data, streaming, and more. 💥 Kedro Boot propose an answer to many issues: #2627, #2169, #1993, #2626, #devrel7, #2663, #2182 #2879 #1846 #795 #933 #2058 #143 #1041 and numerous slack questions about dynamic pipelines, injecting external data, serving pipeline as API, exposing kedro's resources to a generic application. @datajoely wanted this feature for ages 🎁 😉 If the plugin proves to be valuable and effectively addresses the problem, @Yolan Honoré-Rougé and I will ensure its maintenance and support. 🧑‍💻 Takieddine & Yolan

Hi all! Using

kedro-mlflow

is it possible to retrieve a model from S3 based on it’s name and stage, directly from a catalog entry? E.g., I’d like to get the

production

version of the

foobar

model. The

.pkl

is saved as an artifact on S3. I was thinking of building a custom dataset that calls the Mlflow REST API, get’s the path to the model with the given name and stage, and then downloads the artifact from S3, but I wonder if there is a simpler/better way of doing it.

Question about the newly released `kedro-boot`: One of my workflow relies on serving the result of a pipeline given an input-parameter to a dataset. specifically, the api gets a list of ids, and uses these in a query to a database for filtering. the query is executed in a dataset, but these ids are only known at runtime. now i can add a dataset, set these ids to some preset and then overwrite them using omegaconfigloader, or templatedconfigloader and pass extra_params to the kedrosession. However, since

kedro-boot

allows one to override pipeline inputs dynamically, maybe its also possible to overwrite arguments passed to datasets? im specifically asking because

kedro-boot

seems to be the answer to using

kedro

with something like

FastApi

, but

FastApi

is routinely used for CRUD on a database, and kedro has this nice mechanism for handling credentials and such. So actually passing the credentials into a node to access a database is quite ugly.

Hi everyone, I’m looking into

kedro-airflow

and am having a little concern. Please correct me if I’m wrong. Since individual nodes are turned into airflow tasks as instances of the

KedroOperator

whose

execute()

method creates a new

session

, if one were to use versioned datasets, one would be in for a little surprise: A single airflow run would produce non-homogeneously timestamped artifacts… Correct ? Thanks in advance for your inputs / comments. M.

Another question wrt `kedro-boot`: how is do the

AppPipeline

objects work with

find_pipelines

? my

AppPipeline

is found by find_pipelines, but then when in the apps route i do:

kedro_boot_session.run(
            name="web_api",
…)

the view cannot be found. only when i do

kedro boot --app … --pipeline web_api

, my

AppPipeline

is included in the kedro_boot_session._pipeline.views

So I was trying to install

kedro-sagemaker

and got hit by an

AttributeError: cython_sources

error that seems to be related to this issue. I managed to work around it by installing

pyyaml

in advance:

pip install "cython<3.0.0" && pip install --no-build-isolation pyyaml==5.4.1

Has anyone seen anything like it?

Another

kedro-sagemaker

question: I manage to get the pipeline showing in Processing jobs but then I get an

Error: No such command 'sagemaker'.

error. I have

kedro-sagemaker

in my

requirements.txt

file and I’m building and pushing the image myself, so I just do a

kedro sagemaker run

. Any ideas what am I doing wrong?

Looks like https://github.com/kedro-org/kedro-viz/pull/1355 has made it impossible to embed kedro-viz in an iframe. That's a shame. That really should have been left as an option to the user, imho.

Would it be possible to do something like

with KedroSession.create(...) as session:
  session.run(SageMakerPipelinesRunner())

and run a pipeline with

kedro-sagemaker

from inside a Jupyter notebook?

Hi, I'm happy to announce that a new release

0.10.0

of

kedro-vertexai

plugin is here! 🎉 I'm kinda new here so I'm not sure if this belongs to #announcements also, as it's a plugin update. We added the grouping feature to allow more freedom in shaping your pipelines at vertexai. More details in the docs: https://kedro-vertexai.readthedocs.io/en/0.10.0/source/02_installation/02_configuration.html#example https://github.com/getindata/kedro-vertexai/releases/tag/0.10.0

question about kedro-mlflow, regarding the use of

MlflowModelRegistryDataSet

in the Kedro-Mlflow integration for logging models to MLflow’s model registry, as documented in the Kedro-MLflow Python Objects section. Initially, I followed the documentation for using

MlflowModelLoggerDataSet

in the

catalog.yml

file, which I implemented successfully. However, I encountered confusion with

MlflowModelRegistryDataSet

. My initial attempt was based on the following configuration:

my_transformer_model:
  type: kedro_mlflow.io.models.MlflowModelRegistryDataSet
  flavor: mlflow.transformers
  model_name: my_transformer_model_name
  stage_or_version: staging

When trying to save a model using

catalog.save("my_transformer_model", model)

, I received a

DatasetError

indicating that the ‘save’ method is not implemented for

MlflowModelRegistryDataSet

. The documentation provides parameters for this dataset but lacks a clear example for its correct usage in saving and registering a model to MLflow. Moving forward, I found a working solution for logging the transformer model in YAML API:

my_transformer_model:
    type: kedro_mlflow.io.models.MlflowModelLoggerDataSet
    flavor: mlflow.transformers
    save_args:
        registered_model_name: "my_transformer_model_name"

This allowed me to save and load the model to MLflow successfully. This however is not documented as such. For model loading, I could indeed use the initial catalog entry for loading specific versions directly, Yet, I still have unresolved queries w.r.t Model Staging/Versioning*:* How to stage or version the model directly through the API, instead of using the MLflow UI. so using the MlflowModelLoggerDataSet to save, but also specify a version/stage. In addition i was wondering how to view associated metrics with the model training run in the mlflow model UI to efficiently promote the best model to staging. I can imagine that including practical examples in the official documentation, would significantly enhance the user experience.

Hey there, I want to use kedro-mlflow for serving a pipeline via an API. What I'm not sure I understand is the input constraint - that it has to be part of the inputs for the training pipeline. In my case - the training pipeline takes data from 3 sources (a little like the spaceflights tutorial), processes each and joins them together. The API call will be with data that's similar to the joined table. How am I supposed to handle this situation? @Yolan Honoré-Rougé perhaps you can help :)

hi @Takieddine Kadiri

im trying to use kedro-boot in our application to cmmunicate with kedro project i have used the provided example by kedro (data_processing pipeline) kedro run --pipeline data_processing but im confused where to register this (AppPipeline) this below is documented in Github We need to registrer an

AppPipeline

by using

app_pipeline

factory but how we can register this ?

in your example your are using kedro-boot and kedro project together in my case we have a warehouse project we want to remotely control our other project (kedro project) which is deployed on other machine) my project is on other machine and kedro project is on another machine then how can we use kedro boot ?

Has anyone experimented with integrating hydra or hydra-zen as a ConfigLoader or through a plugin in Kedro? I'm particularly intrigued by hydra-zen's method of programmatically generating configurations, its runtime type validation, and the possibility of conducting hyperparameter sweeps with tools like Optuna. If anyone has developed a similar solution or knows someone who has, I'd greatly appreciate an introduction or any leads. Thank you!