<!here> This is your last opportunity to fill in our Databricks survey, if you have used Kedro and Databricks recently. It's a 5-minute survey that will need to be completed by today at 6PM (UTC +01:00).

Hi @Rashida Kanchwala I saw on the Kedro Showcase from October that you are starting to do some planning for Kedro viz development. I'd love to participate if possible. We think kedro viz is a great tool for us to communicate with 'non-technical' stakeholders. For us, one key feature/requirement would be to extend the types of information that can be displayed in the sidebar about a dataset or the intuition behind a node. For example, for each dataset, we store a yaml file with metadata about the dataset. We might want to read in a few choice fields from that file, display a description of the dataset, or even simply being able to display a url to our docs page, where each dataset will basically have its own page.

Hi! I'm not sure if this is quite the right place for something like this, but I created a feature request (https://github.com/kedro-org/kedro/issues/2016) to gauge interest in Avro support for the Kedro

dask.ParquetDataSet

. I had a use case where I needed to parse an existing Avro schema and transform it into a pyarrow schema so that the

<http://dask.to|dask.to>_parquet

function behaves nicely. I was wondering whether this is something the community would be interested in and would appreciate feedback.

Hi All, I would like to find out what users think about project documentation. Given the current doc system (i.e.

kedro build-docs

) will be discontinued with version 0.19, I wonder if a plugin should fill its place. @Asch Harwood suggested that there is a need for communicating with non-technical stakeholders - are there more users thinking this way? Would a plugin that assists with the creation of documentation be useful? In order to aid discussion, I played around with a prototype static site as an example for project documentation. I used the Kedro spaceflight tutorial project as a base, and you can explore the finished documentation here. Given that the Kedro framework defines much of the information needed for project documentation, I think it would be pretty straight forward to create a plugin that would: - create the basic documentation structure - fill in details about pipeline, nodes, data and parameters automatically - insert an interactive Kedro-Viz graph - provide empty templates for additional notes to write - example how to publish as a static website, for example to GitHub Pages Just to clarify, this is not a plugin, just a "fake" output for discussion. I would like to find out if: - such a plugin would be useful - maybe find a few projects other than spaceflight that could be used during development/specification - get clarity on desired functionality - maybe some collaborators interested in making it If you want to find out more you can also clone the project repo. There are instructions in the manual how to build the docs locally and use them. Leave your feedback either on slack or if concrete ideas, please create issues in the repo. Looking forward to hearing from you Sean

<!here> Hey everyone! We're designing a software engineering for data scientists and data engineers learning course (📚 read the rationale). There are so many topics we want to cover but we'll need your help prioritising the first version of this course. Please take a few minutes to fill out our survey by Wednesday, December 14th: https://www.surveys.online/jfe/form/SV_37WxRUy5xkC0xkW

I don’t really know if this is the right place to post this (happy to move it to a different channel), but here goes… We have been working with kedro for 2+ years now (mainly 0.16.6 and 0.17.7) in a highly customised setup to support ML pipelines for multiple product x country combinations. So I thought it would be a good idea to share some feedback. To be honest, I have always had a love-hate relationship with kedro (and especially our setup!), but lately I have started to like kedro more. Here’s why; if have aways compared kedro to python-based orchestration engines such as Prefect or Dagster. In reality, this is an unfair comparison and it makes much more sense to view kedro as the dbt (https://www.getdbt.com) for ML. In that regard, kedro is easy to use, enforces SWE best practices and plays nicely with DS tooling, which makes it an excellent package to use in an ML platform. However, I do think kedro can still pick up a few ideas from e.g Dagster/Prefect, and that brings me to the stuff I don’t like: • running nodes conditionally based on e.g the outcome of an upstream node or an config param • Having an easy way to create a map-reduce type pattern in a pipeline where you use one function multiple times in parallel with different inputs. • Without kedro-viz, it is really hard to see the dependencies between nodes in a pipeline due to the design of the pipeline API and that’s annoying if you just want to do some lookup or review a PR introducing a new pipeline. • Talking about kedro-viz, it would be nice if you could use it to track runtime stats of your pipeline (and nodes). Just like the Prefect API does. • One of the most impressive features in kedro is definitely its I/O abstraction with the catalog. However, in many big projects the catalog becomes huge. Drawing inspiration from Dagster’s software defined assets, I think we should strive to reduce the catalog to only contain the relevant datasets (your data assets from your data product if you think in terms of “data mesh”). I am just wondering if there is a way to enforce this more in a kedro project… Hope the feedback is useful! Apologies for the long post 🙈

With some 80% of the CLI being deprecated, is the basic idea that those departing functions will be replaced by plugins?? I realize this feedback is probably too late, but frankly a big part of the utility of kedro is to provide "training wheels" to data scientists in the form of a well documented CLI. Just to throw an idea out there: maybe instead of a hard deprecation, those commands could remain as placeholders to be explicitly overridden by plugins. In other words they could define the common entry points for most plugins instead of each plugin providing its own.

Interesting discussion in the MLOps community slack, which touches upon some user perceptions on Kedro:

Workflow Orchestration tools..

I have been crushing through the documentation of a bunch of different workflow orchestration tools. This is my inner map so far. [...]

Kubeflow, Metaflow, Flyte, Kedro, and ZenML focus more on ML pipelines and experimentation usability, like easy switching between local and cloud. Kubeflow is for ML what Argo is for data flows, so expect it to be a steep learning curve if you are not a Kubernetes expert, which most data scientists are not, so this might explain why it is frowned upon. All of these are new and shiny, but again I need to dig a little deeper to understand the differences. Kedro is opinionated about project structure and does not seems to be build with big scalable workflows in mind, and I got the feeling that Kedro is like DVC but more aimed towards ML specifically, and thus it might be a good fit for consultants that are building many smaller projects (?), Metaflow, Flyte, and ZenML all deal with how to utilize compute clusters in an easy way. ZenML seems to me like it might have some gaps, but it is also the newest one, so that is to be expected at this point in time.

Another member then replies:

Thanks for starting the thread, it's very interesting!

I'd like to clarify that Kedro is a Python library for building modular data science pipelines. Kedro helps you write data science workflows that are made of reusable components, each with a "single responsibility".

Kedro is not an orchestration tool like Argo Workflows or Kubeflow Pipelines. Check out the deployment guide for how to run Kedro pipelines on Airflow, Argo Workflows or Kubeflow Pipelines. We have successfully used Kedro to build data-science-friendly pipelines that we can still run at scale with Kubeflow Pipelines.

https://mlops-community.slack.com/archives/C015J2Y9RLM/p1675865574676169

Hi everyone! I'm Amanda a designer on the Kedro team 😎 I'm working on engaging our community more in the user research we do to guide our framework design choices. I'm going to start with a few small questions about logging, but do expect to hear from me more 🙂

This content can't be displayed.

@Victoria Sicking has left the channel

Vote on the new Kedro slogan and descriptions 🗳️

So close!

@Oleg Pilipenok has left the channel

This polly is closed. #devrel-team (Polly Sender: @Yetunde) has a polly for you! Votes in this polly are anonymous 🔒.

This polly is closed. @Juan Luis has a polly for you! Votes in this polly are anonymous 🔒.

Hi everyone! I'm Steph, a designer working on Kedro-Viz and recently joined the team 👋 K We'd love to learn more from our non-technical users - these are users who are not as comfortable using the CLI (command-line interface) and they use Kedro-Viz to follow 'what's going on' and inspect the code. I'd love to know - do we have any non-technical users in our community here? or do you work with someone that is a non-technical kedro user, perhaps your team lead or project stakeholders? If you are or know someone, please could you react with a 🙋 emoji or comment below? Many thanks!! 🤓

*Kedro-Viz: How do you use it? * The Kedro team wants to understand how our users are using Kedro-Viz and the outputs of a Kedro project. We’d also love to hear about the challenges you face and suggestions for improvement, your responses will be included and considered during Kedro-Viz development. If you prefer to share more feedback with a Kedro team member, please reach out as we are always up for a chat! From: @Stephanie Kaiser Non-Anonymous | Results Will Not Be Shared | <!date^1682518394^{date_short}, {time}|Apr 26, 2023, 14:13 PM> - <!date^1683123194^{date_short}, {time}|May 3, 2023, 14:13 PM> This polly is closed, with interactive elements

Hi Kedro user! 👋 Are you struggling with long catalog files with numerous dataset entries? Would you like to try out a ✨ new feature ✨ to help reduce the number of datasets you need to define in the catalog? Then respond to this message or reach out to me personally, because I would love to talk to you 😄 K

Hi <!channel>! The Kedro team needs your help! 🙏 We have written a short survey to find out more about how you learn and your experience with the Kedro documentation 📚 We'll use the information we get from it to prioritise improvements & make it easier to learn about Kedro. Please take the survey if you can; if your response is one of the first 30 received, we will send you something fancy with the new Kedro brand on it. K You can find it here: https://www.surveys.online/jfe/form/SV_9phsPFj95kdFkd8 The survey closes on Tuesday, 27th of June at 6PM (UTC+01:00). Thanks in advance! 🤖

If Kedro supports unpacking parameters dictionary, would it causes more confusion than convinence? IMO it is most beneficial if the function is reused multiple time and reduce the need of a thin function wrapper (and an extra file maybe). I am particular interested with people who use kedro to develop packages. https://github.com/kedro-org/kedro/issues/2749

<!here> Hey folks! 👋🏾 We're working on a new theme: "Keep the essentials." The goal is to let you use just the parts of Kedro you need and leave the rest behind. Here are the concepts we are exploring: • jupyter Use

catalog.yml

&

parameters.yml

straight from a Jupyter/Databricks/AWS SageMaker notebook without a project template or an IDE. • party wizard Use the project creation wizard to add features to your project template. Don't need the files and folders created by linting, testing, and documentation? No worries! Just skip those to get a simpler template. We'd love your help testing these ideas! If you can spare 30 minutes to try either of them, then indicate your interest with jupyter or party wizard. Your feedback will help make Kedro more flexible.

👋 Hey kedroid ’s we’ve just finished the first phase of a significant research phase around the current state of deploying Kedro to various ML Ops platforms. It’s pretty chunky but would love some comments / thoughts / reflections? https://github.com/kedro-org/kedro/issues/3094

<!here> anybody using Kedro for satellite/climate/scientific/Earth data? 🌍 we have 2 PRs open for GeoTIFF (using xarray), and NetCDF datasets, and would love to have more eyeballs on them: • GeoTIFF https://github.com/kedro-org/kedro-plugins/pull/355 • NetCDF https://github.com/kedro-org/kedro-plugins/pull/360 all help more than welcome 🙏🏼

Hello to all of our wonderful

PartitionedDataset

users out there! We have a question for you, related to enabling versioning for

PartitionedDataset

--which of the below options makes the most sense to you? 1. https://github.com/kedro-org/kedro/pull/521 proposes to enable versioning of the underlying dataset, by specifying

versioned: true

in the dataset config:

station_data:
  type: PartitionedDataset
  path: data/03_primary/station_data
  dataset:
    type: pandas.CSVDataset
    versioned: true

On the plus side, having the

versioned: true

config on the

dataset

config makes it clear that the versioning is applied to the underlying dataset, not to the

PartitionedDataset

. However, there are some edge cases (see https://github.com/kedro-org/kedro/pull/521#issuecomment-744653023, if you're keen). 2. Alternatively, we can move the

versioned: true

flag to the top level

PartitionedDataset

config:

station_data:
  type: PartitionedDataset
  path: data/03_primary/station_data
  versioned: true
  dataset:
    type: pandas.CSVDataset

Note that the versioning is still of the underlying dataset (e.g.

data/03_primary/station_data/first_station.csv/<version>/first_station.csv

), even though the config is at the top level. 3. None of these options make sense; what you really need is versioning of the top-level dataset. (Note that we don't have a solution designed for this case, but it would be great to know nonetheless!) Please feel free to vote using 1️⃣2️⃣3️⃣, and elaborate further on your thoughts in the thread below!

hello <!here>! 👋🏼 I come with a thorny question today: Python environments python before launching a proper poll, I'd like to ask some qualitative feedback from you: • what tool do you use to create and manage environments for your Kedro projects? venv, virtualenv, virtualenvwrapper, conda, mamba, Pipenv, Poetry, PDM, Hatch, Rye, nothing at all? • where do you place your environments when working with Kedro projects? a global location (

~/.miniconda

,

~/.virtualenvs

), or next to the code (

~/Projects/spaceflights/.venv

)? • when you create a new Kedro project, what are the steps you usually follow? for example 1. create and activate conda environment, 2.

pip install kedro

, 3.

kedro new

• what do you think of the current process? (please leave a reply on the thread 🧵, 1 comment per person to keep the conversation tidy) your feedback and ideas are very much welcome 🙏🏼

• use conda+pip (inside it) • used to hold env near in the project folder • create, install, new • current process is pretty straightforward

to avoid hijacking the thread, @Ben Horsburgh : https://kedro-org.slack.com/archives/C03R8N2M8KT/p1699261657794449?thread_ts=1698932258.475319&amp;cid=C03R8N2M8KT yes! in fact we're now using both: we declare dependencies in

requirements.txt

and them read them in

pyproject.toml

https://github.com/kedro-org/kedro/blob/93dc1a91e4bb476287040ea3db4a610696cacb0c/k[…]project/%7B%7B%20cookiecutter.repo_name%20%7D%7D/pyproject.toml but you can also just avoid

requirements.txt

files entirely. what do you think of this approach?

@Marc Gris finally gave the steps in https://kedro-org.slack.com/archives/C03R8N2M8KT/p1699265578467299?thread_ts=1698932258.475319&cid=C03R8N2M8KT and I'd like to start a new thread by asking: how do you do

kedro new

if you haven't installed Kedro yet? 🙃 cc @Lukas Innig

open question for #user-research: do you use Kedro for ETL pipelines? if so, what good practices do you follow to enable idempotency and efficiency? for example, let's say I'm (E)xtracting data from some API endpoint that has paginated results and (L)oading it in some database of mine. ideally, if I run the pipeline twice, it would only extract the data starting from the last loaded data point, and then it would append the data at the end. this somehow requires that the pipeline reads the status of the loaded data before starting the extraction. but this creates a cycle in the DAG (

A node cannot have the same inputs and outputs

) so it requires you to define a read-only version of the dataset and an appendable version, both referring to the same underlying storage. any thoughts on this approach?