Yetunde
06/09/2023, 4:24 PMkedro
, kedro-viz
, awesome-kedro
, our docs, and our website.
In our releases, we also urge you to look for new features like variable interpolation with our OmegaConfigLoader
in Kedro 0.18.10 and multi-user experiment tracking from Kedro-Viz 6.2.0.
How do you learn more?
• Have a look at our rebranding blogpost
• Check our branding guidelinesJuan Luis
06/22/2023, 9:38 AMJuan Luis
06/26/2023, 8:57 AMJuan Luis
07/03/2023, 2:54 PM0.18.11
is out! K
TLDR: In this release, we added added a newKedro 0.18.11 is a non-breaking release in the 0.18.x series, which means you can upgrade with:official starter and significantly improved the documentation around Databricks deployments. We also fixed some bugs around micropackaging and remote datasets, updated the documentation for Prefect 2.0, and deprecated some class names.databricks-iris
pip install kedro==0.18.11
or
conda/mamba/micromamba install -c conda-forge kedro=0.18.11
🚀 Major Features and improvements
• Added databricks-iris
as an official starter.
🪲 Bug fixes and other changes
• Reworked micropackaging workflow to use standard Python packaging practices.
• Make kedro micropkg package
accept --verbose
.
• Compare for protocol and delimiter in PartitionedDataSet
to be able to pass the protocol to partitions which paths starts with the same characters as the protocol (e.g. <s3://s3-my-bucket>
).
✍️ Documentation changes
• Significant improvements to the documentation that covers working with Databricks and Kedro, including a new page for workspace-only development, and a guide to choosing the best workflow for your use case.
• Updated documentation for deploying with Prefect for version 2.0.
⚠️ Upcoming deprecations
• Renamed dataset and error classes, in accordance with the Kedro lexicon. Dataset classes ending with "DataSet" and error classes starting with "DataSet" are deprecated and will be removed in 0.19.0.
💌 Community contributions
• Many thanks to jmalovera10, @Rahul Kumar, @Julius Hetzel, jacobweiss2305, eduardoconto and tomasvanpottelbergh for contributing PRs to this release!
We welcome every community contribution, large or small. See what we're working on now and report bugs or suggest future features.
Until next time,
The Kedro Team 💛Nero Okwa
07/07/2023, 4:43 PM6.3.2
is out! KKKKK
@here we're excited to announce that Kedro-Viz 6.3.2 is out! 🥳
Kedro-Viz is the interactive development tool for building and visualising data science pipelines with Kedro. It enables you to monitor the status of your ML project, present it to stakeholders, and smoothly bring new team members onboard.
Kedro-Viz also comes with an experiment tracking feature enabling you to compare different runs of a Kedro project, and you can also preview code and datasets.
We've added a new feature and several bug fixes in the 6.3 versions. Check out the demo and get full details in the release notes.
🚀 What can you expect in this release?
• Added validation for layers in transcoding datasets.
• Added support for dataset preview as specified in the Kedro catalog under metadata.
• Updated the UI with the new Kedro rebrand look and feel.
• Enabled original node input and output names in metadata panel.
🪲 Bug fixes and other changes
• Fixed large pipeline layout warning message.
• Fixed flowchart not updating on back/forward clicks after selecting a registered pipeline.
• Fixed broken URL link between experiment tracking and flowchart in a non-default pipeline.
How do I get the latest release?
• Python: pip install kedro-viz==6.3.2
• React: npm install @quantumblack/kedro-viz@latest
What's next?
• Shareable URLs for Kedro-Viz pipeline visualisation.
Follow our roadmap for more updates, and suggest future features to the community.
Until next time,
The Kedro TeamLinenBot
07/31/2023, 9:04 AMtynan.debold
joined #announcements.Merel
08/01/2023, 4:13 PM0.18.12
is now available! K and so are kedro-datasets 1.5.1
& kedro-airflow 0.6.0
📣
TLDR: In the new release of Kedro, we introduce the dataset factories feature, activated all built-in resolvers by default for OmegaConfigLoader (except for oc.env), and added a new CLI command to rank dataset factories. We made some important bug fixes and updated the documentation, notably recommending ruff as the linter.See the 🧵 for details of each release. We welcome every community contribution, large or small. See what we're working on now and report bugs or suggest future features. Until next time, The Kedro Team 💛
Nero Okwa
08/18/2023, 8:34 AMImportError
as kedro-datasets is now lazily loaded.
• Fixed Sidebar search result based on Pretty name setting.
• Fixed issue of encountering a black page in Safari when interacting with modular pipelines.
How do I get the latest release?
• Python: pip install kedro-viz==6.4.0
• React: npm install @quantumblack/kedro-viz@latest
What's next?
• Shareable URLs for Kedro-Viz pipeline visualisation.
• Investigating the debugging use case of Kedro-Viz.
Try these features and provide feedback.
Follow our roadmap for more updates, comment on the tickets, and suggest future features to the community.
Until next time,
The Kedro Team
KKJuan Luis
08/18/2023, 2:40 PMkedro-datasets
1.5.3 is out!
if you've been having problems installing the optional dependencies ("extras") lately, hopefully this release addresses them all. thank you for your patience!
on the other hand, @FlorianGD contributed a fix to api.APIDataSet
so that now it doesn't perform an extra encoding of the responses, and @Flavien contributed an improvement to databricks.ManagedTableDataSet
so that now it's read-only by default. thanks a lot! kedroid
read the complete release notes online: https://github.com/kedro-org/kedro-plugins/releases/tag/kedro-datasets-1.5.3
until next time,
The Kedro Team 💛LinenBot
08/24/2023, 12:08 PMemil
joined #announcements.Juan Luis
08/31/2023, 2:34 PM0.18.13
and kedro-datasets 1.6.0
are available! K
TLDR: In this new release of Kedro we finally introduce official support for Python 3.11, as well as some newKedro 0.18.13 is a non-breaking release in the 0.18.x series, which means you can upgrade with:features, a newOmegaConfigLoader
command, and a leaner project structure. We also made significant documentation improvements around the Data Catalog and the datasets. Preparations for 0.19.0 are underway!kedro catalog resolve
pip install kedro==0.18.13
or
conda/mamba/micromamba install -c conda-forge kedro=0.18.13
🚀 Major Features and improvements
• Added support for Python 3.11.
• Added new OmegaConfigLoader
features: registering of custom resolvers through CONFIG_LOADER_ARGS
and support for global variables.
• Added kedro catalog resolve
CLI command that resolves dataset factories in the catalog with any explicit entries in the project pipeline.
• Implemented a flat conf/
structure for modular pipelines and updated kedro pipeline create
and kedro catalog create
accordingly.
• Made various updates to the Kedro project template and Kedro starters: use of OmegaConfigLoader
, transition from setup.py
to pyproject.toml
, and flat conf/
structure.
🪲 Bug fixes and other changes
• Updated OmegaConfigLoader
to ignore config from hidden directories like .ipynb_checkpoints
.
✍️ Documentation changes
• Restructure beginner and advanced pages about the Data Catalog and datasets.
• Moved contributor documentation to the GitHub wiki.
• Updated example of using generator functions in nodes.
• Added migration guide from the ConfigLoader
and the TemplatedConfigLoader
to the OmegaConfigLoader
.
⚠️ Upcoming deprecations
• The ConfigLoader
and the TemplatedConfigLoader
are deprecated and will be removed in the 0.19.0
release.
• AbstractDataSet
becomes AbstractDataset
and AbstractVersionedDataSet
becomes AbstractVersionedDataset
, old names are deprecated and will be removed in 0.19.0.
• Using the layer
attribute at the top level is deprecated and it will be removed in Kedro 0.19.0
. Please move layer
inside the metadata -> kedro-viz
attributes.
🐍 Migration Tips for Python 3.11
• Users on Windows with Python >=3.8 should note we've pinned pytables
to 3.8.0
due to compatibility issues.
💌 Community contributions
Many thanks to @Laiza Milena Scheid Parizotto and @Jon Cohen for contributing PRs to this release!
🚨 This will be among the last releases in the 0.18.x line, very soon we will start working on 0.19.0. Stay tuned!
We welcome every community contribution, large or small. See what we're working on now and report bugs or suggest future features.
Until next time,
The Kedro Team 💛Nero Okwa
08/31/2023, 5:23 PMpip install kedro-viz==6.5.0
• React: npm install @quantumblack/kedro-viz@latest
What's next?
• Shareable URLs for Kedro-Viz pipeline visualisation.
• Investigating the debugging use case of Kedro-Viz.
Try these features and provide feedback.
Follow our roadmap for more updates, comment on the tickets, and suggest future features to the community.
Until next time,
The Kedro Team
KKLinenBot
09/11/2023, 1:19 PMjonathan_daniel
joined #announcements. Also, nok.lam.chan
joined.Yetunde
09/26/2023, 4:48 PMkedro-mlflow
, kedro-pandera
, and he is thoughtful about progressing Kedro's roadmap and development. You'll experience him as a helpful community member in channels like #plugins-integrations. You can read more about our TSC.
We're also celebrating the open sourcing of a new project that already integrates with Kedro. This is a QuantumBlack project that we've been excited about for a while, and it is called Vizro - I swear we did not name it. 🤡
Vizro is a toolkit for creating modular data visualisation applications, and it is built on Plotly and Dash. It integrates with Kedro and allows you to trigger pipeline re-runs from the dashboard views. 🚀
You can:
• 📊 See a live demo
• ⭐ Visit and star the GitHub repo (if you want)
• 📚 Check out the documentation
• 🦠 Read a Medium blog post about VizroJo Stichbury
09/27/2023, 9:11 AMThe use of direct language with consideration given to different levels of expertise is very effective. This combined with the clear and user friendly navigation makes the documentation very attractive. The layout was really clear.Thanks to all in the team for believing in great communication! 🤩 🏆 And to you the users for giving us feedback, and contributions, to guide us, inspire us, and make it a worthwhile and satisfying task.
Juan Luis
10/03/2023, 2:11 PMJo Stichbury
10/06/2023, 8:35 AMJo Stichbury
10/10/2023, 9:15 AMNero Okwa
10/11/2023, 3:18 PMNero Okwa
10/16/2023, 4:55 PMJo Stichbury
10/17/2023, 8:36 AMJuan Luis
10/19/2023, 11:19 AM0.18.14
is available! K
> TLDR: In this new release of Kedro we added support to customise the pipeline directory structure and overriding the configuration keys, made substantial improvements to the documentation (including a page on how to use Kedro from Jupyter notebooks), and squashed a number of bugs. We had a record number of community contributions thanks to Hacktoberfest. Preparations for 0.19.0 are underway!
Kedro 0.18.14 is the last non-breaking release in the 0.18.x series, which means you can upgrade with:
pip install kedro==0.18.14
or
conda/mamba/micromamba install -c conda-forge kedro=0.18.14
🚀 Major Features and improvements
• Allowed using of custom cookiecutter templates for creating pipelines with --template
flag for kedro pipeline create
or via template/pipeline
folder.
• Allowed overriding of configuration keys with runtime parameters using the runtime_params
resolver with OmegaConfigLoader
.
🪲 Bug fixes and other changes
• Updated dataset factories to resolve nested catalog config properly.
• Made settings.py
optional.
✍️ Documentation changes
• Added a Jupyter notebook example for spaceflights to illustrate how to incrementally add Kedro features.
• Added new documentation about deploying a Kedro project with Amazon EMR Serverless.
• Added new documentation about how to publish a Kedro-Viz project to make it shareable.
• New TSC members added to the page and the organisation of each member is also now listed.
⚠️ Upcoming deprecations
• Remember, all dataset classes will be removed from the core Kedro repository (kedro.extras.datasets
). Install and import them from the kedro-datasets
package instead.
• All dataset classes ending with DataSet
are deprecated and will be removed in Kedro 0.19.0
and kedro-datasets
2.0.0
. Instead, use the updated class names ending with Dataset
.
• The starters pandas-iris
, pyspark-iris
, pyspark
, and standalone-datacatalog
are deprecated and will be archived in Kedro 0.19.0.
• PartitionedDataset
and IncrementalDataset
have been moved to kedro-datasets
and will be removed in Kedro 0.19.0
. Install and import them from the kedro-datasets
package instead.
kedroid hacktoberfest2023 Community contributions
This release received the largest number of community contributions ever, thanks in large part to Hacktoberfest. A million thanks to @Laiza Milena Scheid Parizotto , @Miguel Angel Ortiz Marin, @R.X.M, @Iñigo Hidalgo, @Harmony, @Celsuss, and the rest of the contributors who aren't on this Slack for submitting PRs to this release!
🚨 This will (hopefully) be the last release in the 0.18.x line, next one will be 0.19.0. Stay tuned!
Read the complete release notes on https://github.com/kedro-org/kedro/blob/main/RELEASE.md#release-01814
We welcome every community contribution, large or small. See what we're working on now and report bugs or suggest future features.
Until next time,
The Kedro Team 💛Nero Okwa
10/25/2023, 10:13 AMpip install kedro-viz==6.6.1
• React: npm install @quantumblack/kedro-viz@latest
What's next?
• Publish and share Kedro-Viz from the command line.
• Publish and share Kedro-Viz on other platforms (beyond AWS).
• Investigating the debugging use case of Kedro-Viz.
Try these features and provide feedback.
Follow our roadmap for more updates, comment on the tickets, and suggest future features to the community.
Until next time,
The Kedro Team
KKJuan Luis
10/26/2023, 7:00 AMPartitionedDataSet
and IncrementalDataSet
from the core Kedro repo to kedro-datasets
and renamed to PartitionedDataset
and IncrementalDataset
.
• Added polars.LazyPolarsDataset
, a GenericDataSet
using Polars's lazy API.
• Renamed polars.GenericDataSet
to polars.EagerPolarsDataset
to better reflect the difference between the two dataset classes.
• Added a deprecation warning when using polars.GenericDataSet
or polars.GenericDataset
that these have been renamed to polars.EagerPolarsDataset
• Delayed backend connection for pandas.SQLTableDataset
, pandas.SQLQueryDataset
, and snowflake.SnowparkTableDataset
. In practice, this means that a dataset's connection details aren't used (or validated) until the dataset is accessed. On the plus side, the cost of connection isn't incurred regardless of when or whether the dataset is used.
A kedro-airflow 0.7
• Added support for Python 3.11.
• Added the --all
CLI argument to kedro-airflow
to convert registered all pipelines at once.
• Simplify the output of the kedro airflow create
command.
• Fixed compatibility of kedro-airflow
with older versions of the config loaders (kedro<=0.18.2
).
• Removed support for Python 3.7.
docker kedro-docker 0.4
• Added support for Python 3.11.
• Removed support for Python 3.7.
📡 kedro-telemetry 0.3
• Added support for Python 3.11.
• Removed support for Python 3.7.
many thanks to our community contributors Peter Bludau, Alistair McKelvie, @H. Felix Wittmann, @Matthias Roels, and Simon Brugman 👏🏼
python notice that we dropped Python 3.7 and added support for Python 3.11 across the board. going forward, for kedro-datasets we will follow NEP 29 (hence a 42 month support window - Python 3.8 will be dropped soon!) and for kedro framework and the other official plugins we will drop Python versions that reach their end-of-life status without making a major release.
(And yes, Python 3.12 is coming soon!)
Until next time,
The Kedro Team 💛Juan Luis
11/02/2023, 11:33 AMMerel
11/07/2023, 9:46 AM0.19.0
, which we’re aiming to get to you by the end of the year. There’s a lot happening behind the scenes and we wanted to give you a heads-up about a few changes that you can adopt now to be ready when 0.19.0
hits the streets.
1. We’ve written migration guides to help you transition from the legacy ConfigLoader
and TemplatedConfigLoader
which will be removed in Kedro 0.19.0
in favour of OmegaConfigLoader
2. Datasets move to their own package kedro-datasets and out of the Kedro core package in Kedro 0.19.0
. You can make the change to use that package now. Further, from version 2.0.0
of kedro-datasets
, which will be released together with Kedro 0.19.0
, all dataset names have changed to replace the capital letter “S” in “DataSet” with a lower case “s”. For example, CSVDataSet
is now CSVDataset
.
If you need help to update your projects in preparation of Kedro 0.19.0
or post release, please reach out to the team, we’re more than happy to help with this process.
We are very excited about the changes we’re bringing you in this release and more detailed communications about all changes will go out when the release is done. If you are curious, you can already have a look at the preliminary 0.19.0
release notes: https://github.com/kedro-org/kedro/blob/develop/RELEASE.md#upcoming-release-0190
Until next time,
The Kedro Team 💛Nero Okwa
11/17/2023, 5:18 PMpip install kedro-viz==6.7.0
• React: npm install @quantumblack/kedro-viz@latest
What's next?
• Publish and share Kedro-Viz from the command line.
• Upgrade to React 18.
kedroid hacktoberfest2023 Community contributions
Many thanks to our Hacktoberfest community contribution by Alain Anghelidi, for submitting a PR for this release 👏🏼
You too can contribute by using these features, and providing feedback.
Follow our roadmap for more updates, comment on the tickets, and suggest future features to the community.
Until next time,
The Kedro Team
KKdatajoely
11/22/2023, 12:34 PM0.10.0
of kedro-vertexai
plugin is here! 🎉
I'm kinda new here so I'm not sure if this belongs to #announcements also, as it's a plugin update.
We added the grouping feature to allow more freedom in shaping your pipelines at vertexai. More details in the docs:
https://kedro-vertexai.readthedocs.io/en/0.10.0/source/02_installation/02_configuration.html#example
https://github.com/getindata/kedro-vertexai/releases/tag/0.10.0
Slack conversationJuan Luis
11/29/2023, 11:44 AMOmegaConfigLoader
, how to use S3 and MinIO for your filepaths, and how to deploy Kedro to a Docker image or Airflow.
we hope you enjoy it! K
(also if you can spread the word in our socials, that would be fantastic! https://www.linkedin.com/posts/kedro-python_introduction-to-kedro-building-maintainab[…]384343457793-0uB7?utm_source=share&utm_medium=member_desktop)LinenBot
11/29/2023, 1:12 PMRuben s
joined #announcements. Also, Ruben
joined.