Jo Stichbury
10/06/2023, 8:35 AMJo Stichbury
10/10/2023, 9:15 AMNero Okwa
10/11/2023, 3:18 PMNero Okwa
10/16/2023, 4:55 PMJo Stichbury
10/17/2023, 8:36 AMJuan Luis
10/19/2023, 11:19 AM0.18.14
is available! K
> TLDR: In this new release of Kedro we added support to customise the pipeline directory structure and overriding the configuration keys, made substantial improvements to the documentation (including a page on how to use Kedro from Jupyter notebooks), and squashed a number of bugs. We had a record number of community contributions thanks to Hacktoberfest. Preparations for 0.19.0 are underway!
Kedro 0.18.14 is the last non-breaking release in the 0.18.x series, which means you can upgrade with:
pip install kedro==0.18.14
or
conda/mamba/micromamba install -c conda-forge kedro=0.18.14
🚀 Major Features and improvements
• Allowed using of custom cookiecutter templates for creating pipelines with --template
flag for kedro pipeline create
or via template/pipeline
folder.
• Allowed overriding of configuration keys with runtime parameters using the runtime_params
resolver with OmegaConfigLoader
.
🪲 Bug fixes and other changes
• Updated dataset factories to resolve nested catalog config properly.
• Made settings.py
optional.
✍️ Documentation changes
• Added a Jupyter notebook example for spaceflights to illustrate how to incrementally add Kedro features.
• Added new documentation about deploying a Kedro project with Amazon EMR Serverless.
• Added new documentation about how to publish a Kedro-Viz project to make it shareable.
• New TSC members added to the page and the organisation of each member is also now listed.
⚠️ Upcoming deprecations
• Remember, all dataset classes will be removed from the core Kedro repository (kedro.extras.datasets
). Install and import them from the kedro-datasets
package instead.
• All dataset classes ending with DataSet
are deprecated and will be removed in Kedro 0.19.0
and kedro-datasets
2.0.0
. Instead, use the updated class names ending with Dataset
.
• The starters pandas-iris
, pyspark-iris
, pyspark
, and standalone-datacatalog
are deprecated and will be archived in Kedro 0.19.0.
• PartitionedDataset
and IncrementalDataset
have been moved to kedro-datasets
and will be removed in Kedro 0.19.0
. Install and import them from the kedro-datasets
package instead.
kedroid hacktoberfest2023 Community contributions
This release received the largest number of community contributions ever, thanks in large part to Hacktoberfest. A million thanks to @Laiza Milena Scheid Parizotto , @Miguel Angel Ortiz Marin, @R.X.M, @Iñigo Hidalgo, @Harmony, @Celsuss, and the rest of the contributors who aren't on this Slack for submitting PRs to this release!
🚨 This will (hopefully) be the last release in the 0.18.x line, next one will be 0.19.0. Stay tuned!
Read the complete release notes on https://github.com/kedro-org/kedro/blob/main/RELEASE.md#release-01814
We welcome every community contribution, large or small. See what we're working on now and report bugs or suggest future features.
Until next time,
The Kedro Team 💛Nero Okwa
10/25/2023, 10:13 AMpip install kedro-viz==6.6.1
• React: npm install @quantumblack/kedro-viz@latest
What's next?
• Publish and share Kedro-Viz from the command line.
• Publish and share Kedro-Viz on other platforms (beyond AWS).
• Investigating the debugging use case of Kedro-Viz.
Try these features and provide feedback.
Follow our roadmap for more updates, comment on the tickets, and suggest future features to the community.
Until next time,
The Kedro Team
KKJuan Luis
10/26/2023, 7:00 AMPartitionedDataSet
and IncrementalDataSet
from the core Kedro repo to kedro-datasets
and renamed to PartitionedDataset
and IncrementalDataset
.
• Added polars.LazyPolarsDataset
, a GenericDataSet
using Polars's lazy API.
• Renamed polars.GenericDataSet
to polars.EagerPolarsDataset
to better reflect the difference between the two dataset classes.
• Added a deprecation warning when using polars.GenericDataSet
or polars.GenericDataset
that these have been renamed to polars.EagerPolarsDataset
• Delayed backend connection for pandas.SQLTableDataset
, pandas.SQLQueryDataset
, and snowflake.SnowparkTableDataset
. In practice, this means that a dataset's connection details aren't used (or validated) until the dataset is accessed. On the plus side, the cost of connection isn't incurred regardless of when or whether the dataset is used.
A kedro-airflow 0.7
• Added support for Python 3.11.
• Added the --all
CLI argument to kedro-airflow
to convert registered all pipelines at once.
• Simplify the output of the kedro airflow create
command.
• Fixed compatibility of kedro-airflow
with older versions of the config loaders (kedro<=0.18.2
).
• Removed support for Python 3.7.
docker kedro-docker 0.4
• Added support for Python 3.11.
• Removed support for Python 3.7.
📡 kedro-telemetry 0.3
• Added support for Python 3.11.
• Removed support for Python 3.7.
many thanks to our community contributors Peter Bludau, Alistair McKelvie, @H. Felix Wittmann, @Matthias Roels, and Simon Brugman 👏🏼
python notice that we dropped Python 3.7 and added support for Python 3.11 across the board. going forward, for kedro-datasets we will follow NEP 29 (hence a 42 month support window - Python 3.8 will be dropped soon!) and for kedro framework and the other official plugins we will drop Python versions that reach their end-of-life status without making a major release.
(And yes, Python 3.12 is coming soon!)
Until next time,
The Kedro Team 💛Juan Luis
11/02/2023, 11:33 AMMerel
11/07/2023, 9:46 AM0.19.0
, which we’re aiming to get to you by the end of the year. There’s a lot happening behind the scenes and we wanted to give you a heads-up about a few changes that you can adopt now to be ready when 0.19.0
hits the streets.
1. We’ve written migration guides to help you transition from the legacy ConfigLoader
and TemplatedConfigLoader
which will be removed in Kedro 0.19.0
in favour of OmegaConfigLoader
2. Datasets move to their own package kedro-datasets and out of the Kedro core package in Kedro 0.19.0
. You can make the change to use that package now. Further, from version 2.0.0
of kedro-datasets
, which will be released together with Kedro 0.19.0
, all dataset names have changed to replace the capital letter “S” in “DataSet” with a lower case “s”. For example, CSVDataSet
is now CSVDataset
.
If you need help to update your projects in preparation of Kedro 0.19.0
or post release, please reach out to the team, we’re more than happy to help with this process.
We are very excited about the changes we’re bringing you in this release and more detailed communications about all changes will go out when the release is done. If you are curious, you can already have a look at the preliminary 0.19.0
release notes: https://github.com/kedro-org/kedro/blob/develop/RELEASE.md#upcoming-release-0190
Until next time,
The Kedro Team 💛Nero Okwa
11/17/2023, 5:18 PMpip install kedro-viz==6.7.0
• React: npm install @quantumblack/kedro-viz@latest
What's next?
• Publish and share Kedro-Viz from the command line.
• Upgrade to React 18.
kedroid hacktoberfest2023 Community contributions
Many thanks to our Hacktoberfest community contribution by Alain Anghelidi, for submitting a PR for this release 👏🏼
You too can contribute by using these features, and providing feedback.
Follow our roadmap for more updates, comment on the tickets, and suggest future features to the community.
Until next time,
The Kedro Team
KKdatajoely
11/22/2023, 12:34 PM0.10.0
of kedro-vertexai
plugin is here! 🎉
I'm kinda new here so I'm not sure if this belongs to #announcements also, as it's a plugin update.
We added the grouping feature to allow more freedom in shaping your pipelines at vertexai. More details in the docs:
https://kedro-vertexai.readthedocs.io/en/0.10.0/source/02_installation/02_configuration.html#example
https://github.com/getindata/kedro-vertexai/releases/tag/0.10.0
Slack conversationJuan Luis
11/29/2023, 11:44 AMOmegaConfigLoader
, how to use S3 and MinIO for your filepaths, and how to deploy Kedro to a Docker image or Airflow.
we hope you enjoy it! K
(also if you can spread the word in our socials, that would be fantastic! https://www.linkedin.com/posts/kedro-python_introduction-to-kedro-building-maintainab[…]384343457793-0uB7?utm_source=share&utm_medium=member_desktop)Juan Luis
12/14/2023, 12:57 PM0.19
is available! K
> TLDR: Kedro 0.19 is a new major release that contains some breaking changes with respect to 0.18.x. Among other things, we introduced project tools that help you further customise the result of kedro new
, added support for soft merge in OmegaConfigLoader
, improved the error messages when datasets are not found, and modernised the default project templates by having the build configuration and project metadata in pyproject.toml
. Datasets are gone (you must install kedro-datasets
instead), alternative config loaders are gone too (only OmegaConfigLoader
remains) and some deprecated CLI commands were removed. To help you upgrade, we wrote a migration guide.
*Y*ou can install Kedro 0.19 with:
pip install kedro==0.19.1
or
conda/mamba/micromamba install -c conda-forge kedro=0.19.1
(Yes, there's also 0.19.0, but we detected a problematic bug with it so we went ahead and released 0.19.1 with a fix immediately after)
You can find the release highlights below, for the complete list of changes go to the release notes https://github.com/kedro-org/kedro/blob/main/RELEASE.md#release-0191 and https://github.com/kedro-org/kedro/blob/main/RELEASE.md#release-0190
🚀 Major Features and improvements
• Dropped Python 3.7 support.
• Introduced project tools and example to the `kedro new` CLI flow.
• The new spaceflights starters, spaceflights-pandas
, spaceflights-pandas-viz
, spaceflights-pyspark
, and spaceflights-pyspark-viz
can be used with the kedro new
command with the --starter
flag.
• Added the --conf-source
option to %reload_kedro
, allowing users to specify a source for project configuration.
• Added the functionality to choose a merging strategy for config files loaded with `OmegaConfigLoader`.
• Modified the mechanism of importing datasets, raise more explicit error when dependencies are missing.
• Added validation for configuration file used to override run commands via the CLI.
• Moved the default environment base
and local
from config loader to _ProjectSettings
. This enables the use of config loader as a standalone class without affecting existing Kedro Framework users.
🪲 Bug fixes and other changes
• Added a new field tools
to pyproject.toml
when a project is created.
• Reduced spaceflights
data to minimise waiting times during tutorial execution.
• Added validation to node tags to be consistent with node names.
• Removed pip-tools
as a dependency.
• Accepted path-like filepaths more broadly for datasets.
• Loosened pin for kedro-telemtry
to fix dependency issues in 0.19.0
.
💥 Breaking changes
• Removed ConfigLoader
and TemplatedConfigLoader
.
• Removed kedro.extras.datasets
and tests (use kedro-datasets
instead)
• Removed PartitionedDataset
and IncrementalDataset
from <http://kedro.io|kedro.io>
(import them from kedro-datasets
instead)
• logging
is removed from OmegaConfigLoader
in favour of the environment variable KEDRO_LOGGING_CONFIG
.
• Removed support for defining the layer
attribute at top-level within DataCatalog.
• Renamed data_set
and DataSet
to dataset
and Dataset
everywhere.
• Removed the create_default_data_set()
method in the Runner
in favour of using dataset factories to create default dataset instances.
• The default project template now has only one pyproject.toml
at the root of the project (containing both the packaging metadata and the Kedro build config).
✍️ Documentation changes
• Added new top navigation to easily switch between Framework, Viz, and Datasets.
• Added new search-as-you-type to improve the search experience.
kedroid Community contributions
A million thanks to Jeroldine Akuye Oakley, @Laiza Milena Scheid Parizotto, @Mustapha Abdullahi, @Adam Kells, Ajay Gonepuri for submitting PRs to this release!
🚨 If you are upgrading from Kedro 0.18, have a look at the migration guide.
We welcome every community contribution, large or small. See what we're working on now and report bugs or suggest future features.
Until next time,
The Kedro Team 💛Nero Okwa
12/19/2023, 6:10 PMkedro viz run
, and added several improvements and bug fixes in this version. You can find the full details in the release notes and check out the demo.
🚀 What can you expect in this release?
• Publish and share Kedro-Viz on AWS from the command line using the command kedro viz deploy
.
• Changed the CLI command to run Kedro-viz to`kedro viz run`.
• Upgraded to React 18.
• Dropped support for python=3.7
.
• Dropped support for kedro==0.17.x
.
How do I get the latest release?
• Python: pip install kedro-viz==7.0
• React: npm install @quantumblack/kedro-viz@latest
You too can contribute by using these features, and providing feedback.
Follow our roadmap for more updates, comment on the tickets, and suggest future features to the community.
Until next time,
The Kedro Team
KKJuan Luis
12/21/2023, 12:00 PMJuan Luis
01/08/2024, 11:04 AMJuan Luis
01/23/2024, 10:18 AMtools
• Allowed modern versions of JupyterLab and Jupyter Notebooks
• Removed setuptools dependency
• Added source_dir
explicitly in pyproject.toml
for non-src layout project
• MemoryDataset
entries are now included in free outputs
• Removed black dependency and replaced its functionality with ruff format
• Added logging about not using async mode in SequentialRunner
and ParallelRunner
💥 (Minor) Breaking changes to the API
(These are minor breaking changes that we have decided to include early to avoid disrupting user workflows later on in the cycle)
• Changed input format for tools option obtained from --config
file from numbers to short names.
✍🏼 Documentation changes
• Added documentation about bootstrap_project
and configure_project
.
• Added documentation about kedro run
and hook execution order.
Read the complete release notes on https://github.com/kedro-org/kedro/blob/main/RELEASE.md#release-0192 We welcome every community contribution, large or small. See what we're working on now and report bugs or suggest future features.
Until next time,
The Kedro Team 💛Nero Okwa
01/23/2024, 10:37 AMkedro viz build
CLI command, enabling you to generate a Kedro-Viz instance using your Kedro project data. This makes it easy to share Kedro-Viz on any static hosting platform. Detailed instructions for this feature are available in our documentation.
This version also includes other improvements and bug fixes outlined in the release notes. You can check out the demo.
🚀 What can you expect in this release?
• Added the build CLI command kedro viz build
to enable users to create a build directory of Kedro-Viz instance with Kedro project data.
How do I get the latest release?
• Python: pip install kedro-viz==7.1.0
• React: npm install @quantumblack/kedro-viz@latest
You too can contribute by using these features, and providing feedback.
Follow our roadmap for more updates, comment on the tickets, and suggest future features to the community.
Until next time,
The Kedro Team 💛
KKJuan Luis
01/31/2024, 12:59 PMJuan Luis
02/16/2024, 11:54 AMkedro-boot
, a new plugin created by @Takieddine Kadiri and @Yolan Honoré-Rougé to streamline several cool use cases! including running Kedro as part of a long-running server KJuan Luis
02/29/2024, 3:42 PM%load_node
magic to load and debug nodes, compatible with IPython, Jupyter, VSCode, and Databricks
We’ve listened to lots of you talk about how central Notebooks📓 are for your Kedro workflow.
This release we’ve been hard at work making @Nok Lam Chan’s experiment posted a couple weeks ago on #user-research a reality! you can now prototype Kedro nodes faster in the notebook platform of your choice! We’ve tested this on 🧱 , vscode and jupyter so let us know what you think and how you’d like this feature to grow in the future.
To try it out, run pip install kedro[jupyter]
and follow our documentation.
Also:
• New MatlabDataset
which uses SciPy to load and save .mat
files
• Preview functionality for matplotlib, Plotly, and tracking datasets
• Allow additional parameters for sqlalchemy engine when using sql datasets.
🪲 Bug fixes and other changes
• kedro catalog resolve
now works with dataset factories that use PartitionedDataset
• Updated JSON Schema for Kedro 0.19
• Simplified installation of pandas
datasets on Windows and macOS
✍🏼 Documentation changes
• Improved and fixed documentation around the data collected by kedro-telemetry
https://docs.kedro.org/en/stable/configuration/telemetry.html
Read the complete release notes for kedro, kedro-datasets, and kedro-telemetry on GitHub
Thanks a lot to our community contributors @Manuel Spierenburg, @Felipe Monroy, @Samuel Lee SJ, @Fazil B. Topal, and MosaicMan
✨
We welcome every community contribution, large or small. See what we're working on now and report bugs or suggest future features.
Until next time,
The Kedro Team 💛Nero Okwa
03/04/2024, 10:19 PMkedro-datasets>=2.1
We've added two major new features: Publish and share Kedro-Viz on Azure and GCP. These features enable you to share your pipeline visualisation with other users, without the need for them to install Kedro-Viz. Detailed instructions for this feature are available in our documentation.
This version also includes other improvements such as new changes to the preview
feature, migration to Pydantic V2, and 🪲bug fixes outlined in the release notes. You can check out the demo.
🚀 What can you expect in this release?
• Publish and share Kedro-Viz on Azure and GCP.
• Introduced new changes to the preview
functionality on Kedro-Viz.
• Migrated to pydantic=2.0
.
• Dropped support for python=3.8
.
How do I get the latest release?
• Python: pip install kedro-viz==8.0.1
• React: npm install @quantumblack/kedro-viz@latest
You too can contribute by using these features, and providing feedback.
Follow our roadmap for more updates, comment on the tickets, and suggest future features to the community.
Until next time,
The Kedro Team
KKKKKRavi Kumar Pilla
04/02/2024, 2:18 PMpublish-kedro-viz
is a GitHub action that simplifies the process of deploying Kedro-Viz, which is a visual representation of your Kedro project, directly within the Git repository where your Kedro project is stored. By using this action, you can effortlessly showcase your Kedro-Viz on GitHub Pages. Please follow the docs for more information.
🚀 What can you expect in this release?
• This is an initial release of publish-kedro-viz action
• We have added an initial setup for the action
• Added a README to guide users with the setup of the action and its usage in their workflow
How do I get the latest release?
• GitHub Actions MarketPlace: https://github.com/marketplace/actions/publish-kedro-viz
Until next time,
The Kedro Team
KKKKKJuan Luis
04/04/2024, 5:56 AMJitendra Gundaniya
04/17/2024, 5:47 PMJSONDataset
preview and 🪲bug fixes outlined in the release notes. You can check out the demo.
🚀 What can you expect in this release?
• Extending stateful URLs with node filters and expand/collapse modular pipelines.
• Introduce --include-hooks
option and remove --ignore-plugins
from cli commands.
• Add Dataset Factory Patterns to Experiment Tracking.
• Add support for JSONDataset
preview.
• Upgrade to Node 18.
How do I get the latest release?
• Python: pip install kedro-viz==9.0.0
• React: npm install @quantumblack/kedro-viz@latest
This release includes a breaking change for users of our React components, due to our upgrade to Node 18.
You too can contribute by using these features, and providing feedback.
Follow our roadmap for more updates, comment on the tickets, and suggest future features to the community.
Until next time,
The Kedro Team
KKKKKElena Khaustova
04/18/2024, 12:21 PMnetcdf.NetCDFDataset
and ibis.TableDataset
• Added Python 3.12
support for kedro-datasets
• Kedro commands now work from any subdirectory within a Kedro project.
• Dropped the dependency on toposort
in favour of the built-in graphlib
module.
• Added the --telemetry
flag to kedro new
, allowing the user to register consent to have user analytics collected at the same time as the project is created.
• Improved the performance of Pipeline
object creation and summing.
🪲 Bug fixes and other changes
• Fixed bug with loading datasets from Hugging Face. Now allows passing parameters to the load_dataset function.
• Updated kedro pipeline create
and kedro pipeline delete
to read the base environment from the project settings.
• Updated CLI command kedro catalog resolve
to read credentials properly.
• Updated .gitignore
to prevent pushing Mlflow local runs folder to a remote forge when using mlflow and git.
• Fixed error handling message for malformed yaml/json files in OmegaConfigLoader.
• Fixed a bug in node
-creation allowing self-dependencies when using transcoding, that is datasets named like name@format
💥 Breaking changes to the API
• Methods _is_project
and _find_kedro_project
have been moved to kedro.utils
✍🏼 Documentation changes
• Added documentation on best practices for testing nodes and pipelines.
• Clarified docs around using custom resolvers without a full Kedro project.
Read the complete release notes for kedro and kedro-datasets on GitHub.
Thanks a lot to our community contributors @Ondrej Zacha, @Puneet Saini, @Riley Brady, Andrew Cao, @Eduardo Romero López and Jerome Asselin ✨
We welcome every community contribution, large or small. See what we're working on now and report bugs or suggest future features.
Until next time,
The Kedro Team 💛Juan Luis
04/23/2024, 12:26 PM