Merel
11/07/2023, 9:46 AM0.19.0
, which we’re aiming to get to you by the end of the year. There’s a lot happening behind the scenes and we wanted to give you a heads-up about a few changes that you can adopt now to be ready when 0.19.0
hits the streets.
1. We’ve written migration guides to help you transition from the legacy ConfigLoader
and TemplatedConfigLoader
which will be removed in Kedro 0.19.0
in favour of OmegaConfigLoader
2. Datasets move to their own package kedro-datasets and out of the Kedro core package in Kedro 0.19.0
. You can make the change to use that package now. Further, from version 2.0.0
of kedro-datasets
, which will be released together with Kedro 0.19.0
, all dataset names have changed to replace the capital letter “S” in “DataSet” with a lower case “s”. For example, CSVDataSet
is now CSVDataset
.
If you need help to update your projects in preparation of Kedro 0.19.0
or post release, please reach out to the team, we’re more than happy to help with this process.
We are very excited about the changes we’re bringing you in this release and more detailed communications about all changes will go out when the release is done. If you are curious, you can already have a look at the preliminary 0.19.0
release notes: https://github.com/kedro-org/kedro/blob/develop/RELEASE.md#upcoming-release-0190
Until next time,
The Kedro Team 💛Nero Okwa
11/17/2023, 5:18 PMpip install kedro-viz==6.7.0
• React: npm install @quantumblack/kedro-viz@latest
What's next?
• Publish and share Kedro-Viz from the command line.
• Upgrade to React 18.
kedroid hacktoberfest2023 Community contributions
Many thanks to our Hacktoberfest community contribution by Alain Anghelidi, for submitting a PR for this release 👏🏼
You too can contribute by using these features, and providing feedback.
Follow our roadmap for more updates, comment on the tickets, and suggest future features to the community.
Until next time,
The Kedro Team
KKdatajoely
11/22/2023, 12:34 PM0.10.0
of kedro-vertexai
plugin is here! 🎉
I'm kinda new here so I'm not sure if this belongs to #announcements also, as it's a plugin update.
We added the grouping feature to allow more freedom in shaping your pipelines at vertexai. More details in the docs:
https://kedro-vertexai.readthedocs.io/en/0.10.0/source/02_installation/02_configuration.html#example
https://github.com/getindata/kedro-vertexai/releases/tag/0.10.0
Slack conversationJuan Luis
11/29/2023, 11:44 AMOmegaConfigLoader
, how to use S3 and MinIO for your filepaths, and how to deploy Kedro to a Docker image or Airflow.
we hope you enjoy it! K
(also if you can spread the word in our socials, that would be fantastic! https://www.linkedin.com/posts/kedro-python_introduction-to-kedro-building-maintainab[…]384343457793-0uB7?utm_source=share&utm_medium=member_desktop)Juan Luis
12/14/2023, 12:57 PM0.19
is available! K
> TLDR: Kedro 0.19 is a new major release that contains some breaking changes with respect to 0.18.x. Among other things, we introduced project tools that help you further customise the result of kedro new
, added support for soft merge in OmegaConfigLoader
, improved the error messages when datasets are not found, and modernised the default project templates by having the build configuration and project metadata in pyproject.toml
. Datasets are gone (you must install kedro-datasets
instead), alternative config loaders are gone too (only OmegaConfigLoader
remains) and some deprecated CLI commands were removed. To help you upgrade, we wrote a migration guide.
*Y*ou can install Kedro 0.19 with:
pip install kedro==0.19.1
or
conda/mamba/micromamba install -c conda-forge kedro=0.19.1
(Yes, there's also 0.19.0, but we detected a problematic bug with it so we went ahead and released 0.19.1 with a fix immediately after)
You can find the release highlights below, for the complete list of changes go to the release notes https://github.com/kedro-org/kedro/blob/main/RELEASE.md#release-0191 and https://github.com/kedro-org/kedro/blob/main/RELEASE.md#release-0190
🚀 Major Features and improvements
• Dropped Python 3.7 support.
• Introduced project tools and example to the `kedro new` CLI flow.
• The new spaceflights starters, spaceflights-pandas
, spaceflights-pandas-viz
, spaceflights-pyspark
, and spaceflights-pyspark-viz
can be used with the kedro new
command with the --starter
flag.
• Added the --conf-source
option to %reload_kedro
, allowing users to specify a source for project configuration.
• Added the functionality to choose a merging strategy for config files loaded with `OmegaConfigLoader`.
• Modified the mechanism of importing datasets, raise more explicit error when dependencies are missing.
• Added validation for configuration file used to override run commands via the CLI.
• Moved the default environment base
and local
from config loader to _ProjectSettings
. This enables the use of config loader as a standalone class without affecting existing Kedro Framework users.
🪲 Bug fixes and other changes
• Added a new field tools
to pyproject.toml
when a project is created.
• Reduced spaceflights
data to minimise waiting times during tutorial execution.
• Added validation to node tags to be consistent with node names.
• Removed pip-tools
as a dependency.
• Accepted path-like filepaths more broadly for datasets.
• Loosened pin for kedro-telemtry
to fix dependency issues in 0.19.0
.
💥 Breaking changes
• Removed ConfigLoader
and TemplatedConfigLoader
.
• Removed kedro.extras.datasets
and tests (use kedro-datasets
instead)
• Removed PartitionedDataset
and IncrementalDataset
from <http://kedro.io|kedro.io>
(import them from kedro-datasets
instead)
• logging
is removed from OmegaConfigLoader
in favour of the environment variable KEDRO_LOGGING_CONFIG
.
• Removed support for defining the layer
attribute at top-level within DataCatalog.
• Renamed data_set
and DataSet
to dataset
and Dataset
everywhere.
• Removed the create_default_data_set()
method in the Runner
in favour of using dataset factories to create default dataset instances.
• The default project template now has only one pyproject.toml
at the root of the project (containing both the packaging metadata and the Kedro build config).
✍️ Documentation changes
• Added new top navigation to easily switch between Framework, Viz, and Datasets.
• Added new search-as-you-type to improve the search experience.
kedroid Community contributions
A million thanks to Jeroldine Akuye Oakley, @Laiza Milena Scheid Parizotto, @Mustapha Abdullahi, @Adam Kells, Ajay Gonepuri for submitting PRs to this release!
🚨 If you are upgrading from Kedro 0.18, have a look at the migration guide.
We welcome every community contribution, large or small. See what we're working on now and report bugs or suggest future features.
Until next time,
The Kedro Team 💛Nero Okwa
12/19/2023, 6:10 PMkedro viz run
, and added several improvements and bug fixes in this version. You can find the full details in the release notes and check out the demo.
🚀 What can you expect in this release?
• Publish and share Kedro-Viz on AWS from the command line using the command kedro viz deploy
.
• Changed the CLI command to run Kedro-viz to`kedro viz run`.
• Upgraded to React 18.
• Dropped support for python=3.7
.
• Dropped support for kedro==0.17.x
.
How do I get the latest release?
• Python: pip install kedro-viz==7.0
• React: npm install @quantumblack/kedro-viz@latest
You too can contribute by using these features, and providing feedback.
Follow our roadmap for more updates, comment on the tickets, and suggest future features to the community.
Until next time,
The Kedro Team
KKJuan Luis
12/21/2023, 12:00 PMJuan Luis
01/08/2024, 11:04 AMJuan Luis
01/23/2024, 10:18 AMtools
• Allowed modern versions of JupyterLab and Jupyter Notebooks
• Removed setuptools dependency
• Added source_dir
explicitly in pyproject.toml
for non-src layout project
• MemoryDataset
entries are now included in free outputs
• Removed black dependency and replaced its functionality with ruff format
• Added logging about not using async mode in SequentialRunner
and ParallelRunner
💥 (Minor) Breaking changes to the API
(These are minor breaking changes that we have decided to include early to avoid disrupting user workflows later on in the cycle)
• Changed input format for tools option obtained from --config
file from numbers to short names.
✍🏼 Documentation changes
• Added documentation about bootstrap_project
and configure_project
.
• Added documentation about kedro run
and hook execution order.
Read the complete release notes on https://github.com/kedro-org/kedro/blob/main/RELEASE.md#release-0192 We welcome every community contribution, large or small. See what we're working on now and report bugs or suggest future features.
Until next time,
The Kedro Team 💛Nero Okwa
01/23/2024, 10:37 AMkedro viz build
CLI command, enabling you to generate a Kedro-Viz instance using your Kedro project data. This makes it easy to share Kedro-Viz on any static hosting platform. Detailed instructions for this feature are available in our documentation.
This version also includes other improvements and bug fixes outlined in the release notes. You can check out the demo.
🚀 What can you expect in this release?
• Added the build CLI command kedro viz build
to enable users to create a build directory of Kedro-Viz instance with Kedro project data.
How do I get the latest release?
• Python: pip install kedro-viz==7.1.0
• React: npm install @quantumblack/kedro-viz@latest
You too can contribute by using these features, and providing feedback.
Follow our roadmap for more updates, comment on the tickets, and suggest future features to the community.
Until next time,
The Kedro Team 💛
KKJuan Luis
01/31/2024, 12:59 PMJuan Luis
02/16/2024, 11:54 AMkedro-boot
, a new plugin created by @Takieddine Kadiri and @Yolan Honoré-Rougé to streamline several cool use cases! including running Kedro as part of a long-running server KJuan Luis
02/29/2024, 3:42 PM%load_node
magic to load and debug nodes, compatible with IPython, Jupyter, VSCode, and Databricks
We’ve listened to lots of you talk about how central Notebooks📓 are for your Kedro workflow.
This release we’ve been hard at work making @Nok Lam Chan’s experiment posted a couple weeks ago on #user-research a reality! you can now prototype Kedro nodes faster in the notebook platform of your choice! We’ve tested this on 🧱 , vscode and jupyter so let us know what you think and how you’d like this feature to grow in the future.
To try it out, run pip install kedro[jupyter]
and follow our documentation.
Also:
• New MatlabDataset
which uses SciPy to load and save .mat
files
• Preview functionality for matplotlib, Plotly, and tracking datasets
• Allow additional parameters for sqlalchemy engine when using sql datasets.
🪲 Bug fixes and other changes
• kedro catalog resolve
now works with dataset factories that use PartitionedDataset
• Updated JSON Schema for Kedro 0.19
• Simplified installation of pandas
datasets on Windows and macOS
✍🏼 Documentation changes
• Improved and fixed documentation around the data collected by kedro-telemetry
https://docs.kedro.org/en/stable/configuration/telemetry.html
Read the complete release notes for kedro, kedro-datasets, and kedro-telemetry on GitHub
Thanks a lot to our community contributors @Manuel Spierenburg, @Felipe Monroy, @Samuel Lee SJ, @Fazil B. Topal, and MosaicMan
✨
We welcome every community contribution, large or small. See what we're working on now and report bugs or suggest future features.
Until next time,
The Kedro Team 💛Nero Okwa
03/04/2024, 10:19 PMkedro-datasets>=2.1
We've added two major new features: Publish and share Kedro-Viz on Azure and GCP. These features enable you to share your pipeline visualisation with other users, without the need for them to install Kedro-Viz. Detailed instructions for this feature are available in our documentation.
This version also includes other improvements such as new changes to the preview
feature, migration to Pydantic V2, and 🪲bug fixes outlined in the release notes. You can check out the demo.
🚀 What can you expect in this release?
• Publish and share Kedro-Viz on Azure and GCP.
• Introduced new changes to the preview
functionality on Kedro-Viz.
• Migrated to pydantic=2.0
.
• Dropped support for python=3.8
.
How do I get the latest release?
• Python: pip install kedro-viz==8.0.1
• React: npm install @quantumblack/kedro-viz@latest
You too can contribute by using these features, and providing feedback.
Follow our roadmap for more updates, comment on the tickets, and suggest future features to the community.
Until next time,
The Kedro Team
KKKKKRavi Kumar Pilla
04/02/2024, 2:18 PMpublish-kedro-viz
is a GitHub action that simplifies the process of deploying Kedro-Viz, which is a visual representation of your Kedro project, directly within the Git repository where your Kedro project is stored. By using this action, you can effortlessly showcase your Kedro-Viz on GitHub Pages. Please follow the docs for more information.
🚀 What can you expect in this release?
• This is an initial release of publish-kedro-viz action
• We have added an initial setup for the action
• Added a README to guide users with the setup of the action and its usage in their workflow
How do I get the latest release?
• GitHub Actions MarketPlace: https://github.com/marketplace/actions/publish-kedro-viz
Until next time,
The Kedro Team
KKKKKJuan Luis
04/04/2024, 5:56 AMJitendra Gundaniya
04/17/2024, 5:47 PMJSONDataset
preview and 🪲bug fixes outlined in the release notes. You can check out the demo.
🚀 What can you expect in this release?
• Extending stateful URLs with node filters and expand/collapse modular pipelines.
• Introduce --include-hooks
option and remove --ignore-plugins
from cli commands.
• Add Dataset Factory Patterns to Experiment Tracking.
• Add support for JSONDataset
preview.
• Upgrade to Node 18.
How do I get the latest release?
• Python: pip install kedro-viz==9.0.0
• React: npm install @quantumblack/kedro-viz@latest
This release includes a breaking change for users of our React components, due to our upgrade to Node 18.
You too can contribute by using these features, and providing feedback.
Follow our roadmap for more updates, comment on the tickets, and suggest future features to the community.
Until next time,
The Kedro Team
KKKKKElena Khaustova
04/18/2024, 12:21 PMnetcdf.NetCDFDataset
and ibis.TableDataset
• Added Python 3.12
support for kedro-datasets
• Kedro commands now work from any subdirectory within a Kedro project.
• Dropped the dependency on toposort
in favour of the built-in graphlib
module.
• Added the --telemetry
flag to kedro new
, allowing the user to register consent to have user analytics collected at the same time as the project is created.
• Improved the performance of Pipeline
object creation and summing.
🪲 Bug fixes and other changes
• Fixed bug with loading datasets from Hugging Face. Now allows passing parameters to the load_dataset function.
• Updated kedro pipeline create
and kedro pipeline delete
to read the base environment from the project settings.
• Updated CLI command kedro catalog resolve
to read credentials properly.
• Updated .gitignore
to prevent pushing Mlflow local runs folder to a remote forge when using mlflow and git.
• Fixed error handling message for malformed yaml/json files in OmegaConfigLoader.
• Fixed a bug in node
-creation allowing self-dependencies when using transcoding, that is datasets named like name@format
💥 Breaking changes to the API
• Methods _is_project
and _find_kedro_project
have been moved to kedro.utils
✍🏼 Documentation changes
• Added documentation on best practices for testing nodes and pipelines.
• Clarified docs around using custom resolvers without a full Kedro project.
Read the complete release notes for kedro and kedro-datasets on GitHub.
Thanks a lot to our community contributors @Ondrej Zacha, @Puneet Saini, @Riley Brady, Andrew Cao, @Eduardo Romero López and Jerome Asselin ✨
We welcome every community contribution, large or small. See what we're working on now and report bugs or suggest future features.
Until next time,
The Kedro Team 💛Juan Luis
04/23/2024, 12:26 PMNok Lam Chan
05/09/2024, 3:06 PMDmitry Sorokin
06/04/2024, 11:07 AMraise_errors
argument to find_pipelines
. If True
, the first pipeline for which autodiscovery fails will cause an error to be raised. The default behaviour is still to raise a warning for each failing pipeline.
• It is now possible to use Kedro without having rich
installed.
• Updated custom logging behavior: conf/logging.yml
will be used if it exists and KEDRO_LOGGING_CONFIG
is not set; otherwise, default_logging.yml
will be used.
• Updated the kedro-telemetry
to generate a unique UUID for each user and added support for python 3.12.
🪲 Bug fixes and other changes
• User defined catch-all dataset factory patterns now override the default pattern provided by the runner.
• Removed arbitrary upper bound for s3fs
in kedro-datasets
.
• NetCDFDataset
support for NetCDF4 via engine="netcdf4"
and engine="h5netcdf"
💥 Upcoming deprecations for Kedro 0.20.0
• All micro-packaging commands (kedro micropkg pull
, kedro micropkg package
) are deprecated and will be removed in Kedro 0.20.0. If you are an active user and are impacted by this deprecation, please leave a comment here explaining your use case.
✍🏼 Documentation changes
• Improved documentation for custom starters
• Added a new docs section on deploying Kedro project on AWS Airflow MWAA
• Detailed instructions on using globals
and runtime_params
with the OmegaConfigLoader
Read the complete release notes for kedro, kedro-datasets, and kedro-telemetry on GitHub
Thanks a lot to our community contributors @Dominik Zalewski, cleeks and @Charles Guan ✨
We welcome every community contribution, large or small. See what we're working on now and report bugs or suggest future features.
Until next time,
The Kedro Team 💛Huong Nguyen
06/04/2024, 12:06 PMpip install kedro-viz==9.1.0
• React: npm install @quantumblack/kedro-viz@latest
You too can contribute by using these features, and providing feedback.
Follow our roadmap for more updates, comment on the tickets, and suggest future features to the community.
Until next time,
The Kedro TeamJuan Luis
06/04/2024, 2:02 PMJuan Luis
06/04/2024, 4:34 PMJuan Luis
06/06/2024, 10:00 AMIn this post, I'll share how I'm using Kedro to manage Machine Learning (ML) pipelines while efficiently storing and executing SQL queries within my Python Kedro project. By integrating Ibis via the Kedro Ibis table dataset, I'm able to keep all my SQL code within the Kedro project and execute it on the database side, specifically using Google BigQuery.
Nok Lam Chan
07/05/2024, 2:42 PM0.1.0
You can now find a little Kedro K icon in the bottom of your VSCode.
Highlight:
• Enhance workflow to switch configure environment (via settings/commands/UI icon)
• Go to definition
now resolves environment properly
Other changes:
- Expanded pipeline discovery to support *pipeline*.py
patterns and improved handling of nested subdirectories.
- Add new extension setting Kedro: Environment
to change the configuration environment.
- Add new command kedro: Select Environment
to change the configuration environment.
- Go to Definition
now search default_run_env
first.
- Add Kedro Icon and Status bar to select environment.
- Fix a bug that cause server panic when config is empty.
• Hover
config preserve the original order (Thanks @David Stanley)Juan Luis
07/19/2024, 5:25 PMkedro-databricks
, a new plugin by @Jens Peder Meldgaard! K🧱