Ofir
01/28/2023, 6:26 PMDeepyaman Datta
01/29/2023, 8:01 PMShould Kedro tasks (and pipelines) be thin wrappers that import my existing Python code, or not? what are the best practices if you already have an existing code base and Git repository with your code?Sure, you can do that, if you have an existing package you are publishing/can import. Else, you can use Kedro as a way of organizing that existing codebase.
While I get the concept of Kedro project and having a workspace per data model, I don’t get how do I sync the code across projects/workspaces/experiments.I don't totally understand your question here. Maybe this will help, else you can clarify: Kedro has a separation between logic (
src
directory with pipelines) and configuration (conf
directory, with specifics around the data loading + parameters). You can define a reusable pipeline (e.g. for classification) as a Kedro project, package it (will only package logic), and deploy in different contexts with different configuration. Alternatively, you can pull in reusable pipelines into different Kedro projects.Ofir
01/30/2023, 9:01 AMJo Stichbury
01/30/2023, 9:42 AMOfir
01/30/2023, 9:48 AMJo Stichbury
01/30/2023, 9:53 AMJuan Luis
01/30/2023, 9:56 AMOfir
01/30/2023, 9:57 AMJuan Luis
01/30/2023, 10:05 AMOfir
01/30/2023, 10:07 AMJuan Luis
01/30/2023, 10:14 AMBrandon Meek
01/30/2023, 3:47 PMrequirements.txt
file (so you can rebuild your environment for reproducibility) whenever I create a pipeline that I'll use across projects I use micro-packaging and set the destination to my starter, so when updates are made to that pipeline it won't affect projects that have already been created and are using that pipeline unless I update it. With this process there's no copy/pasting, and I don't run into issues where something was updated and it breaks my projectDeepyaman Datta
01/30/2023, 4:06 PMOfir
01/30/2023, 4:11 PM