Brandon Meek
03/09/2023, 5:55 PMDeepyaman Datta
03/09/2023, 5:58 PMBrandon Meek
03/09/2023, 6:02 PMAbstractDataSet
, I have a library that implements several datasets for my employer's proprietary data tools and I'd love to be able to share it with my colleagues, including the ones that I haven't convinced to start using Kedro, so they could use the code API, but there's a lot of overhead with a full install of Kedro that wouldn't be necessary for using just the AbstractDataSet
Deepyaman Datta
03/09/2023, 6:30 PMBrandon Meek
03/09/2023, 6:31 PMNok Lam Chan
03/09/2023, 11:24 PMBrandon Meek
03/09/2023, 11:50 PMMerel
03/10/2023, 9:27 AMAbstractDataSet
out of core Kedro as part of moving all dataset implementations into itβs own repo (kedro-datasets
). However, we decided against it. You can read the discussion back here: https://github.com/kedro-org/kedro/issues/1776#issuecomment-1234432081Nok Lam Chan
03/10/2023, 1:22 PMAbstractDataSet
will be too much since things like _release
from_config
is not useful without other component like a kedro run or DataCatalog
Matthias Roels
03/10/2023, 1:35 PMBrandon Meek
03/10/2023, 1:39 PMNok Lam Chan
03/10/2023, 1:50 PMBrandon Meek
03/10/2023, 2:22 PMDeepyaman Datta
03/10/2023, 4:05 PMInstead, we should see Kedro as an ecosystem of core components (e.g. data catalog, datasets, modular pipelines), plus an opinionated (relatively lightweight) wrapper that stitches these all together and includes best practices. However, a power user can use each of these components or a combination thereof with their own lightweight stitching, if they don't like the way Kedro does it.But, of course, there's effort and alignment required on that, since I'm pretty sure there are arguments in the other direction too. π