I want to share a truly random experience with everyone: I'm facing the problems that Kedro solves, which is 5k lines of ETL/DS python in a single file that needs to be broken down. It's hard to adopt Kedro because of the perceived complexity and boilerplate which requires buy-in wholesale (I know we can adopt library components individually but it's still not straightforward), so I was trying to simplify the framework's interface. Got stuck, asked chatgpt and 100 exchanges later, we went full circle to the current API as the most optimal one. This is ChatGPT's opinion of the API:

Super interesting, I also played with chatgpt on API design a while ago and I get similar result. What solution did you end up go with? We are trying to remove some of the boilerplate to make it as lean as possible (or at least easier to work with if you have your own template already). I am curious what issues did you bump into when you try to use Kedro library components only)

@Lim H. Is it not blasphemy using ChatGPT for this specific problem? 👀 🤣

this is both hilarious and reassuring, thanks a lot for sharing @Lim H. 😄 still, any way you think we can lower the perceived complexity or make adopting individual library components more easy? we're actively working on these areas, so even if ChatGPT blessed our API ✨ , we can keep that design while flattening the learning curve

@datajoely blasphemy to which God? 😄 @Juan Luis @Nok Lam Chan my main problem is I don't know where to start dropping Kedro into an existing project of reasonable complexity without deep refactoring and this is an issue. Say I want to adopt the catalog. To do it quickly I would use the Python API. But then the catalog definition is in the code as dictionary with ~50 datasets and that's the same problem as before. After that, to be idiomatic, I will then have to convert it back to YAML at some point. Also where can I start with parameters refactoring?

One of the reflections from one of the early devs was that if dataclasses existed when they made it we would have gone down that route. I wonder if we should do a spike of what a pydantic style catalog looks like? I’m currently on a pydantic / pandera spree for everything so it’s very front of mind

One thing I hacked together yesterday was allowing people to define a.

pipeline.py

with a bunch of

func_a(a: DataCatalog.A) -> DataCatalog.B
func_b(a: DataCatalog.B) -> DataCatalog.C

and it automatically works out this is a pipeline of

A -> func_a -> B -> func_b

the main problem is it lost the pure function nature of the node but I guess you can just do that under the hood

Perhaps there is a model where you do a little prototype and then the team takes it over?

I'm still not entirely convinced this is any better than the current API, hence the conversation with chatgpt lol.

I love the idea of being able to have a two-way conversion between YAML and Python classes (YAML and Python dictionary is already trivial) and it's something we could definitely explore

Two way conversion of YAML and JSON serialisation of Pipeline definitions are my current wishlist 🙏

I'd like to note one thing: > Say I want to adopt the catalog. To do it quickly I would use the Python API. But then the catalog definition is in the code as dictionary with ~50 datasets and that's the same problem as before. > > After that, to be idiomatic, I will then have to convert it back to YAML at some point. what would you think about writing the catalog in YAML directly and loading it in Python using

catalog = DataCatalog.from_config(OmegaConfigLoader.load_file("catalog.yml"))

without having to use the Kedro project template at all @Lim H.? (context and discussion: https://github.com/kedro-org/kedro/issues/2819)

My ideas on designing some of those classes were to not be far from the moment, where the

ConfigLoader

can be replaced with a simple dictionary object, the

DataCatalog

with a dictionary-like object and then everything should work