https://kedro.org/ logo
#questions
Title
# questions
m

Mark Druffel

02/14/2024, 8:49 PM
This is a very open ended question, but I'm having trouble finding any reference to someone using nbdev to develop a kedro pipeline. All I can seem to dig up is this repo from @Nok Lam Chan, but I think it was for the kedro repo not a pipeline.... 🤷 Does anyone know of a project where that's been done or have any advice on the subject? Brand new to nbdev so I'm just trying to figure out if there are any good templates out there to help me get started w/ nbdev & kedro
d

datajoely

02/14/2024, 10:41 PM
I’m not sure I’ve seen anyone do this - it’s an interesting idea certainly
m

Mark Druffel

02/14/2024, 10:51 PM
I'm still new to kedro so maybe I'll feel differently after some time, but I find the need to use
kedro ipython
or
kedro jupyter
a lot to write my code anyways because I can't think through what all I want to do in a node without "playing with" the data... feels like embracing the nbdev approach might streamline things for me but maybe I'm overthinking it 🤷 I'm gonna see if I can figure out how to work it into my kedro workflow and will share details if I have any success
d

datajoely

02/15/2024, 10:12 AM
@Nok Lam Chan is currently prototyping this https://kedro-org.slack.com/archives/C03R8N2M8KT/p1705422918899009 Would it be useful in your workflow?
🔥 1
n

Nok Lam Chan

02/15/2024, 1:57 PM
@Mark Druffel I view developing a pipeline mostly "one node at a time", so developing in notebook or the new feature would help. I am a fan of nbdev, if you have suggestions how we can adopt similar approaches i would love to hear more
m

Mark Druffel

02/15/2024, 10:57 PM
@datajoely
%load_node
would be very useful. I would definitely use if for debugging and making changes to nodes. That said, it does present a chicken / egg problem & still requires copy / paste. 1. Chicken / egg -
%load_node
only works if you have a node. Maybe
%new_node
could be used to address that?
%new_node <input_node(s)> <input_params(s)>
🤔 2. Copy / paste - any changes would have to be copy / pasted from the notebook to the node. This is a big part of what nbdev tries to solve so potentially that's a good fit to resolve that challenge 🤷 @Nok Lam Chan I'm new to nbdev so I doubt I'll be able to suggest anything you couldn't think through yourself, but I just added it to my repo and am testing it out. I'll share any relevant findings.
Also thank you both for all the helpful feedback. I'm very new to kedro and am really loving / impressed by the community!
❤️ 2
d

datajoely

02/16/2024, 9:57 AM
🙏 appreciate the feedback, any crazy ideas or suggestions please let us know