https://kedro.org/ logo
Powered by
#questions
Title
# questions
l

Luke Dziurzynski

08/30/2023, 3:38 PM
Is it possible to invoke a kedro pipeline with parameters, like string parameters, from other python libraries? I'm wondering if I could invoke a kedro pipeline from my discord bot, where the user could specify some text parameters and send that to be transformed by the pipeline
🔥 1
d

datajoely

08/30/2023, 3:49 PM
I will get back to you on this, the short answer is yes - but it takes a little of work
in the future we want to make this much much easier
the one issue here is that Kedro was designed with Reproducibility in mind so ephemeral pipelines like this is a slightly different pattern
👍 1
but I love the idea of integrating Kedro into a midjourney like workflow 🔥
💯 1
l

Luke Dziurzynski

08/30/2023, 3:54 PM
Nice!! yeah, I tried to sketch out a couple diagrams of the basic flow (below). Definitely let me know if this is possible. I would like to start building these interactions in a more modular way because they are about to get a bit more complicated
Midjourney-like workflow is exactly what we're building
Or alternatively, could I run a kedro pipeline with a custom dataset as an input? For context, we have a discord server where players can enter into a march-madness type of battle, where each player enters in a text description of a hero, which then gets transformed by genAI into a structured hero object with hitpoints. Then each hero competes with each other hero until there is one winner
Screenshot 2023-08-30 at 8.46.43 AM.png
Screenshot 2023-08-30 at 8.52.28 AM.png
❤️ 1
🔥 1
⚔️ 1
If there is something that's simple/makes more sense for ephemeral pipelines, would be down to test out your recommendation(s).
n

Nok Lam Chan

08/30/2023, 4:15 PM
Custom dataset may work - but i think the easiest way to do this is via 
KedroSession
, essentially you want to run the pipeline like a python function and just take the parameters. i.e. 
my_pipeline(params=user_params)
You may have to write a thin wrapper around this to control what can be passed.
🙌 1
w

William Caicedo

08/30/2023, 4:17 PM
I’ve ran a Kedro pipeline from inside a Slackbot before. Essentially I had the bot source file located inside my Kedro project and launched the pipeline upon interaction with the bot, something like this:
Copy code
metadata = bootstrap_project(Path.cwd())
    data_date = get_some_date()
    with KedroSession.create(metadata.package_name, 
                             extra_params= {'param_1': param_1,
                                            'param_2': param_2, 
                                            'data_date': data_date
                                            }) as session:
        pipeline_name = 'one' if some_condition else 'two'
        session.run(pipeline_name=pipeline_name)
🆒 1
🙌 1
👍 1
n

Nok Lam Chan

08/30/2023, 4:19 PM
This should work if the inputs are just parameters. If it’s some payload then you may need custom dataset
👍 1
l

Luke Dziurzynski

08/30/2023, 5:20 PM
How might I inject a custom dataset while using a kedro session?
Could I pass in database tablenames as params and then read from those tables during my kedro pipeline?
n

Nok Lam Chan

08/30/2023, 5:58 PM
Could I pass in database tablenames as params and then read from those tables during my kedro pipeline? (edited)
The above approach to inject params should work
👍 1
2 Views
Powered by