Is it possible to invoke a kedro pipeline with par...
# questionsl
Luke Dziurzynski
08/30/2023, 3:38 PMIs it possible to invoke a kedro pipeline with parameters, like string parameters, from other python libraries? I'm wondering if I could invoke a kedro pipeline from my discord bot, where the user could specify some text parameters and send that to be transformed by the pipeline
🔥 1
d
datajoely
08/30/2023, 3:49 PMI will get back to you on this, the short answer is yes - but it takes a little of work
datajoely
08/30/2023, 3:50 PMin the future we want to make this much much easier
datajoely
08/30/2023, 3:51 PMthe one issue here is that Kedro was designed with Reproducibility in mind so ephemeral pipelines like this is a slightly different pattern
👍 1
datajoely
08/30/2023, 3:51 PMbut I love the idea of integrating Kedro into a midjourney like workflow 🔥
💯 1
l
Luke Dziurzynski
08/30/2023, 3:54 PMNice!! yeah, I tried to sketch out a couple diagrams of the basic flow (below). Definitely let me know if this is possible. I would like to start building these interactions in a more modular way because they are about to get a bit more complicated
Luke Dziurzynski
08/30/2023, 3:55 PMMidjourney-like workflow is exactly what we're building
Luke Dziurzynski
08/30/2023, 3:56 PMOr alternatively, could I run a kedro pipeline with a custom dataset as an input? For context, we have a discord server where players can enter into a march-madness type of battle, where each player enters in a text description of a hero, which then gets transformed by genAI into a structured hero object with hitpoints. Then each hero competes with each other hero until there is one winner
Luke Dziurzynski
08/30/2023, 3:56 PMLuke Dziurzynski
08/30/2023, 3:56 PM❤️ 1
🔥 1
⚔️ 1
Luke Dziurzynski
08/30/2023, 4:03 PMIf there is something that's simple/makes more sense for ephemeral pipelines, would be down to test out your recommendation(s).
n
Nok Lam Chan
08/30/2023, 4:15 PMCustom dataset may work - but i think the easiest way to do this is via
KedroSessionmy_pipeline(params=user_params)🙌 1
w
William Caicedo
08/30/2023, 4:17 PMI’ve ran a Kedro pipeline from inside a Slackbot before. Essentially I had the bot source file located inside my Kedro project and launched the pipeline upon interaction with the bot, something like this:
Copy code
metadata = bootstrap_project(Path.cwd())
data_date = get_some_date()
with KedroSession.create(metadata.package_name,
extra_params= {'param_1': param_1,
'param_2': param_2,
'data_date': data_date
}) as session:
pipeline_name = 'one' if some_condition else 'two'
session.run(pipeline_name=pipeline_name)🆒 1
🙌 1
👍 1
n
Nok Lam Chan
08/30/2023, 4:19 PMThis should work if the inputs are just parameters. If it’s some payload then you may need custom dataset
👍 1
l
Luke Dziurzynski
08/30/2023, 5:20 PMHow might I inject a custom dataset while using a kedro session?
Luke Dziurzynski
08/30/2023, 5:22 PMCould I pass in database tablenames as params and then read from those tables during my kedro pipeline?
n
Nok Lam Chan
08/30/2023, 5:58 PMCould I pass in database tablenames as params and then read from those tables during my kedro pipeline? (edited)The above approach to inject params should work
👍 1
3 Views