Hi all, question on the best way to set up a simpl...
# questions
z
Hi all, question on the best way to set up a simple one-node pipeline that accomplishes the following: • The node's main function in the generate some output from a GenAI model (e.g. calls Stable Diffusion or DALL-E) • The inputs to the node are: ◦ A list of prompts stored in the global parameters yml that could change from execution to execution (
prompts: list[str]
) ◦ An integer for the number times to call one prompt, also stored in the global parameters yml (
variations: int
) • Ideally the output of this node is
n_prompts x m_variation
images, knowing that this could be dynamic. An example catalog entry might be
prompt_1_var_1.jpg
I feel like I knew how to do this in the past with Jinja2 configuration, but I know that's not the latest and greatest and I am just now dipping my toes back in to Kedro after a year or so away
m
Just write for loop in your node to have
n_prompts x m_variations
and use
PartitionedDataset
for saving it. https://docs.kedro.org/en/stable/data/partitioned_and_incremental_datasets.html#partitioned-dataset-save
z
This looks like a fantastic option, simple and elegant. Thanks for the response!