Hey what is the right pattern for nodes submitting API requests? For example using openai credentials? Right now i am thinking to set credentials in environment variables, then inject these to parameters using omegaconf so nodes can receive them as inputs. Any other (recommended) approaches here?

I don’t think it’s been released yet - but we have a langchain dataset contributed by @Ian Whalen https://github.com/kedro-org/kedro-plugins/blob/main/kedro-datasets/kedro_datasets_experimental/langchain/_openai.py

That's one way if you want to use

credentials.yml

for your keys Another approach would just be to use a dotenv hook so they get set at runtime. I've been liking that approach too. So your

settings.py

would look like:

import logging

from dotenv import load_dotenv
from kedro.framework.hooks import hook_impl

logger = logging.getLogger(__name__)

class DotenvHook:
    @hook_impl
    def after_context_created(self, *args, **kwargs) -> None:
        loaded = load_dotenv()

        if loaded:
            logger.info("Dotenv loaded successfully.")

        else:
            logger.warning("Dotenv not found!")


HOOKS = (DotenvHook(),)

... other settings things ...

Then you just dump in your openai keys to your

.env

Not sure which one is better! Using

credentials.yml

is probably more kedronic (?) and if you're using LangChain those datasets could be helpful

I would advise adding a

tryfirst

argument to the hook just to make sure this get executed before anything. https://docs.kedro.org/en/stable/hooks/introduction.html#hook-execution-order And also curious if it makes sense for Kedro to support

dotenv

natively? It's actually 1 line of code because we use

dynaconf

and it integrate dotenv natively (technically

load_dot_env

) is also just one line of code though.

I was wondering about something like this too Nok! Maybe we can move this discussion to github? I felt like I might be missing something with how the team wants people to use

conf/env

's and

credentials.yml

I wasn't sure if

dotenv

clashed with this

i actually have a discussion open to remove

dynaconf

https://github.com/kedro-org/kedro/discussions/3877, feel free to open a discussion to talk about how credentials should works with env var etc.