Hey what is the right pattern for nodes submitting...
# questions
t
Hey what is the right pattern for nodes submitting API requests? For example using openai credentials? Right now i am thinking to set credentials in environment variables, then inject these to parameters using omegaconf so nodes can receive them as inputs. Any other (recommended) approaches here?
d
I don’t think it’s been released yet - but we have a langchain dataset contributed by @Ian Whalen https://github.com/kedro-org/kedro-plugins/blob/main/kedro-datasets/kedro_datasets_experimental/langchain/_openai.py
👍 1
🥳 1
i
That's one way if you want to use
credentials.yml
for your keys Another approach would just be to use a dotenv hook so they get set at runtime. I've been liking that approach too. So your
settings.py
would look like:
Copy code
import logging

from dotenv import load_dotenv
from kedro.framework.hooks import hook_impl

logger = logging.getLogger(__name__)

class DotenvHook:
    @hook_impl
    def after_context_created(self, *args, **kwargs) -> None:
        loaded = load_dotenv()

        if loaded:
            logger.info("Dotenv loaded successfully.")

        else:
            logger.warning("Dotenv not found!")


HOOKS = (DotenvHook(),)

... other settings things ...
Then you just dump in your openai keys to your
.env
Not sure which one is better! Using
credentials.yml
is probably more kedronic (?) and if you're using LangChain those datasets could be helpful
n
I would advise adding a
tryfirst
argument to the hook just to make sure this get executed before anything. https://docs.kedro.org/en/stable/hooks/introduction.html#hook-execution-order And also curious if it makes sense for Kedro to support
dotenv
natively? It's actually 1 line of code because we use
dynaconf
and it integrate dotenv natively (technically
load_dot_env
) is also just one line of code though.
👍 1
i
I was wondering about something like this too Nok! Maybe we can move this discussion to github? I felt like I might be missing something with how the team wants people to use
conf/env
's and
credentials.yml
I wasn't sure if
dotenv
clashed with this
n
i actually have a discussion open to remove
dynaconf
https://github.com/kedro-org/kedro/discussions/3877, feel free to open a discussion to talk about how credentials should works with env var etc.
👍 1