Hi everyone! I wanted first to thank all the participants of the recent Kedro documentation survey thankyou and second, to apologise 🙇‍♀️. We are still waiting for our rebranded Kedro merch, so we have yet to send anything out to those who responded, but I am following up on that. 👕 K kedroid Can I ask a follow up question to everyone on this channel? It's still about docs, specifically the Spaceflights tutorial. We received feedback that you'd like more examples of how to extend Spaceflights for other, more advanced, scenarios, e.g. to add S3 as a filestore, deployment options, etc. We wouldn't extend the starter but we would add extra example code and how-to sections in the docs. The question is: what do you think we should add to extend Spaceflights for common tasks and scenarios? Please leave me a comment in the 🧵 And, finally, if you're interested in the outcome of the documentation user research, there's now a milestone on GitHub with some of the major activities we have planned. It's all work in progress but I'm sharing here for transparency. We always appreciate feedback and suggestions!

how to extend to use .env files for aws

@J. Camilo V. Tieck Did you hit any issue particular using

.env

? I am not sure if there are any specific about kedro.

you need to edit settings.py to load the .env. this is what I do:

from kedro.config import TemplatedConfigLoader
CONFIG_LOADER_CLASS = TemplatedConfigLoader
import os
from dotenv import load_dotenv
load_dotenv()
# Keyword arguments to pass to the `CONFIG_LOADER_CLASS` constructor.
CONFIG_LOADER_ARGS = {
    "globals_pattern": "*globals.yml", # read the globals dictionary from project config
    "globals_dict": os.environ, # read environment variables -> set by .env
}

and then I template the credential file:

estate_valuation_dev_s3bucket:
    client_kwargs:
      aws_access_key_id: ${AWS_ACCESS_KEY_ID}
      aws_secret_access_key: ${AWS_SECRET_ACCESS_KEY}

Thanks @J. Camilo V. Tieck for the suggestion. Just for my benefit, why would you want to do this? It's not a trick question btw, I don't work with Python often enough to know what you'd achieve with this and how to frame it to others as something they'd need.

Injecting environment variable into

globals_dict

is a common pattern. Even if you are not using

.env

file it could be still useful to inject variables. For example:

import os

injected_variables = {}
for key, value in os.environ:
  if key.startswith("PROJECT_SPECIFIC_CONFIG"):
     injected_variables[key] = value

CONFIG_LOADER_ARGS = {
    "globals_pattern": "*globals.yml", # read the globals dictionary from project config
    "globals_dict": injected_variables, # read environment variables -> set by .env
}

@J. Camilo V. Tieck. I guess the unclear bit here is that it’s not clear where should you put this bit in?

import os
from dotenv import load_dotenv
load_dotenv()

Do we have this documented somewhere? Cc @datajoely

it’s definitely in a issue/discussion but when dow e say that the OmegaConf solution is the main one?

@J. Camilo V. Tieck FYI, with OmegaConfigLoader this will be a build-in solution. The solution is exactly like how you template currently.

dev_s3:
  client_kwargs:
    aws_access_key_id: ${oc.env:AWS_ACCESS_KEY_ID}
    aws_secret_access_key: ${oc.env:AWS_SECRET_ACCESS_KEY}

https://docs.kedro.org/en/latest/configuration/advanced_configuration.html#how-to-load-credentials-through-environment-variables

Idea on extensions/how-tos: Incrementally convert existing project to Kedro I'm currently converting a non-Kedro project. To keep the existing project functional, I'm not replacing everything at once. • I'm starting by building the data catalog for the

final_data_set

and working backwards from there. My first pipeline will be to concat

data_set1

,

data_set2

, ..., to create

final_data_set

. • Next I'll wrap my entire existing pipeline in a single

node

before gradually refactoring the existing code to use more

nodes

and

pipelines

. While I would love to rewrite this project from scratch using Kedro, I'm trying to find the best way to adopt Kedro while retaining existing functionality. How exactly this should be done likely depends on how the original project is structured, but maybe there are some areas that would be common across all projects.

@Chris Schopp +1 on the single node approach, I have done this many times already. The first step is to wrap the entire script in a pipeline, then split more node where you needed. i.e. Maybe you need to persist a particular file, so it makes sense to first convert that bit of code to a node and let Kedro handle the I/O. From there you can slowly convert more bits to Kedro.

@Nok Lam Chan the way I’m doing this is to add this code with the imports and everything in the src/my_package/settings.py file. the settings.py is usually all commented out.

from kedro.config import TemplatedConfigLoader
CONFIG_LOADER_CLASS = TemplatedConfigLoader
import os
from dotenv import load_dotenv
load_dotenv()
# Keyword arguments to pass to the `CONFIG_LOADER_CLASS` constructor.
CONFIG_LOADER_ARGS = {
    "globals_pattern": "*globals.yml", # read the globals dictionary from project config
    "globals_dict": os.environ, # read environment variables -> set by .env
}

Thanks! I think that makes a lot of sense. I think there is some overloading of the word “environment” here because Kedro has its own

environment

and that’s where we usually suggest to keep the local credentials.