Juan Luis
11/02/2023, 1:37 PM~/.miniconda
, ~/.virtualenvs
), or next to the code (~/Projects/spaceflights/.venv
)?
• when you create a new Kedro project, what are the steps you usually follow? for example 1. create and activate conda environment, 2. pip install kedro
, 3. kedro new
• what do you think of the current process?
(please leave a reply on the thread 🧵, 1 comment per person to keep the conversation tidy)
your feedback and ideas are very much welcome 🙏🏼datajoely
11/02/2023, 2:03 PMIan Whalen
11/02/2023, 2:07 PMwhat tool do you use to create and manage environments for your Kedro projects?Conda for virtual env, then pip for package installation
where do you place your environments when working with Kedro projects?Default conda directory (
~/.anaconda
) but store package versions in a requirements.txt
or something like that
when you create a new Kedro project, what are the steps you usually follow?I use the workflow you point out above. Create env, install kedro, kedro new
what do you think of the current process?I think its fine! It reflects what one should do with any other kind of project.
marrrcin
11/02/2023, 2:23 PMJavier del Villar
11/02/2023, 2:27 PMJavier Perez
11/02/2023, 2:48 PMLukas Innig
11/02/2023, 3:01 PMMatthias Roels
11/02/2023, 7:10 PMFROM base_image
, a COPY
to copy the relevant directories and an ENTRYPOINT
and/or CMD
• The initial base image lives in a separate repo containing all base images we use. We install relevant linux libraries and python dependencies. We use pip-tools to generate a requirements.txt
with all versions pinned (==
) from a <http://requirements.in|requirements.in>
. Install is done with pip as root (no need for any venv or anything as we are already using containers for isolation).
I would like to make the split less dramatic in future projects, but to do so, I miss the equivalent of Rust Workspaces in the Python world. Essentially a splitting your Rust project into workspaces allow you to create subprojects, each with its own set of dependencies defined. But in the end, a general lockfile is created to ensure we have consistent versions across workspaces.Takieddine Kadiri
11/02/2023, 7:12 PMJordan Barlow
11/03/2023, 4:14 PMwhat tool do you use to create and manage environments for your Kedro projects?Micromamba, Poetry. Would also like to experiment with Rye/Pixi + Kedro
where do you place your environments when working with Kedro projects?Next to the code.
when you create a new Kedro project, what are the steps you usually follow?Usually create a conda environment and use Poetry inside it. I have an environment lockfile, and a packages lockfile, which makes shoving things in a container and having my package installed in editable mode straightforward. However, once Pixi can install from PyPI and build/install local development packages, I can stick with a single tool, perhaps 😃
what do you think of the current process?Personally, I'm not a massive fan of pip alone - I like the convenience of a workflow tool.
Ben Horsburgh
11/06/2023, 9:07 AMminiconda
• Global named environments
• conda create...
-> pip install -U kedro
-> kedro new
-> cd ...
pip install src/requirements.txt
-----
Process is easy.
I hate requirements.txt
files - they allow stuff that works in pip
but not anywhere else! Can we use pyproject.toml
instead?Marc Gris
11/06/2023, 10:12 AMvenv
& pip
• I like to have the .venv
in my projects (easily go to definitions etc..)
• my steps:
◦ kedro new
◦ cd <new-project>
◦ python -m venv .venv && source .venv/bin/activate
◦ pip install -r src/requirements.txt
Iñigo Hidalgo
11/06/2023, 6:38 PMconda
to install different python versions, so a windows pyenv
equivalent (I know pyenv-win exists, but I couldn't get it to work without admin rights), I have conda environments called py38
, and py310
which I use as a base.
• Then I use venv
to create a local .venv
then do pip install requirements
. I don't ever use kedro new
, I just use plain cookiecutter
which is installed through pipx
so is globally available
If I were to use kedro new
I'd probably aim to have it installed through pipx, though I never saw the value of that vs cookiecutter
relative to the potential headache I'd have with conflicting versions of kedro depending on the PATH
orderkedro new
I'd probably aim to have it installed through pipx, though I never saw the value of that vs cookiecutter
relative to the potential headache I'd have with conflicting versions of kedro depending on the PATH
orderFlorianGD
11/07/2023, 8:36 AMvenv
with the help of rtx. I use rtx
to install a version of python, create the virtualenv and activate it as I cd
into the directory. Then pip install -r requirements.txt
(or requirements.lock
in CI and production)
This is the .rtx.toml
that I mainly use:
dotenv = ".env"
[tools]
python = {version = "3.10.11", virtualenv = ".venv"}
(the dotenv
entry is to load the env variables with rtx
as well, not needed here)Juan Luis
11/07/2023, 8:41 AMkedro new
could create a Kedro project in the current directory, which is difficult to do because the underlying library, cookiecutter
, doesn't support it. however, from the responses I get that the main annoyance is that people tend to have a "global Kedro" and then a "project-specific" Kedro.
feel free to drop your thoughts in https://github.com/kedro-org/kedro/issues/681#issuecomment-1798043381Markus Sagen
11/08/2023, 6:07 AMdatajoely
11/08/2023, 9:45 AM