I m curious does anyone uses the conda ecosystem not only to Kedro #random

I’m curious, does anyone uses the conda ecosystem ...

Jordan

01/19/2023, 4:15 AM

I’m curious, does anyone uses the conda ecosystem not only to manage environments, but also to manage Kedro project dependencies?

datajoely

01/19/2023, 8:57 AM

I’m in category 1, not a huge fan of conda for the other stuff This talk I watched a few years ago talked about the different env manager options and covered why I like it’s approach to env management

https://www.youtube.com/watch?v=YKfAwIItO7M▾

datajoely

01/19/2023, 8:58 AM

In general for the packaging/dependency stuff we try and stick to

pip

and

pypa

1st party standards rather than things like Poetry, PDM, PyEnv etc

K 1

datajoely

01/19/2023, 8:58 AM

(Or 1st party complimentary integrations like

pip-compile

)

Juan Luis

01/19/2023, 10:11 AM

I'm leaning towards pip for installation these days, but been a big fan of conda. for tricky stuff like gpu-enabled deep learning frameworks, geospatial stuff and the like, I'd definitely use conda (rather: its younger cousin,

mamba

, which is blazing fast even for

conda-forge

packages).

K 1

Ian Anderson

01/19/2023, 12:27 PM

We use

conda

for environment management and

kedro

project dependencies. It was in part because Anaconda has some managed services that helped our public sector org from the security perspective. So there’s always a command that pulls from a managed distribution channel on a server inside the firewall:

conda env create -n new_environment --file src/environment.yml

Not trying to change the subject, I just know that Jordan’s question was fundamental as we were trying to figure out how to do the source code part of reproducible, modular, maintainable data science code. How do orgs handle security concerns around OSS and tools like

pip

K 2

Juan Luis

01/19/2023, 1:53 PM

How do orgs handle security concerns around OSS and tools like
pip
?

I think possible answers are a) They don't (highly risky) b) They use

--extra-index-url

(not secure at all, see https://scribe.rip/@alex.birsan/dependency-confusion-4a5d60fec610) c) They have a PyPI proxy and block all access to

<http://pypi.org|pypi.org>

(the only secure option)

👍 2

Jordan

01/19/2023, 3:03 PM

This talk I watched a few years ago talked about the different env manager options and covered why I like it’s approach to env management

Nice, I’ll check this out

Jordan

01/19/2023, 3:13 PM

For most of my personal projects, I have no particular need to package them, hence

setup.py

is kind of irrelevant. The only thing I use it for is to install my package in editable mode during development.

conda develop src

, which just adds the relevant dirs that include

__init__.py

to the path, is enough to meet my needs most of the time, but it’s unofficially depreciated these days, so not really a long term solution. So looks like

pip install --no-build-isolation --no-deps -e src

would be the best way to install in editable mode if one were using conda to manage all dependencies.

Juan Luis

01/19/2023, 3:50 PM

I see you're subscribed to https://github.com/conda/conda-build/issues/4251 as well 😄

Jordan

01/19/2023, 3:58 PM

I see you’re subscribed to https://github.com/conda/conda-build/issues/4251 as well 😄

That’s hilarious, I had no idea I was talking to the same person. Looks like we move in the same circles 😉

🙃 2

5 Views

Open in Slack

Previous Next