Hey Everyone Interested to know how do you guys manage your requirements.txt file to reproduce the same environment. What tools do you prefer to keep the requirements.txt file updated

As of the last few months -

uv

is the only tool you need

ohkays let me check on this

it's seriously changed the game

Basically i want to create a requirements.txt file for my project but for some reason no matter what method i am trying i am just missing out on many dependencies

so you can do it two ways

Let me explain you the scenario So i started building an application and I forgot to create any kind of requirements.txt file or requirements.in file . Then I used tools like pip

pip freeze

This simply dumps all the top level and sub packages in the requirements.txt file for some reason i did not found this very optimal. Then I came across pipreqs for a minimal requirements. This created a requirements.in file which only dumps the packages which are directly imported in the project source code. Later when i used pip-tools to compile and generate a requirements.txt file I found many dependencies were missing like

kedro-datasets[pandas-csvdataset]
kedro-datasets[ibis]
kedro-kubeflow==0.8.0
ibis-framework==9.0.0
s3fs==2024.9.0
psycopg2-binary==2.9.9

you're on the right path. either you add your dependencies to a

<http://requirements.in|requirements.in>

, or to the

[project.dependencies]

table of

pyproject.toml

(we in Kedro are moving towards the latter)

ok but what is the standard way while developing any application @Juan Luis . Like the best way I was aware of till now is to manually add the dependency along-with the version. But is that how people do it ? because there are high chances of people forgetting to add dependencies manually. I want to know the workflow to manage dependencies either to

<http://requirements.in|requirements.in>

or

pyproject.toml

.

If you do

uv add

it does the hard bit for you

@datajoely so you mean whenever we want to make use of a new python package in the application we simply do uv add and it takes care of logging the package along with version somewhere like requirements.txt and we never miss out on anything

And it does a pip compile!

@datajoely just for my understanding pip compile is needed if we want to publish our project as a python project and people can simply pip install and use it in their projects ?

Pip compile is used to make sure packages are compatible with each other

So if 2 packages have the common dependency(say X), pip compile will find the version of X which is compatible for both the packages?

Exactly!

I was just comparing uv with poetry looks like poetry also does the same thing . Any thoughts ?

Id encourage you to read through their docs it will make lots of sense

Amazing 🙂

So you can do

uv pip install -r requirements.txt

but the more modern way is to do

uv add

as you go along

getting into an issue Resolved 147 packages in 922ms error: Failed to prepare distributions Caused by: Failed to fetch wheel: psycopg2-binary==2.9.9 Caused by: Build backend failed to determine requirements with

build_wheel()

(exit status: 1)

So that's an example of something that's not compatible with the others

which python

nothing shows up So basically what i was really doing is I am trying to setup a kedro project in my new laptop. I simply cloned the project from github and ran

uv pip compile requirements.in --universal --output-file requirements.txt

After that I did

uv init

- got some error saying that project already initialised and

pyproject.toml

already exists , So i deleted the default one created by

kedro init

and reran uv unit . It published a minimal

pyproject.toml

[project]
name = "warehouse-pipeline"
version = "0.1.0"
description = "Add your description here"
readme = "README.md"
requires-python = ">=3.13"
dependencies = []

and a uv.lock file

version = 1
requires-python = ">=3.13"

[[package]]
name = "warehouse-pipeline"
version = "0.1.0"
source = { virtual = "." }

Now I am trying to make use of requirements file to setup the environment.

Great

Some quick Questions 1. since the pyproject.toml introduced by uv init is way different than what was there earlier by kedro init . It won't create any issue 2. Also is there a way to completely get rid of requirements file and push all dependencies to pyproject.toml only

So 2 is the aim

so yeaah no need to package it as it will be running like an application instead of getting used by other applications

If there aren't too many spending time doing

uv add

will compile them too

uv sync

actually removes all the installed packages from requirements.txt becuase it actually trying to sync with dependencies defined in

pyproject.toml

or

uv.lock

which is empty for me currently 😛

@chatgpt please summarise this thread 😅

@Juan Luis sorry for such big threads but i learned fundamentals of using a dependency management tool like uv and later explored poetry as well.

haha no worries! it was not a criticism. I'm happy that we helped you untangle some of these packaging tricks. what would be your main takeaways?

@Juan Luis 1. Basic pip installations in conda is not that powerful as compared to using tools like poetry and uv. Mind it Conda is too heavy an ecosystem as well 😂 2. Whether poetry or uv, both takes care of the virtual environment for your project, no need to rely on other tools like conda , pyenv for that. 3. Installing a dependency is lot easier by using commands like

poetry add

or

uv add

, they both are backed by powerful dependency resolution algorithms. 4. The best thing is they will maintain a list of dependencies for you in

pyproject.toml

5. Both of them offers a nice way to lock the exact versions of dependencies and the transitive dependencies in some kind of lock file, This lock file can be checked into VCS and can be later used by developers to reproduce an exact similar environment in which the application or library was developed. 6. The above points are mainly from dependency management point of view and we get rid of tools like

pip

,

pipreqs

,

pip-tools

. One thing to note here is tools like poetry and uv does not just do dependency resolution but it manages for you and it does it really well. 7. At the end it makes the life easier to build and publish a package if needed. 8. Uv is superfast in everything . Cheers 🥂

amazing summary, thanks a lot!!

I think I heard 'blog post', was that just me? 👀

when uv 0.5 is out? 🙃