Using uv for Python Package Management¶

Our HPC users can use uv, an extremely fast package and project manager written in Rust. uv can act as a drop-in replacement for pip and venv, and is designed for high performance and reproducible Python workflows.

What is uv?¶

uv is a Python package and project manager written in Rust. It provides:

• Very fast environment and package operations (create, install, resolve).
• A unified interface that can replace:
• pip for package installation and management.
• venv for creating virtual environments.
• A modern, reproducible workflow for Python projects (including pyproject.toml-based projects).

While uv is not a replacement for Conda in all scenarios, it excels at pure Python workloads where system-level dependencies are not required.

Using uv on NSF NCAR HPC Systems¶

uv is available as an environment module on NCAR HPC systems.

Before using uv, ensure no conda environments are active:

conda deactivate

Then, load the uv module:

module load uv

By default, uv stores its cache in your home directory, which can quickly exceed your quota. The module automatically sets the cache location to /glade/derecho/scratch/$USER/.cache/uv. You can verify that by running uv cache dir and confirming the Cache dir points to your scratch space, not your home directory.

module unload conda
module load uv

Working with Python Projects using uv¶

uv supports managing Python projects, which define their dependencies in a pyproject.toml file.

You can create a new Python project using the uv init command:

cd /glade/work/$USER
uv init my-analysis

This creates a project directory with the following structure:

my-analysis/
├── pyproject.toml    # Project configuration and dependencies
├── README.md         # Project description
└── .python-version   # Pinned Python version

The pyproject.toml file is where you define your project's dependencies.

Managing Dependencies¶

Now you can add dependencies to your project:

cd my-analysis
# Add packages
uv add numpy pandas matplotlib xarray

Each time you add packages, uv updates your pyproject.toml and creates or updates a uv.lock file that pins exact versions of all dependencies. This lock file ensures that anyone can reproduce your exact environment.

You can also install packages from a requirements.txt file:

uv add -r requirements.txt

You can remove packages with uv remove:

uv remove numpy

To upgrade packages to their latest compatible versions, use:

uv lock --upgrade-package numpy

The --upgrade-package flag will attempt to update the specified package to the latest compatible version, while keeping the rest of the lockfile intact.

Running Commands in uv Project Environments¶

Now, to run Python in your project environment:

# Run a script
uv run python analyze_data.py

# Start an interactive Python session
uv run python

The uv run command automatically activates your project environment and keeps it in sync with your pyproject.toml and lockfile, guaranteeing your command runs in a consistent, locked environment.

You can use uv pip list to see installed packages in your project environment:

uv pip list

Creating and Activating a Virtual Environment with uv¶

uv can be used to create standalone virtual environments similar to python -m venv.

To create an environment:

mkdir -p /glade/work/$USER/uv-envs

# Create a new uv virtual environment
uv venv /glade/work/$USER/uv-envs/myenv

# Activate the environment
source /glade/work/$USER/uv-envs/myenv/bin/activate   

uv venv /glade/work/$USER/uv-envs/myenv-py3.10 --python python3.10

source /glade/work/$USER/uv-envs/myenv-py3.10/bin/activate

python --version

Low-level uv pip Commands¶

These commands are intended for legacy workflows or cases where the high-level project commands (uv init, uv add, uv run) do not provide enough control.

Once your uv environment is active, you can install packages using uv’s pip interface:

uv pip install numpy pandas matplotlib

This behaves similarly to pip, but with much faster dependency resolution.

Similarly you can install from a requirements.txt file:

uv pip install -r requirements.txt

or you can uninstall packages:

uv pip uninstall numpy

Other useful uv pip commands include:

• uv pip check: Check that the current environment has compatible packages.
• uv pip tree: View the dependency tree for the environment.
• uv pip install -r pyproject.toml: Install packages from a pyproject.toml file.

Locking packages in an environment can be done with:

• uv pip compile: Compile requirements into a lockfile.
• uv pip sync: Sync an environment with a lockfile.

Reproducing uv Environments¶

One of uv's key strengths is built-in support for reproducibility through lock files.

When you create a uv project and add dependencies, uv automatically generates a uv.lock file that specifies exact versions of all packages and their dependencies.

To reproduce an environment from a project with a lock file:

# Clone or copy a project with pyproject.toml and uv.lock
cd my-analysis

# Install exact versions from lock file
uv sync --frozen

The --frozen flag tells uv to use the exact versions in the lock file without attempting to update them.

Creating Jupyter kernels for uv environments¶

uv environments can be used in JupyterLab sessions on the NCAR JupyterHub service.

First, install the ipykernel package into your uv environment:

source .venv/bin/activate
uv pip install ipykernel

At this point, your environment should appear automatically as a kernel option when you start a Jupyter server on Derecho or Casper. This method is convenient and ensures that all environment settings are properly applied when the kernel launches.

You can also manually create a kernel specification with a custom name:

uv run python -m ipykernel install --user --name=my-uv-analysis-kernel

Using uv in Batch Jobs¶

#!/bin/bash
#PBS -N uv_venv_job
#PBS -A PROJECT_CODE
#PBS -l select=1:ncpus=4:mem=16GB
#PBS -l walltime=02:00:00
#PBS -q main

# Load uv module
module load uv

# Run your script
uv run python analyze.py