Benchmarking

Alongside the library, the repository ships the pipeline used to produce the paper’s numbers. It is not part of the installed package – it lives under scripts/ in the source tree and needs the repro extra:

pip install "croma[repro]"

To embed a tileset you also need the encoder sources, which are only distributed as git repositories and so cannot be pinned in the package metadata:

pip install -r scripts/bench/requirements-encoders.txt

Scoring embeddings that already exist needs none of them.

The pipeline

Three commands, split along the seams of what is expensive:

# 1. Embed a tileset once, into output/embeddings/<tileset>/.
#    --manifest is needed only the first time; it derives manifest.csv.
python scripts/bench/extract_embeddings.py \
  --tileset pathorob-camelyon \
  --manifest data/pathorob/manifests/pathorob-camelyon.csv \
  --models UNI,Virchow2

# 2. Compute metrics for one benchmark at one protocol. Reads embeddings,
#    never writes any. Results land in output/metrics/<protocol>/<benchmark>/.
python scripts/bench/benchmark.py --benchmark camelyon --protocol median-k

# 3. Render that run's figure set from the written artifacts.
python scripts/bench/render.py output/metrics/median-k/camelyon

Each <Model>.npy and <Model>.npy.json sidecar is one artifact. The sidecar is published last as the completion marker and records the checkpoint revision (an immutable SHA for pinned models, otherwise an explicit null), extraction-contract version, precision, batch size, output dtype and shape, and tileset-manifest fingerprint. Re-running extraction skips a pair only when that complete contract still matches. An orphaned, malformed, stale, or mismatched pair fails loudly; inspect the mismatch and pass --force only when you intend to replace it.

When canonical image storage is too slow for extraction, --image-path-map may redirect reads to a local mirror. The CSV must contain sample_id, canonical_image_path (the exact image_path stored in manifest.csv), and access_path. It must cover the frozen manifest one-to-one; map rows may be in any order. Access paths never enter tile identity or sidecar provenance, so the canonical manifest and its fingerprint remain unchanged.

Embeddings are a tileset; benchmarks are views

A benchmark is a row-view of a tileset, and every benchmark over a tileset shares that tileset’s embeddings. Adding an encoder means embedding it once; it then joins every benchmark over that tileset automatically. Benchmarks are declared in scripts/bench/benchmarks.py, and scripts/bench/run_benchmarks.sh median-k sweeps them all. The rationale is recorded in ADR-0007.

tau across models

benchmark.py resolves tau per model by default – each model gets the median typed neighbour distance of its own embedding at the operating k. This matches the library default and is the only setting under which MaRI is comparable across models. Passing --tau <float> pins one temperature for every model; the run then prints which models it is off-scale for. See Choosing tau.

Encoder-specific environments

Two checkpoint families (Mascaret and Phaet, and the RudolfV 2 family) use remote-code runtimes whose pins conflict with the shared encoder environment, so each is embedded from a dedicated venv built from its own requirements file under scripts/bench/ (requirements-waiv.txt, requirements-rudolfv2.txt). Each family also has an opt-in gated-weight smoke test (CROMA_RUN_WAIV_SMOKE=1 / CROMA_RUN_RUDOLFV2_SMOKE=1) that the default offline suite skips. Exact preprocessing contracts and immutable checkpoint revisions live in the model registry.