Benchmarking

A benchmark is a fixed protocol: the data, its splits, the downstream model, the metric and the seeds are all held constant, and only the encoder varies. soma reproduce prepares the data, runs the protocol and scores the result; soma leaderboard ranks completed runs without retraining. Each benchmark page states its data and task, its protocol, the results soma has recorded and how to reproduce them.

Benchmarks

Benchmark	Task	Data	Metric
EVA	Patch classification, nucleus segmentation, slide classification	Ten public datasets of the kaiko-ai/eva leaderboard	Balanced accuracy, foreground mean Dice
OCELOT	Cell detection	OCELOT 2023 cell patches	Mean F1 at 3 µm
HEST	Spatial gene-expression prediction	Nine HEST-Benchmark tasks	Mean Pearson
CRoMa	Robustness to the medical centre, task-free	Three PathoROB tile cohorts	CRoMa margin

Two more detection benchmarks are registered, detection/midog (MIDOG 2022 mitosis detection) and detection/monkey (MONKEY inflammatory-cell detection). Their data preparation is described in Curation; their results pages follow once the rows are recorded on this release.

Reproduce a benchmark

1. Prepare the raw data as the benchmark’s page describes, in a directory you pass as --raw-root. soma list benchmarks prints every registered protocol.

2. Run the protocol with its default encoder, or any compatible installed one:

   soma reproduce eva/bach --encoder uni2 --raw-root /path/to/eva/bach --output-root runs/eva-bach
   

3. Read the score. When a packaged reference exists for that encoder, soma prints the measured value next to it. The comparison is informational: an encoder without a reference still runs, and a gap never fails the command.

--seeds 1 runs seed 0 only, for a quick smoke test; omit it for the canonical seed set. --curated-dir reuses prepared manifests, --from-run-dir rescores one existing run, and --record appends the score to the benchmark’s results ledger, which is what the results tables in these pages are built from.

Compare encoders

--encoders runs an ordered panel on one benchmark or on a family:

soma reproduce eva/bach --encoders uni2 virchow2 --raw-root /path/to/eva/bach --output-root runs/eva-bach
soma reproduce eva --encoders uni2 virchow2 --raw-root /path/to/eva --output-root runs/eva

A family prefix such as eva runs every registered member; EVA and HEST expect one raw-data subdirectory per member, CRoMa shares one prepared root. soma checks every benchmark and encoder pairing against slide2vec’s encoder registry before any run starts. Benchmarks then run in canonical order and encoders in the given order, with the raw data curated once per benchmark. Each benchmark writes a cross-encoder leaderboard under its output root, for example runs/eva/bach/leaderboards/eva/bach.*; members of a family are never combined into one rank. If an encoder fails, the later ones still run, the leaderboard covers the completed runs, and the command reports PARTIAL with a nonzero exit status.

To add a private encoder to the panel, see Benchmark an in-house encoder.

Compare other choices

soma leaderboard ranks completed runs along any axis and writes CSV, JSON and HTML tables, including any packaged reference. Runs that share a dataset, splits and task form one comparison. To compare aggregators, decoders, spacings or feature modes, run ordinary configs that differ only in that axis under one output root, then rank them:

soma leaderboard --root runs/agg-sweep --vary aggregator

Where a benchmark lives

Each benchmark is one package under soma/benchmark/: its curator, its recipes, its scorer, the packaged reference rows (reference.csv) and the results ledger --record appends to (results.csv). The soma command imports soma.benchmark at startup; from Python, import soma.benchmark registers the benchmarks (import soma alone does not). See CLI for every option, Run outputs for the run artifacts and Reproduce a packaged benchmark programmatically for the Python workflow.