Evaluation

Scoring is two verbs with one noun between them. soma.predict() runs a model over a loader and returns Predictions: one row per sample, with the columns the task head declares. soma.evaluate() joins those predictions with the cohort’s targets by sample_id and returns an Evaluation. Targets never travel inside predictions: they come from soma.data.Cohort.targets(), so predictions saved by a run, or made outside soma, are scored the same way.

soma.predict() runs the model where its parameters live, or on the device it is given ("auto", "cpu", "cuda:1"). A model with no parameter runs on the GPU when there is one. A run predicts on its trainer.device.

Score predictions

import soma

cohort = soma.Cohort.from_csv("dataset.csv", "splits.csv", targets=["label"])
predictions = soma.Predictions.from_parquet("run/predictions_test.parquet")
# or, after soma.fit: predictions = soma.predict(result.learner, fold.tests["test"])
evaluation = soma.evaluate(
    predictions,
    cohort.targets("test"),
    metrics=["auroc", "balanced_accuracy"],
    subgroups=["center"],
)
evaluation.to_json("evaluation.json")

evaluate scores exactly the ids it is given. It refuses a prediction without a target, a target without a prediction, and duplicate ids; the error names the count. Fill the samples a model could not score with Baseline (class prior, mean value or constant risk, fitted on the training targets), or filter the targets. When the predictions carry is_placeholder, the metrics add coverage, num_samples, num_real_samples and num_placeholder_samples. The pipeline fills samples without features this way.

soma evaluate does the same from the command line (see CLI), without a run directory.

predict writes side artefacts through callbacks (see Callbacks on fit and on predict): soma.evaluation.heatmaps.SaveAttention(dir) writes each bag’s normalised tile attention to <dir>/<sample_id>.npz.

Choose metrics

metrics is "default" (the family’s defaults, listed in the task zoo), or a list of registry keys, dotted import paths and (predictions, targets) -> float callables. A callable receives the two frames aligned by sample_id, with labels already mapped to class indices. soma.evaluation.metric_registry lists each metric’s families and direction; a key outside the predictions’ family is refused. In a config, evaluation.metrics takes the built-in names of the head’s family (an empty list means the defaults); score a metric of your own from Python, or with soma evaluate --metrics by its dotted path.

soma.evaluation.compare() runs a paired permutation test of several prediction sets against the best one.

Subgroup metrics

evaluation.subgroups names dataset.csv columns whose distinct values receive their own metrics:

evaluation:
  subgroups: [center, grade]

For each column, every value with at least two samples gets the run’s metrics and its sample count. A significance test then compares each group with the rest of the split by permuting group membership 1,000 times; it needs at least 10 samples in the group and two in the remainder. Patient-level runs read each patient’s metadata from its first slide. Results are written to subgroup_metrics_<split>.json for each test split.

Holding out test data

Set evaluation.holdout_test: true to train and report tune results without test inference or test artifacts. Checkpoint selection and tune evaluation continue normally. This supports selecting a candidate on tune results before evaluating it on the held-out test cohort.

Test identities are recorded in test_results.json. Re-scoring an already recorded test identity is skipped unless evaluation.overwrite_test: true; resuming unfinished folds is allowed. See Run outputs for experiment identity.

Segmentation and detection predictions

For a segmentation or detection head, soma.predict() runs the head’s postprocess and returns one of two prediction types instead of a table, and soma.evaluate() scores it against a target source rather than a targets frame:

	LabelMapPredictions	PointPredictions
Holds	One (H, W) uint8 label map per sample	A frame sample_id, x, y, class, score in level-0 pixels, plus every sample id (a sample with no detection is still scored)
Saved as	save(dir): one PNG per sample and index.parquet; load(dir) reads it back	to_parquet(path) / from_parquet(path)
Scored against	LabelMapSource or CachedLabelMapSource	PointSource
Metrics	mean_dice, mean_iou, dataset_global_mean_dice, foreground_mean_dice, dice_per_class	mean_f1, f1_per_class, precision, recall, mean_f1_per_image

Label maps are scored from per-sample confusion counts. Points are matched to the annotations per class within the matching distance δ, with Hungarian assignment by default. A metric given as a {name, params} leaf can change the matching: matching: greedy matches by descending score, and match_distance overrides δ (in level-0 pixels).

Point predictions keep every peak above the head’s min_score. Choosing an operating point is a separate step: with_threshold takes one threshold or a {class: threshold} mapping, and sweep_threshold() picks the threshold that maximises a metric on the given targets (one per class with per_class=True). Sweep on tune and apply the result to test:

import soma
from soma.evaluation import PointPredictions, sweep_threshold

targets = soma.data.PointSource.from_manifest(soma.AnnotationManifest.from_csv("dataset.csv"))
tune = PointPredictions.from_parquet("run/predictions_tune.parquet")
test = PointPredictions.from_parquet("run/predictions_test.parquet")

threshold, _ = sweep_threshold(tune, targets.subset(tune.sample_ids), per_class=True)
evaluation = soma.evaluate(
    test.with_threshold(threshold),
    targets.subset(test.sample_ids),
    metrics=[{"name": "mean_f1", "params": {"matching": "greedy"}}],
)

Grid predict callbacks write side artefacts: SaveOverlays (predictions and supervision drawn over the tile), SaveProbabilities (per-sample float16 softmax, segmentation) and SaveHeatmaps (per-sample float16 heatmaps, detection), for example soma.predict(learner, loader, [SaveOverlays("overlays/test", images=images)]).

Per-sample confusion matrices are recomputable from saved label maps with soma.evaluation.confusion_records().

soma evaluate also accepts a label-map directory or a points parquet. It scores label maps against the masks in the cohort CSV’s label_mask_path column and points against the files in its points_path column; both must already hold class indices.

Reference

class soma.config.EvaluationConfig(metrics=<factory>, subgroups=<factory>, holdout_test=False, overwrite_test=False, save_overlays=True, save_probabilities=False, save_heatmaps=False, attention_heatmaps=None, representation=None)

Bases: Editable

evaluation: soma.evaluate()’s metrics and subgroups (columns), and what a run scores and saves. holdout_test skips test work; overwrite_test re-scores a test set. The grid shape saves overlays, the segmentation softmax and the detection heatmaps; attention_heatmaps (cmap, alpha, blur_sigma) renders tile attention; representation makes the run a task-free CRoMa evaluation.

soma.evaluation.predict(learner_or_model, loader, callbacks=(), *, device=None)

Predict every sample of loader.

A bare model is wrapped in its shape’s learner (soma.learn.learner_for()). The learner runs in eval mode under torch.inference_mode on device ("auto", "cpu", "cuda:1"…), moved there first. Without one it runs where its parameters live, and a learner with no parameter runs as "auto": on the GPU when there is one. Each batch goes through transfer_batch and predict_step. For the set shape the frame is built from the step’s columns only, in loader order, and the head’s task_family, prediction_columns and classes describe it: a Predictions. A grid head (one declaring target_kind) assembles its own noun from the per-sample outputs (assemble_predictions): LabelMapPredictions or PointPredictions, covering every sample of the loader.

soma.evaluation.evaluate(predictions, targets, metrics='default', subgroups=())

Score predictions against targets; the prediction noun picks the scoring.

• Predictions (set shape) against a targets frame such as soma.data.Cohort.targets(), joined by sample_id.

• PointPredictions against a PointSource: class-aware point matching within the predictions’ match_distance_px (Hungarian by default; a metric leaf with params matching="greedy" / match_distance in level-0 px overrides).

• LabelMapPredictions against a LabelMapSource: per-image confusion counts reduced to Dice, IoU and the per-class scores.

Both sides must cover the same ids: an unknown prediction id, a target without a prediction, or a duplicate id is refused. Fill samples the model could not score with Baseline first; restrict a target source with subset.

Parameters:

• predictions – What the model said.

• targets – One row per sample, indexed by sample_id (or with a sample_id column), holding the target columns of the predictions’ family and any metadata columns subgroups names; a target source for grid nouns.

• metrics (str | Sequence[str | Callable[[DataFrame, DataFrame], float]] | None) – "default" (the family’s catalogue defaults; mean_pearson for a multi-output regression’s value_<k> columns) or a list of registry keys, {name, params} leaves (grid), dotted paths and callables.

• subgroups (Sequence[str]) – Target columns to break the metrics down by (set shape only).

Return type:

Evaluation

class soma.evaluation.Predictions(frame, *, task_family, prediction_columns=None, classes=None)

Bases: object

A frame of per-sample predictions keyed by sample_id.

Parameters:

• frame (DataFrame) – One row per sample: a sample_id column, the declared prediction columns, an optional boolean is_placeholder column, and any extra columns a custom learner adds.

• task_family (str) – The head’s task_family; it selects the metrics that apply.

• prediction_columns (Sequence[str] | None) – The columns the head declares. Defaults to every column of frame except sample_id and is_placeholder.

• classes (Sequence[Any] | None) – The raw label of each class index, in prob_<i> order. With it, evaluate() maps the cohort’s raw labels onto class indices; without it, labels must already be integer class indices.

class soma.evaluation.Evaluation(metrics, task_family, num_samples, subgroups=<factory>)

Bases: object

The scores of one set of predictions.

metrics maps each metric to its value (plus coverage and the sample counts when the predictions carry placeholders); subgroups holds the per-column, per-group metrics and permutation p-values ({"metrics": ..., "stats": ...}, empty without subgroup columns).

class soma.evaluation.Baseline(task_family, values, *, prediction_columns, classes=None)

Bases: object

Per-family constant predictions fitted on training targets.

Classification predicts the training class prior (and its argmax), ordinal the mean training label (rounded and clipped for the label), regression the mean training value (each value_<k> column’s mean for a multi-output regression), survival a constant risk of 0.0 (placeholders tie among themselves). Build one with fit().

soma.evaluation.compare(predictions_list, targets, metric, *, n_permutations=1000, seed=42)

Paired permutation test of each prediction set against the best one.

Every set is aligned to targets as in evaluate(); sets declaring different classes (including a set without classes next to one with) are refused. For each permutation the prediction columns of the best set and another are swapped on a random half of the samples and the metric delta recomputed. The best set gets 1.0; a set whose metric cannot be computed, or that lacks one of the best set’s prediction columns, gets None.

Return type:

list[float | None]

class soma.evaluation.LabelMapPredictions(label_maps, *, classes, ignore_index=255, geometry=None)

Bases: object

One predicted (H, W) uint8 label map per sample, held in memory.

Parameters:

• label_maps (Mapping[str, Any]) – {sample_id: (H, W) uint8}.

• classes (Sequence[Any]) – The name of each class index.

• ignore_index (int) – The value targets use for unsupervised pixels.

• geometry – The producing head’s GridGeometry, if any.

class soma.evaluation.PointPredictions(frame, *, classes, sample_ids, match_distance_px, threshold=None)

Bases: object

Detected points of a set of samples, in level-0 pixels.

Parameters:

• frame (DataFrame | None) – One row per detection: sample_id, x, y, class, score. None or an empty frame when nothing was detected.

• classes (Sequence[Any]) – The name of each class index.

• sample_ids (Sequence[str]) – Every sample the predictions cover, including those without a detection; evaluate holds them to the targets’ id set.

• match_distance_px (float) – The matching tolerance δ in level-0 pixels (the producing head’s match_distance at its spacing).

• threshold (float | Mapping[int, float] | None) – The score threshold already applied (informational), or None.

soma.evaluation.sweep_threshold(predictions, targets, metric='mean_f1', grid=None, *, per_class=False, matching='hungarian', match_distance=None)

Pick the score threshold that maximises metric on targets.

Returns (threshold, Evaluation) with the evaluation of the thresholded predictions. per_class=True picks one threshold per class (returned as {class_index: t}) maximising that class’s F1, which also maximises mean_f1 (class-aware matching makes each class’s F1 depend on its own points only); it supports metric="mean_f1". grid lists the candidate thresholds; by default the candidates are score quantiles and a uniform grid over the observed scores, plus (per class) quantiles of the scores of points within δ of a same-class annotation, bracketed by keep-all and drop-all. Ties keep the higher threshold.

soma.evaluation.confusion_records(predictions, targets, *, fold=0)

Per-sample confusion records recomputed from saved label maps.

predictions is a LabelMapPredictions (e.g. a run’s predictions_tune/), targets the label-map source it was scored against. Pixels at the predictions’ ignore_index in the target count nowhere. The class vocabulary is the predictions’ classes; a label outside it on a supervised pixel raises ValueError naming the sample.

Return type:

list[SegmentationConfusionRecord]