Evaluation

Evaluation defines the metric contract for a run and the optional subgroup breakdowns that appear in the saved outputs and reports.

The main configuration object is soma.config.EvalConfig.

class soma.config.EvalConfig(metrics=<factory>, subgroups=<factory>, save_segmentation_overlays=True, save_segmentation_probabilities=False, save_detection_overlays=True, save_detection_heatmaps=False, holdout_test=False, overwrite_test=False)

Bases: object

Evaluation metrics and subgroup analysis configuration.

Metrics are validated against the selected task family, and subgroup columns are used to break down the reported metrics in the run outputs. The dense-artifact toggles form a symmetric 2×2 over the segmentation and detection paths — cheap visual overlays default on (but suppressible, for a metrics-only run), heavy raw outputs default off: save_segmentation_overlays writes the pred/GT color overlays; save_segmentation_probabilities additionally writes a per-tile float16 (C, H, W) softmax sidecar under probs/ — opt-in because it is ~C×/precision× larger than the always-written argmax raster, and unlocks post-hoc soft-Dice/calibration/entropy/ensembling without re-running inference; save_detection_overlays writes the detection point overlays; save_detection_heatmaps writes the colormap overlays and the npz sidecar. holdout_test skips all test-split work (no test inference, no predictions_test.csv, no test entries in metrics.json/summary.json) and reports tune only — the model-selection protocol for benchmark sweeps: rank every candidate by tune score, then re-run only the winner with the test held out in. The test split may still be declared in splits.csv; it is simply not touched. Tune evaluation, threshold sweeps, checkpoint selection, and training are unaffected.

overwrite_test governs the test-results clobber guard (issue #247): because experiment identity is test-invariant, one checkpoint may be scored against several test sets over its life. Each test set’s result is namespaced by its test identity; re-scoring an already-scored test set is refused by default (a loud skip that leaves the prior result intact) unless overwrite_test is set. It is an operational flag, not part of the experiment identity.

class soma.config.SubgroupConfig(columns=<factory>)

Bases: object

Columns used for subgroup metric breakdowns.

Metric families

The default metrics depend on the task family:

Task family	Default metrics	Notes
binary_classification	auroc, balanced_accuracy, auprc, f1	Standard binary classification reporting
multiclass_classification	auroc_macro, balanced_accuracy, f1_macro	Multi-class classification
ordinal_classification	qwk, balanced_accuracy	Ordered labels
regression	mae, r2	Continuous targets
segmentation	mean_dice, mean_iou	Dense per-pixel prediction
detection	mean_f1	Class-aware F1 at matching distance δ (f1_per_class / precision / recall / mean_f1_per_image also available); see Detection

Use the smallest set of metrics that answers the scientific question, and keep it fixed when comparing runs.

Subgroup metrics

Subgroup columns are read from dataset.csv and used to break down metrics for each distinct value in the selected columns.

evaluation:
  metrics: [auroc, balanced_accuracy]
  subgroups:
    columns: [center, grade]

The run outputs write subgroup tables to subgroup_metrics_<split>.json and the HTML report includes the same breakdowns. The detailed statistical tests used for subgroup comparisons are described in Reporting.

Evaluation results

The per-split evaluation output is represented by soma.evaluation.report.EvaluationReport.

class soma.evaluation.report.EvaluationReport(split, metrics, predictions)

Bases: object

Evaluation results for a single split.

split

Split name (e.g. ‘test’, ‘tune’).

metrics

Metric name -> value.

predictions

Per-sample prediction records.

class soma.evaluation.report.SamplePrediction(sample_id, true_label, predicted_label=None, probabilities=None, predicted_value=None, raw_score=None, risk_score=None, event=None, is_placeholder=False, missing_reason=None)

Bases: object

Per-sample prediction record for audit trail.

For survival predictions, true_label holds the time-to-event, event the 0/1 indicator, and risk_score the predicted risk (higher = earlier predicted event).