API Reference¶

Reference for the Python API. See Getting Started for introductory examples and Metrics for what each metric measures.

croma exposes three metric classes, the downstream probe protocol and two reductions, and one alignment helper. Each metric class is a namespace of classmethods – you never instantiate them; apd, nipd and probe_sweep are plain functions. The short names are the ones to import:

from croma import CRoMa, MaRI, RI, apd, nipd, probe_sweep

Import as	Class	Returns
`RI`	`RobustnessIndex`	`RobustnessResult`
`MaRI`	`MarginAwareRobustnessIndex`	`RobustnessResult`
`CRoMa`	`CrossConfounderRobustnessMargin`	`CRoMaResult`
`apd`	`apd()`	`float`
`nipd`	`nipd()`	`float`
`probe_sweep`	`probe_sweep()`	`numpy.ndarray`

RI¶

class croma.RobustnessIndex¶

Bases: BaseRobustnessIndex

classmethod compute(features, manifest, *, confounder_column, k_candidates, evaluation_design='all')¶

Compute RI at the operating k selected by kNN balanced accuracy.

Parameters:: evaluation_design (str) – "all" (the default) or "paired_2x2"; see the constants in croma.metrics.base for what each scope scores.
Return type:: RobustnessResult

classmethod compute_curve(features, manifest, *, confounder_column, k_values, evaluation_design='all')¶

RI at every k in k_values, under evaluation_design (default "all").

Return type:: dict[int, float]

MaRI¶

class croma.MarginAwareRobustnessIndex¶

Bases: BaseRobustnessIndex

classmethod recommend_tau(features, manifest, *, confounder_column, k, evaluation_design='all')¶

Recommended tau for this dataset: the median typed (SO/OS) neighbour distance.

Returns nan when no typed neighbour exists within the top-k set, and 0.0 when typed neighbours exist but half or more of them sit at distance 0 (a collapsed embedding, or a manifest that duplicates rows). Neither can be put on a meaningful scale, but they are distinct datasets: see TAU_FALLBACK.

Return type:: float

classmethod compute(features, manifest, *, confounder_column, k_candidates, tau=None, evaluation_design='all', warn_tau=True)¶

Compute MaRI at the operating k selected by kNN balanced accuracy.

Parameters:

evaluation_design (str) – "all" (the default) or "paired_2x2"; see the constants in croma.metrics.base for what each scope scores.
tau (float | None) – Distance-decay temperature. Leave as None (recommended) to resolve it automatically as this dataset’s median typed-neighbour distance at the operating k – the on-scale value. A pinned tau is only comparable across models whose typed-neighbour distances share a scale; because that scale is a property of each embedding, a single fixed tau silently sharpens the margin for some models and flattens it for others.
warn_tau (bool) – Warn when a pinned tau sits off the typed-neighbour scale. Ignored when tau is resolved automatically, which is on-scale by construction.

Return type:

RobustnessResult

classmethod compute_curve(features, manifest, *, confounder_column, k_values, tau=None, evaluation_design='all')¶

MaRI at every k in k_values, all scored at a single tau.

Auto-tau (the default) is resolved once, at the operating k selected over k_values, and then held fixed across the sweep: a tau that moved with k would confound the curve’s shape with the temperature’s.

Return type:: dict[int, float]

CRoMa¶

class croma.CrossConfounderRobustnessMargin¶

Bases: object

classmethod compute(features, manifest, *, confounder_column, evaluation_design='all', m=5, alpha=0.1, start_k=200, k_growth_factor=2.0)¶

Compute the cross-confounder robustness margin.

Parameters:: evaluation_design (str) – "all" (the default) or "paired_2x2"; see the constants in croma.metrics.base for what each scope scores.
Return type:: CRoMaResult | dict[int, CRoMaResult]

Probe protocol¶

probe_sweep produces the matrix the reductions below consume. It trains a probe to predict the biological class from frozen embeddings while a schedule walks the training set from balanced to fully confounded, and scores each probe on test rows that do not move. It takes embeddings and a split assignment: no model is loaded, no manifest read and no output layout touched.

from croma import apd, probe_sweep
from croma.downstream import pathorob_schedule

accuracies = probe_sweep(
    embeddings,                  # (n_rows, n_features)
    center_index,                # (n_rows,) confounder index per row
    class_index,                 # (n_rows,) biological class index per row
    schedule=pathorob_schedule("camelyon", rows_per_slide=300),
    rows_per_slide=300,
)
apd(accuracies)

croma.probe_sweep(embeddings, confounders, labels, *, schedule, rows_per_slide=1, iterations=20, seed=1000, validation_fraction=None)¶

Run the confounder-biased probe sweep and return its accuracy matrix.

One row per split, one column per resampling replicate:

accuracies[s][i] = balanced accuracy of the probe trained on split s, replicate i

Row 0 is whatever the caller put first in schedule – by construction the balanced baseline, the split at which the confounder carries no information about the biological class. Every later row is a more confounded split. That is exactly the matrix croma.apd() and croma.nipd() reduce, so the return value goes into either of them untouched.

Each replicate reshuffles every (confounder, class) cell by whole slides, then walks the schedule, cutting each split’s training rows off the front of the cells, its validation rows from directly behind them, and its test rows from a tail that starts beyond the widest training block any split asks for. The tail therefore holds the same rows for every split, which is what makes accuracies across splits comparable: the training composition moves, the test composition does not.

Parameters:

embeddings (_Buffer | _SupportsArray[dtype[Any]] | _NestedSequence[_SupportsArray[dtype[Any]]] | complex | bytes | str | _NestedSequence[complex | bytes | str]) – (n_rows, n_features) frozen representations. Rows are grouped into slides by position, so they must arrive in the order the slides do.
confounders (_Buffer | _SupportsArray[dtype[Any]] | _NestedSequence[_SupportsArray[dtype[Any]]] | complex | bytes | str | _NestedSequence[complex | bytes | str]) – (n_rows,) confounder index per row – the medical centre, scanner or provider whose influence is being injected. These are the i of a schedule entry, so they index it and must run 0 .. n_confounders - 1.
labels (_Buffer | _SupportsArray[dtype[Any]] | _NestedSequence[_SupportsArray[dtype[Any]]] | complex | bytes | str | _NestedSequence[complex | bytes | str]) – (n_rows,) biological class index per row: the j of a schedule entry, running 0 .. n_classes - 1. This is what the probe predicts.
schedule (Sequence[tuple[Sequence[tuple[int, int, int]], int]]) – One SplitPlan per split, the balanced baseline first. Two splits at minimum, since a sweep with no confounded split reduces to nothing.
rows_per_slide (int) – How many consecutive rows make up one slide. Slides, not rows, are the unit that is shuffled and counted, so train and test never share one.
iterations (int) – How many resampling replicates to run – the width of the matrix.
seed (int) – Seed of the replicate seeds. The sweep is a pure function of it, so two runs with the same seed return the same matrix.
validation_fraction (float | None) – Size of each cell’s validation slice, as a fraction of its training rows. None selects PathoROB’s own rule, 1 / max_train_slides of the cell’s training rows, which is the faithful setting for a sweep over patches. It underflows to zero when a slide is one row, which is why a slide-level sweep has to state the fraction instead.

Return type:

ndarray

Returns:

The (n_splits, iterations) matrix of balanced accuracies on the held-out in-domain rows.

Raises:

ValueError – If the inputs are not aligned rectangular arrays of non-negative integer indices; if rows_per_slide or iterations is not positive, or iterations exceeds the 10,000 seeds a replicate can be drawn from; if validation_fraction is outside [0, 1); or if the schedule holds fewer than two splits, names a cell no row carries, asks a cell for more rows than it holds or than max_train_slides allows, or leaves some split with no training, validation or test rows. Those last few are the same fault wearing different clothes: a schedule and a cohort that were not written for each other, which would otherwise run to completion at a confounder bias nobody asked for.

Scoring unseen confounders¶

PathoROB scores every probe both on held-out rows of the confounders it trained on and on an unseen confounder. Both come off one training pass, so they are one call. This form also takes arrange_slides, which replaces the one step of a replicate that decides which slides a split trains on and which sit in the held-out tail – for a cohort whose slides cannot be ordered freely. Its default is the sweep’s own shuffle, which is the reference protocol.

croma.downstream.probe_sweep_over_test_sets(embeddings, confounders, labels, *, schedule, test_sets=None, rows_per_slide=1, iterations=20, seed=1000, validation_fraction=None, arrange_slides=None)¶

Run one sweep and score every probe it trains on more than one test set.

Same protocol as probe_sweep(), and its in-domain matrix is identical – this form only adds test sets that were never trained on, typically rows from confounders the sweep never saw. They ride along on the probes the sweep already trained, so asking for an unseen-confounder matrix costs one prediction per split and replicate rather than a second sweep.

Parameters:

embeddings (_Buffer | _SupportsArray[dtype[Any]] | _NestedSequence[_SupportsArray[dtype[Any]]] | complex | bytes | str | _NestedSequence[complex | bytes | str]) – As probe_sweep().
confounders (_Buffer | _SupportsArray[dtype[Any]] | _NestedSequence[_SupportsArray[dtype[Any]]] | complex | bytes | str | _NestedSequence[complex | bytes | str]) – As probe_sweep().
labels (_Buffer | _SupportsArray[dtype[Any]] | _NestedSequence[_SupportsArray[dtype[Any]]] | complex | bytes | str | _NestedSequence[complex | bytes | str]) – As probe_sweep().
schedule (Sequence[tuple[Sequence[tuple[int, int, int]], int]]) – As probe_sweep().
test_sets (Mapping[str, tuple[_Buffer | _SupportsArray[dtype[Any]] | _NestedSequence[_SupportsArray[dtype[Any]]] | complex | bytes | str | _NestedSequence[complex | bytes | str], _Buffer | _SupportsArray[dtype[Any]] | _NestedSequence[_SupportsArray[dtype[Any]]] | complex | bytes | str | _NestedSequence[complex | bytes | str]]] | None) – Extra test sets as {name: (embeddings, labels)}. Their labels are biological class indices, as labels is; they carry no confounder index, because nothing about a test set is confounder-dependent. Their names come back as the keys of the result and cannot be "in_domain".
rows_per_slide (int) – As probe_sweep().
iterations (int) – As probe_sweep().
seed (int) – As probe_sweep().
validation_fraction (float | None) – As probe_sweep().
arrange_slides (Callable[[Sequence[Sequence[Sequence[Sequence[int]]]], Random], Sequence[Sequence[Sequence[Sequence[int]]]]] | None) – How one replicate orders each cell’s slides – the step that decides which slides a split trains on and which sit in the held-out tail. None selects the sweep’s own step, a shuffle of every cell with the replicate’s generator, which is the reference protocol. Pass a SlideArrangement for a cohort whose slides cannot be ordered freely: PathoROB’s Tolkach-ESCA sweep, for one, draws a train/test case split per replicate and pushes those cases to the tail, because a case there carries patches of several biological classes and may not be trained on and tested on at once. An arrangement receives the cells in input order and returns them reordered; it may draw from the generator it is given, and must draw before it shuffles for its sweep to match a driver that does, since the two share one stream. It may not add, drop or break up a slide.

Return type:

dict[str, ndarray]

Returns:

{"in_domain": matrix} plus one (n_splits, iterations) matrix per named test set. Each is a well-formed input for croma.apd() and croma.nipd().

Raises:

ValueError – Everything probe_sweep() raises, plus a test set whose embeddings and labels disagree in length, whose feature count differs from embeddings, that is empty, or that is named "in_domain"; and an arrange_slides that returns anything but a rearrangement of the slides it was handed.

PathoROB’s schedules¶

croma.downstream.pathorob_schedule(dataset, *, rows_per_slide, n_splits=None)¶

PathoROB’s own schedule for one of its three downstream datasets.

A schedule is the sequence of training compositions a sweep walks, from balanced to fully confounded. This one is not croma’s: it comes from the split-mapping helper vendored verbatim from PathoROB, so a sweep run on it is the reference protocol rather than something resembling it. Pass the result straight to probe_sweep().

Cells are addressed as (confounder, class) in the order PathoROB lists them – camelyon as RUMC, UMCU x normal, tumor, and so on – so the caller’s confounders and labels have to be indexed the same way. Datasets with their own schedule, croma’s included, build the sequence themselves; nothing here is required.

Parameters:

dataset (str) – "camelyon", "tcga" or "tolkach_esca" – PathoROB’s names, kept as PathoROB spells them (note "tcga" for the 4x4 cohort).
rows_per_slide (int) – How many rows one slide contributes, which is what the schedule’s per-slide counts are multiplied by. PathoROB publishes 300 for camelyon, 30 for tcga and 100 for tolkach_esca; a slide-level sweep passes 1.
n_splits (int | None) – How far to walk the schedule. Defaults to the number of splits PathoROB runs for this dataset (PATHOROB_SPLITS), and cannot exceed it – past that point the formulas run the favourable cells negative, so a longer walk would not be more confounded, it would be undefined. A shorter one is a prefix of the reference protocol, which is what a smoke run wants.

Return type:

list[tuple[Sequence[tuple[int, int, int]], int]]

Returns:

One SplitPlan per split, balanced baseline first.

Raises:

ValueError – If dataset is not one PathoROB has a schedule for, if rows_per_slide is not positive, or if n_splits is not positive or runs past the schedule’s last split.

These two, and croma.downstream.IN_DOMAIN – the key the sweep’s own held-out matrix comes back under – are reachable under croma.downstream but are not promoted to the top level, so they carry no stability promise.

Downstream reductions¶

Unlike the three metrics above, apd and nipd read no embeddings: they reduce the balanced accuracies a confounder-biased probe sweep already produced. Both take the same (n_splits, n_iterations) matrix, with the balanced baseline in row 0 and each later row a progressively more confounded split. nipd additionally takes the Cramér’s-V coordinate of every row.

APD¶

apd is PathoROB’s metric, reported as the faithful reference. Its reduction is vendored verbatim from PathoROB (BSD 3-Clause, © 2025 BIFOLD Pathomics; see the distribution’s NOTICE) rather than reimplemented, so a value reported as “APD” is the value PathoROB would report. It takes no chance argument: it normalizes by raw accuracy.

from croma import apd

apd(accuracies)  # -> e.g. -0.046

croma.apd(accuracies)¶

Average Performance Drop: the mean relative accuracy change a confounder costs.

accuracies is an (n_splits, n_iterations) matrix of balanced-accuracy scores from a confounder-biased probe sweep, where row 0 is the balanced baseline and every later row is a progressively more confounded split. APD is the mean, over the confounded splits, of each split’s accuracy relative to the baseline, minus one:

APD = mean_i( mean_{s>0}( accuracies[s][i] / accuracies[0][i] ) - 1 )

So 0 means no split lost accuracy, and a value near -1 that the confounded splits collapsed. Closer to zero is more robust.

This is PathoROB’s metric, not croma’s, and it is reported because it is faithful: the reduction itself is vendored verbatim from PathoROB (croma.downstream._pathorob) rather than re-derived, so a number reported as “APD” means what the PathoROB paper means by it. This function adds argument validation and averages the replicate axis – exactly what PathoROB’s own driver does to turn the per-replicate scores into the scalar it publishes.

Note that the ratio is taken per replicate and averaged afterwards, preserving PathoROB’s published reduction. croma.nipd() is a separate estimand: it averages repeats first, normalizes by above-chance baseline skill, and integrates over Cramér’s V.

Parameters:: accuracies (_Buffer | _SupportsArray[dtype[Any]] | _NestedSequence[_SupportsArray[dtype[Any]]] | complex | bytes | str | _NestedSequence[complex | bytes | str]) – (n_splits, n_iterations) balanced accuracies, row 0 the balanced baseline. At least two splits and one replicate.
Return type:: float
Returns:: The mean relative performance drop, a plain float.
Raises:: ValueError – If accuracies is not a 2-D matrix with at least one confounded split and at least one replicate, if it holds a non-finite score, or if any replicate’s baseline accuracy is zero or negative. The last is the reduction’s domain running out: mean-of-ratios divides by each replicate’s own baseline, so a zero there has no ratio and a negative one inverts every sign.

nIPD¶

nipd is the normalized integrated performance degradation: the signed area under the chance-normalized degradation curve over Cramér’s V. It divides performance changes by baseline skill – balanced accuracy above chance – rather than by raw baseline accuracy. This corrects unequal baseline headroom across models and tasks.

For mean balanced accuracy across repeated training runs, \(\bar a(V)\), baseline \(a_0 = \bar a(0)\) and chance \(\pi\),

\[g(V) = \frac{\bar a(V) - a_0}{a_0 - \pi}, \qquad \operatorname{nIPD} = \int_0^1 g(V)\,dV.\]

The integral is estimated by the trapezoidal rule at the supplied Cramér’s-V coordinates. Consequently, interval widths – not the number of sampled conditions – weight the curve. The coordinates must be finite, strictly increasing, aligned with the accuracy rows and span 0 to 1. The mean baseline must exceed chance; there is no additional weak-skill threshold.

from croma import nipd

nipd(
    accuracies=[
        [0.90, 0.90],
        [0.70, 0.70],
        [0.50, 0.50],
    ],
    cramers_v=[0.0, 0.5, 1.0],
    chance=0.5,
)  # -> -0.5

croma.nipd(accuracies, cramers_v, chance)¶

Normalized Integrated Performance Degradation over Cramér’s V.

accuracies holds one row per sampled Cramér’s-V value and one column per repeated training run. Row 0 is the balanced baseline at V=0. We first average balanced accuracy across repeats at each sampled value, then express the change from baseline as a fraction of baseline skill (accuracy above chance):

g(V) = (mean_accuracy(V) - mean_accuracy(0)) / (mean_accuracy(0) - chance)

nIPD is the signed area under g on [0, 1], estimated by trapezoidal integration over the supplied Cramér’s-V coordinates. Thus 0 means no degradation, increasingly negative values mean greater shortcut susceptibility, and -0.5 is the area of a linear fall from baseline performance at V=0 to chance performance at V=1.

Parameters:

accuracies (_Buffer | _SupportsArray[dtype[Any]] | _NestedSequence[_SupportsArray[dtype[Any]]] | complex | bytes | str | _NestedSequence[complex | bytes | str]) – (n_splits, n_iterations) balanced accuracies, row 0 the balanced baseline.
cramers_v (_Buffer | _SupportsArray[dtype[Any]] | _NestedSequence[_SupportsArray[dtype[Any]]] | complex | bytes | str | _NestedSequence[complex | bytes | str]) – One finite, strictly increasing Cramér’s-V value per accuracy row, beginning at 0 and ending at 1.
chance (float) – Balanced-accuracy chance level, 1 / n_biological_classes.

Return type:

float

Returns:

The signed trapezoidal area under the normalized degradation curve.

Raises:

ValueError – If the accuracy matrix is malformed, the Cramér’s-V coordinates are malformed or do not span [0, 1], chance lies outside [0, 1), or mean baseline balanced accuracy is at or below chance.

For reproducibility, \(\bar a(V)\) is formed before normalization: nIPD therefore uses a ratio of repeat means. apd retains PathoROB’s mean-of-repeat-specific-ratios order because changing it would break faithfulness to the reference implementation.

Alignment¶

croma.expand_features_to_manifest(*, features, manifest, embedding_manifest)¶

Return type:: ndarray

Result types¶

class croma.types.RobustnessResult(dataset, k, value, std, n_pairs, pair_values, sample_values, sample_values_aligned, occurrence_defined_mask, sample_undefined_types, occurrence_subsets, occurrence_source_indices, undefined_frac=0.0, ss_dominated_undefined_frac=0.0, oo_dominated_undefined_frac=0.0, mixed_undefined_frac=0.0, evaluation_design='all', evaluation_unit='sample', alpha=0.1, median_value=nan, q_alpha=nan, ltm_alpha=nan, tau=nan)¶

Bases: object

dataset: str¶

k: int¶

value: float¶

std: float¶

n_pairs: int¶

pair_values: ndarray¶

sample_values: ndarray¶

sample_values_aligned: ndarray¶

occurrence_defined_mask: ndarray¶

sample_undefined_types: ndarray¶

occurrence_subsets: ndarray¶

occurrence_source_indices: ndarray¶

undefined_frac: float = 0.0¶

ss_dominated_undefined_frac: float = 0.0¶

oo_dominated_undefined_frac: float = 0.0¶

mixed_undefined_frac: float = 0.0¶

evaluation_design: str = 'all'¶

evaluation_unit: str = 'sample'¶

alpha: float = 0.1¶

median_value: float = nan¶

q_alpha: float = nan¶

ltm_alpha: float = nan¶

tau: float = nan¶

class croma.types.CRoMaResult(dataset, m, value, std, n_pairs, pair_values, sample_values, sample_values_aligned, occurrence_defined_mask, undefined_frac, evaluation_design='all', evaluation_unit='sample', occurrence_subsets=None, occurrence_source_indices=None, k_start=0, k_final=0, retries=0, alpha=0.1, q_alpha=nan, ltm_alpha=nan, f0=nan)¶

Bases: object

One CRoMa evaluation: the pooled margin, its distribution, and its tail.

f0 is the confounder-dominant fraction \(F(0)\) – the empirical CDF of the per-sample margin at zero, i.e. the fraction of defined evaluation units whose margin is <= 0. Exact zero counts as confounder-dominant; undefined units are excluded from the denominator, as they are for q_alpha and ltm_alpha, and are reported separately by undefined_frac. It is nan when nothing is defined.

dataset: str¶

m: int¶

value: float¶

std: float¶

n_pairs: int¶

pair_values: ndarray¶

sample_values: ndarray¶

sample_values_aligned: ndarray¶

occurrence_defined_mask: ndarray¶

undefined_frac: float¶

evaluation_design: str = 'all'¶

evaluation_unit: str = 'sample'¶

occurrence_subsets: ndarray | None = None¶

occurrence_source_indices: ndarray | None = None¶

k_start: int = 0¶

k_final: int = 0¶

retries: int = 0¶

alpha: float = 0.1¶

q_alpha: float = nan¶

ltm_alpha: float = nan¶

f0: float = nan¶