Composite¶
This notebook walks through soma’s composite encoder — concatenating the dense outputs of several frozen foundation models into one richer per-position vector before a single decoder consumes the stack:
Dataset(+masks) -> [encoder A grid] ┐
├-> CompositeDenseFeatureStore -> train (decoder + head) -> evaluate
[encoder B grid] ┘
It builds directly on the dense-prediction walkthrough: the flow is identical, but instead of one DenseFeatureStore from a single encoder we extract two members independently and present a load-time channel-concat view (CompositeDenseFeatureStore) to train. The decoder sees a single wider grid and is otherwise unchanged — this is the `composite: <../encoders/composite.rst>`__ path the YAML config exposes, written out as building blocks.
Tiny synthetic data, runs on CPU, ungated encoders — the numbers are meaningless; the point is the API. We concatenate two public 224 px / patch-16 encoders —
`phikon<https://huggingface.co/owkin/phikon>`__ and`hibou-b<https://huggingface.co/histai/hibou-b>`__ — at their native window (a 14×14 token grid each), which avoids position-embedding interpolation.
⚠️ Scaffolding (not soma API)¶
As in the dense walkthrough, dense supervision lives in per-sample files, not a scalar label: dataset.csv carries sample_id, image_path, mask_path, where the mask is an integer-class raster the same size as the ROI.
We fabricate small 224 px ROI tiles (the dense flow consumes fixed-size tiles/ROIs, not whole WSIs) plus their masks. Nothing here is part of soma — it is just enough scaffolding to have something to feed the encoders.
[ ]:
import logging, warnings
warnings.filterwarnings('ignore')
logging.getLogger().setLevel(logging.ERROR)
import tempfile
from pathlib import Path
import numpy as np
import pandas as pd
import tifffile
from PIL import Image
WORK = Path(tempfile.mkdtemp(prefix='soma-composite-tutorial-'))
ROIS = WORK / 'rois'; MASKS = WORK / 'masks'
for d in (ROIS, MASKS): d.mkdir()
rng = np.random.default_rng(0)
SIZE = 224 # phikon / hibou-b native window
SPACING = 0.5 # microns/pixel (both members read at the same spacing in v1)
NUM_CLASSES = 3 # 0 = background, 1, 2 = tissue classes
def make_roi(path):
img = np.clip(np.stack([np.full((SIZE, SIZE), 150),
np.full((SIZE, SIZE), 70),
np.full((SIZE, SIZE), 160)], -1).astype(np.int16)
+ rng.integers(-30, 30, (SIZE, SIZE, 3)), 0, 255).astype(np.uint8)
tifffile.imwrite(path, img, photometric='rgb', tile=(SIZE, SIZE),
resolution=(20000, 20000), resolutionunit='CENTIMETER')
def make_mask(path):
m = np.zeros((SIZE, SIZE), np.uint8)
m[SIZE // 4:SIZE // 2, SIZE // 4:SIZE // 2] = 1
m[SIZE // 2:3 * SIZE // 4, SIZE // 2:3 * SIZE // 4] = 2
Image.fromarray(m).save(path)
ids = [f'roi{i:02d}' for i in range(8)]
for sid in ids:
make_roi(ROIS / f'{sid}.tif')
make_mask(MASKS / f'{sid}.png')
split = ['train'] * 4 + ['tune'] * 2 + ['test'] * 2
splits_csv = WORK / 'splits.csv'
pd.DataFrame({'sample_id': ids, 'split': split, 'fold': 0}).to_csv(splits_csv, index=False)
img_paths = [str(ROIS / f'{s}.tif') for s in ids]
# Feature extraction only needs the images; supervision lives in the mask manifest.
extract_csv = WORK / 'extract.csv'
pd.DataFrame({'sample_id': ids, 'image_path': img_paths,
'label': 0}).to_csv(extract_csv, index=False)
seg_csv = WORK / 'seg.csv'
pd.DataFrame({'sample_id': ids, 'image_path': img_paths,
'mask_path': [str(MASKS / f'{s}.png') for s in ids]}).to_csv(seg_csv, index=False)
print('segmentation manifest:')
print(pd.read_csv(seg_csv).head(3).to_string(index=False))
1. Extract each member into its own dense store¶
A composite has no separate cache: each member is extracted independently with DenseTileFeatureExtractor (exactly as in the single-encoder dense flow) and cached on its own. We run the same ROIs through phikon and hibou-b, writing each member’s grids to its own feature directory. Both emit a 14×14 token grid here, but they need not — members may differ freely in patch size / grid; the composite reconciles them at load time.
[ ]:
from soma import (
Dataset, DenseTileFeatureExtractor, EncoderConfig, CacheConfig,
)
MEMBERS = ['phikon', 'hibou-b']
member_stores = []
for name in MEMBERS:
extractor = DenseTileFeatureExtractor(
Dataset(extract_csv),
EncoderConfig(name=name),
target_size=SIZE,
spacing_um=SPACING,
backend='openslide',
cache=CacheConfig(enabled=False),
)
store = extractor.run(str(WORK / 'dense' / name))
member_stores.append(store)
print(f'{name}: feature_dim={store.feature_dim}, samples={len(store.available_samples)}')
2. Concatenate into a CompositeDenseFeatureStore¶
CompositeDenseFeatureStore is a thin load-time view over the per-member stores — it does not re-extract or re-cache anything. For the decoder / detection paths the natural choice is concat_resolution='grid': each member’s native grid is resampled to a common token grid, then channels stack into (Σdᵢ, h, w) for the decoder to consume. (The decoder-free pixel classifier instead uses 'target', concatenating at full mask resolution.)
member_norms per-member normalizes before concat so a large-magnitude encoder cannot dominate. The concatenated feature dim is the sum of the members’.
[ ]:
from soma.dense.composite import CompositeDenseFeatureStore
composite_store = CompositeDenseFeatureStore(
member_stores,
concat_resolution='grid', # auto for the decoder / detection paths
member_norms=['none', 'l2'], # one per member, aligned with MEMBERS
)
print('composite feature_dim:', composite_store.feature_dim,
'=', ' + '.join(str(s.feature_dim) for s in member_stores))
print('shared samples:', len(composite_store.available_samples))
3. Train segmentation on the composite¶
From here the flow is identical to the single-encoder dense walkthrough: the composite store drops into train wherever a DenseFeatureStore would go. The decoder simply sees a wider per-position vector; nothing else changes.
[ ]:
from soma.dataset import SegmentationManifest
from soma import (
Splits, DecoderConfig, TaskConfig, TrainingConfig, EvalConfig,
PreprocessingConfig, train,
)
seg_manifest = SegmentationManifest(seg_csv)
seg_splits = Splits(splits_csv, seg_manifest)
seg_result = train(
feature_store=composite_store,
dataset=seg_manifest,
splits=seg_splits,
dataset_type='segmentation',
decoder=DecoderConfig(name='lightweight_conv'),
task=TaskConfig(name='segmentation', params={'num_classes': NUM_CLASSES}),
training=TrainingConfig(epochs=3, batch_size=2, learning_rate=1e-3, seed=0),
evaluation=EvalConfig(metrics=['mean_dice', 'mean_iou']),
# our PNG masks carry no spacing metadata; declare the ROI spacing so they
# register against the grids extracted at the same spacing.
preprocessing=PreprocessingConfig(requested_spacing_um=SPACING),
run_dir=str(WORK / 'runs' / 'composite-segmentation'),
)
print('composite segmentation run dir:', seg_result.run_dir)
4. The one-shot Pipeline equivalent¶
The same composite, driven from a single config. The members live under a composite: block (which is XOR the single encoder:); the pipeline extracts each member and builds the concat view for you. concat_resolution auto-resolves to grid on the decoder path, so it can be left unset.
(Shown for reference, not executed.)
from soma import (
Pipeline, PipelineConfig, PreprocessingConfig, CompositeConfig, EncoderMemberConfig,
DecoderConfig, TaskConfig, TrainingConfig, EvalConfig, CacheConfig,
)
config = PipelineConfig(
dataset_csv=str(seg_csv),
splits_csv=str(splits_csv),
output_root='output/composite-segmentation',
dataset_type='segmentation',
preprocessing=PreprocessingConfig(
backend='openslide', requested_tile_size_px=224, requested_spacing_um=0.5,
),
composite=CompositeConfig(encoders=[ # XOR `encoder=`
EncoderMemberConfig(name='phikon'),
EncoderMemberConfig(name='hibou-b', member_norm='l2'),
]),
decoder=DecoderConfig(name='lightweight_conv'),
task=TaskConfig(name='segmentation', params={'num_classes': 3}),
training=TrainingConfig(epochs=3, batch_size=2, learning_rate=1e-3),
evaluation=EvalConfig(metrics=['mean_dice', 'mean_iou']),
cache=CacheConfig(enabled=True),
)
results = Pipeline(config).run()