Memory-Supported Synergistic Adaptation for Training-Free Test-Time Medical Image Segmentation
Abstract
Test-time adaptation (TTA) aims to mitigate distribution shifts by adapting models with unlabeled target data at inference time. While TTA with vision-language models (VLMs) has shown promising results in classification, extending it to medical image segmentation remains challenging. In this setting, the adaptation gains from optimizing on VLM-generated predictions are often outweighed by the degradation to the VLM's strong pretrained features caused by noisy, update-driven learning, resulting in limited and unstable improvements.
We therefore propose Memory-Supported Synergistic Adaptation (MSSA), a novel training-free TTA framework for medical image segmentation. Without updating model parameters, MSSA dynamically selects reliable image–text predictions to construct an online memory, uses them as text-guided semantic priors, and couples them with cross-image structural alignment for robust adaptation. Specifically, MSSA consists of (i) a noise-aware memory construction module that filters and stabilizes cross-modal predictions, and (ii) a relevance-driven prototype alignment module that aligns the target sample with structurally consistent memory samples and their reliable predictions to improve adaptation.
Extensive experiments on multiple medical segmentation benchmarks demonstrate that MSSA consistently improves VLM-based segmentation models and outperforms existing fine-tuning-based TTA methods by a clear margin, with gains of up to 12.2% DSC and 11.7% mIoU.
Method Overview
MSSA operates in a synergistic two-stage pipeline without any parameter updates or backpropagation:
(a) Image-Text Coarse Grounding: Given a test image and a text prompt generated by an LLM, BiomedCLIP produces a saliency map that is refined via SAM with Gaussian Point Selection and Dual Majority Voting (for both saliency maps and SAM outputs) to yield a stable candidate segmentation.
(b) Noise-Aware Memory Construction & Image-Image Prototype Matching: High-quality candidates are admitted into an online memory bank via a dual-criterion scoring mechanism combining semantic alignment (Qsem) and spatial smoothness (Qsmo). For each query image, the most feature-similar anchor is retrieved from the memory bank and used for non-parametric prototype matching via ALPNet, producing the final accurate segmentation.
Figure: The overall framework of MSSA. (a) Image-Text Coarse Grounding. (b) Noise-Aware Memory Construction and Image-Image Prototype Matching.
Qualitative Results
Qualitative comparison on Optic Disc segmentation across Domains A–E. MSSA produces anatomically coherent masks that closely match the ground truth.
Qualitative comparison on Lung segmentation (MC, SZ, COVID-QU-Ex). MSSA remains robust under substantial domain shifts and pathological variations.
Comparison of image-text only, image-image only, and our synergistic MSSA paradigm on the Optic Disc dataset. Neither modality alone achieves the performance of their combination.
Quantitative Results
On the Optic Disc segmentation benchmark, MSSA achieves an average DSC of 83.9% and mIoU of 73.1%, surpassing the best training-based TTA method (TTCS) by +12.2% DSC and the strongest zero-shot baseline (MedCLIPSAMv2) by +12.5% DSC.
On Lung segmentation datasets (COVID-QU-Ex, MC, SZ), MSSA achieves an average DSC of 78.4%, outperforming the best TTA baseline by over +15% DSC on the challenging COVID-QU-Ex dataset.
BibTeX
@inproceedings{li2026mssa,
title={Memory-Supported Synergistic Adaptation for Training-Free Test-Time Medical Image Segmentation},
author={Li, Lingrui and Pu, Nan and Zhao, Dong and Li, Wenjing and French, Andrew P and Chen, Xin and Zhong, Zhun},
booktitle={European Conference on Computer Vision (ECCV)},
year={2026},
url={https://lingrayy.github.io/MSSA/}
}