Segmentation Assisted Incremental Test Time Adaptation in an Open World

Manogna Sreenivas     Soma Biswas

Indian Institute of Science, Bengaluru, India
British Machine Vision Conference (BMVC), 2025    

Abstract

In dynamic environments, unfamiliar objects and distribution shifts are often encountered which challenge the generalization abilities of the deployed trained models. This work addresses Incremental Test Time Adaptation (ITTA) of Vision-Language Models (VLMs), tackling scenarios where unseen classes and unseen domains continuously appear during testing. Unlike traditional Test Time Adaptation approaches where the test samples come only from a predefined set of classes, our framework allows models to adapt simultaneously to both covariate and label shifts, actively incorporating new classes as they emerge. Towards this goal, we establish a new benchmark for ITTA, integrating single-image TTA methods for VLMs with active labeling to query an oracle for samples potentially representing unseen classes during test time. We propose a segmentation assisted active labeling module, termed SegAssist, which is training-free and repurposes the VLM’s segmentation capabilities to refine active sample selection, prioritizing samples likely to belong to unseen classes. Extensive experiments on several benchmark datasets demonstrate the potential of SegAssist to enhance the performance of VLMs in real-world scenarios, where continuous adaptation to emerging data is essential.

Incremental Test Time Adaptation

ITTA framework
ITTA framework: We address the problem of Incremental Test-Time Adaptation (ITTA) in an open-world using VLMs, where both covariate and label shifts can occur. The task is to predict a label for each incoming test sample while also adapting to distributional shifts and dynamically incorporating new classes as they emerge. This is done by integrating active labeling with TTA.

SegAssist for Active Sample Selection

SegAssist
SegAssist framework: A plug-in module which can be integrated with existing TTA and active labeling methods. It prioritizes unseen class samples by performing pixel wise segmentation of uncertain samples, selecting it for active labeling only if the pixels are predominantly classified as “background”.

Incremental Class Detection Delay (ICDD) Metric

ICDD Metric
ICDD Metric: We design Incremental Class Detection Delay, a metric to quantify the performance of an ITTA system in identifying new classes in a timely manner. It compares the rate of new class introductions in the ground truth sequence with the rate of class detections by the ITTA framework, visualized as a normalized Area Under the Curve (AUC) plot.

Experimental Results

results
Table: Comparison of SegAssist with prior methods in ITTA framework.

BibTeX


            @article{sreenivas2025segassist,
              title={Segmentation Assisted Incremental Test Time Adaptation in an Open World},
              author={Sreenivas, Manogna and Biswas, Soma},
              journal={British Machine Vision Conference},
              year={2025}
            }