pSTarC: Pseudo Source Guided Target Clustering for Fully Test-Time Adaptation

Test Time Adaptation (TTA) is a pivotal concept in machine learning, enabling models to perform well in real-world scenarios, where test data distribution differs from training. In this work, we propose a novel approach called pseudo Source guided Target Clustering (pSTarC) addressing the relatively unexplored area of TTA under real-world domain shifts. This method draws inspiration from target clustering techniques and exploits the source classifier for generating pseudo-source samples. The test samples are strategically aligned with these pseudo-source samples, facilitating their clustering and thereby enhancing TTA performance. pSTarC operates solely within the fully test-time adaptation protocol, removing the need for actual source data. Experimental validation on a variety of domain shift datasets, namely VisDA, Office-Home, DomainNet-126, CIFAR-100C verifies pSTarC’s effectiveness. This method exhibits significant improvements in prediction accuracy along with efficient computational requirements. Furthermore, we also demonstrate the universality of the pSTarC framework by showing its effectiveness for the continuous TTA framework.

1. Feature Generation:

We randomly initialize a feature bank which is iteratively optimized keeping the classifier fixed to minimize the entropy of the features while maximizing the diversity across classes.

\begin{align} \mathbf{f}^{*} = arg\min_{\mathbf{f}} \mathcal{L}_{ent}(\mathbf{f}; h) + \beta \mathcal{L}_{div}(\mathbf{f}; h) \end{align}

where

\begin{align} \mathcal{L}_{e n t}\left(\mathbf{f} ; h\right) =-\frac{1}{N} \sum_{i=1}^N \sum_{k=1}^C p_k\log p_k; \quad\quad \mathcal{L}_{d i v}\left(\mathbf{f} ; h\right)=\sum_{k=1}^C \hat{p}_k \log \hat{p}_k \end{align}

2. Pseudo Source Guided Target Clustering:

Given the learnt features, we aim to bring the low entropy samples towards the corresponding pseudo-source features. We anchor the high entropy target samples to its own prediction. We also enforce consistency between the predictions of the test sample and its strong augmentation.

\begin{align} \mathcal{L}_{\textrm{pSTarC}}(x_k) =\underbrace{\vphantom{\sum_{p_{j}^{+}\in \mathbf{p}^{+}} p_{k}^{T}p_{j}^{+}}-p_{k}^{T}\tilde{p}_{k}}_\text{$L_{aug}$} - \underbrace{\sum_{p_{j}^{+}\in \mathbf{p}^{+}} p_{k}^{T}p_{j}^{+}}_{\text{$L_{attr}$}} + \underbrace{\vphantom{\sum_{p_{j}^{+}\in \mathbf{p}^{+}} p_{k}^{T}p_{j}^{+}}\lambda \sum_{x_{j}\in\mathbf{x}_t} p_{k}^{T}p_{j} }_\text{$L_{disp}$} \end{align}