HiLo: A Learning Framework for Generalized Category Discovery Robust to Domain Shifts

Generalized Category Discovery (GCD) is a challenging task in which, given a partially labelled dataset, models must categorize all unlabelled instances, regardless of whether they come from labelled categories or from new ones.

In this paper, we challenge a remaining assumption in this task: that all images share the same domain. Specifically, we introduce a new task and method to handle GCD when the unlabelled data also contains images from different domains to the labelled set.

Our proposed `HiLo' networks extract High-level semantic and Low-level domain features, before minimizing the mutual information between the representations. Our intuition is that the clusterings based on domain information and semantic information should be independent. We further extend our method with a specialized domain augmentation tailored for the GCD task, as well as a curriculum learning approach.

We construct a benchmark from corrupted fine-grained datasets as well as a large-scale evaluation on DomainNet with real-world domain shifts, reimplementing a number of GCD baselines in this setting. We demonstrate that HiLo outperforms SoTA category discovery models by a large margin on all evaluations.

Samples are drawn through our proposed curriculum sampling approach, considering the difficulty of each sample. Labelled and unlabelled samples are paired and augmented through PatchMix which we subtly adapt in the embedding space for contrastive learning for GCD. The mixed-up embeddings are then processed by our network with a high-level (for semantic) and low-level (for domain) feature design, allowing for the domain-semantic disentangled feature learning via mutual information minimization.

PatchMix augments the labelled and unlabelled data by mixing up these patches in the embedding space. We randomly sample from Beta distribution to control the proportion of patches from images. The confidence factor is determined by the overall proportion of known semantics in the mixed samples and the attention scores for all the patches of the input image, which is then assigned based on the similarity score or the actual label to guide the training.

We compare HiLo with previous state-of-the-art NCD/GCD methods and one domain adaptation method on DomainNet and our created corrupted fine-grained datasets (SSB-C). The results are shown below. We can see that our method consistently outperforms previous state-of-the-art methods.

The results on three corrupted fine-grained datasets are shown below.

Visualization of domain and semantic features via projecting them through PCA. We randomly sample instances from the entire dataset and apply PCA to project the semantic and domain features into a 2-dimensional space.

HiLo is much more effective in focusing on the foreground object even in the presence of significant domain shifts.