SAE-FT preserves CLIP robustness while fine-tuning on downstream tasks

Fine-tuning CLIP models typically improves downstream task accuracy but erodes their robustness to distribution shifts—a trade-off researchers have struggled to reconcile without expensive text-guidance overhead.

SAE-FT, a new method from Fabian Morelli, Arnas Uselis, Ankit Sonthalia, and Seong Joon Oh, addresses that tension by operating exclusively on the visual encoder. The technique trains a Sparse Autoencoder on the pre-trained CLIP model's representations, then uses it to identify semantically meaningful features. During fine-tuning, SAE-FT penalizes the addition or removal of those features, constraining how much the model's internal representations can drift. That constraint prevents catastrophic forgetting—the phenomenon where a model loses its original capabilities—while keeping the process interpretable: practitioners can inspect which semantic features changed and by how much.

The approach matters because CLIP's zero-shot capabilities make it a foundation for dozens of downstream vision tasks, from object detection to image generation pipelines. When teams fine-tune CLIP on domain-specific datasets to boost accuracy, they often discover the model has lost its ability to generalize to new distributions—exactly the robustness that made CLIP valuable in the first place. Existing mitigation strategies frequently rely on text-tower guidance, which doubles compute cost and adds architectural complexity.

SAE-FT sidesteps that overhead. By working only on the visual representations and using a Sparse Autoencoder as a semantic lens, the method achieves mechanistic transparency: you can see which features the model is preserving and which it's modifying. On ImageNet and its associated distribution-shift benchmarks, SAE-FT matches or exceeds current state-of-the-art robust fine-tuning methods. The authors report computational efficiency gains over text-guided alternatives, since the text tower remains untouched during training.

Code and implementation details are available on GitHub. For practitioners running CLIP-based pipelines in production, the method offers a path to domain adaptation without sacrificing the robustness that justifies using a foundation model in the first place.