Six alignment methods reshape language models in fundamentally different ways

Alignment algorithms are typically judged by output quality alone, but a preprint posted to arXiv this week cracks open the black box to show how six popular methods—PPO, DPO, SimPO, ORPO, GRPO, and KTO—actually rewire language models under the hood. The paper, arXiv:2606.09850, applies layer-wise linear probing, Sparse Autoencoders, and crosscoders across three open-weight model families to map where preference signals land and how alignment reshapes the geometry of latent space. The finding: preference representations consistently concentrate in early-to-mid or mid-to-late layers, but the six objectives induce qualitatively different internal changes—behavioral alignment does not guarantee uniform internal restructuring.

KTO and GRPO emerge as the cleanest interventions, enhancing linear separability through constructive feature sharing and sparse, high-salience recruitment. DPO and ORPO, by contrast, degrade separability via non-constructive geometric rotation and feature attenuation—the model learns the desired behavior, but the internal representation becomes messier. PPO and SimPO largely preserve baseline geometry, suggesting they operate through different mechanisms entirely. The paper also documents architecture-dependent variability, meaning the same alignment objective can produce different internal effects depending on the model family.

The work frames alignment as a heterogeneous intervention rather than a monolithic process, with direct implications for safety auditing and interpretability. If two methods produce similar refusal behavior but one degrades internal feature structure while the other preserves it, the choice between them is no longer just about benchmark performance—it's about whether the model's internals remain legible to downstream analysis. The authors argue for standardized feature-level auditing and mechanism-aware optimization objectives, rather than treating alignment as a purely behavioral problem. What remains open is whether these geometric signatures hold across larger models, whether they predict jailbreak robustness or generalization failures, and whether future alignment methods can be designed to preserve or enhance internal structure by default. Replicating the analysis on 70B+ parameter models and testing whether the identified transformations correlate with real-world safety incidents would clarify whether mechanism-level insights can guide the next generation of alignment research.