GEM reformulates LLM data curation on the hypersphere, lifts downstream accuracy 1.2%

Pre-training large language models has become less about piling on tokens and more about getting the mix right—and a team of researchers now argues that the geometry of that mix matters as much as the categories themselves.

GEM (Geometric Entropy Mixing), introduced in a preprint on arXiv this week, is a data curation framework that treats corpus composition as a variational problem on the hypersphere rather than in flat Euclidean space. The authors contend that human taxonomies suffer from "ontological misalignment" and that standard clustering methods collapse under embedding anisotropy—the tendency of neural embeddings to occupy narrow cones rather than spread uniformly. By reformulating the objective with a mixing-balance regularizer and solving it via a provable Minorize-Maximize algorithm, GEM discovers balanced semantic structures that Euclidean heuristics miss. The framework decouples the generative prior from the optimization step, which the paper claims prevents the cluster collapse that plagues k-means and similar approaches.

To scale the method to web-scale corpora, the authors use teacher-student distillation and introduce the Geometric Influence Score (GIS), a metric for interpretable taxonomy generation. Experiments on 1.1B-parameter models show that GEM integrated into existing mixing strategies like DoReMi and RegMix lifts average downstream accuracy by up to 1.2 percentage points. The preprint frames the hypersphere as a "robust coordinate system for predictable data mixing," positioning geometric fidelity as a missing ingredient in current data pipelines.

The authors do not report wall-clock training cost comparisons or release code alongside the preprint, so reproducibility and practical overhead remain open questions for practitioners evaluating the approach.