Explorative Modeling pretrains generative models 4.1× faster by exploring K candidates per step

Explorative Modeling is a new training paradigm from Alexi Gladstone, Heng Ji, and Yilun Du that factors the training loop rather than the generation procedure, exploring K candidate matches between model generations and data and training on the best. The arXiv preprint, posted this week, argues that generative models have never been truly end-to-end trained because existing scalable approaches factor the generation procedure itself—autoregressive models decompose into token-by-token steps, diffusion models into denoising steps—which prevents predictions from committing to a single mode and instead forces them to blur across many.

Explorative Models (XMs) flip that approach. Instead of factoring how the model generates, they factor how the model trains: at each step, the system explores K candidate outputs, picks the one closest to the ground truth, and backpropagates only through that match. The authors report that scaling exploration adds a third pretraining axis beyond parameters and data, with gains climbing from 7% to 36% as data scales and from 13% to 23% as models grow. Concretely, exploration improves FLOP efficiency by 4.1×, sample efficiency by 6.2×, and parameter efficiency by 47%. The strongest image-generation recipe in the paper reaches 1.43 FID on ImageNet without guidance, near state-of-the-art, and XMs match diffusion on control tasks with 16–256× fewer inference steps.

The paper tests XMs across continuous and discrete domains—images, video, and language—and finds that efficiency gains more than double at 3× the compute. The authors frame exploration as both a drop-in pretraining boost for existing generative models and a standalone end-to-end paradigm for reconstructive generation. The preprint includes ablations showing that gains from exploration increase with scale, suggesting the axis compounds with standard scaling laws rather than saturating.

The next question is whether XMs hold up on larger-scale pretraining runs and whether the K-candidate exploration overhead remains practical at frontier model sizes. The paper's ImageNet results are strong but stop short of diffusion state-of-the-art with guidance, and the language experiments are preliminary. If the efficiency claims replicate at scale, XMs could reshape how labs allocate compute between data, parameters, and now exploration.