PAJAMA distills LLM judges into programs, cuts eval cost by 100×

PAJAMA, a system from researchers at UC Berkeley and the University of Wisconsin, distills LLM-as-a-judge decision logic into executable programs that score model outputs directly. The approach eliminates per-sample API costs and latency while maintaining transparency — the resulting programmatic judges can be inspected, edited, and run locally. Across five datasets and four model families, the system matches the accuracy of a 13B-parameter LLM judge.

The core idea is program synthesis: instead of prompting a large model at evaluation time, PAJAMA generates a committee of small programs that collectively replicate the judge's scoring behavior. When confidence is low, the system escalates cases to a fallback LLM, creating a hybrid routing mechanism that improves both accuracy and throughput. On RewardBench, a reward model trained on verdicts from these programmatic judges outperformed one trained on proprietary LLM labels at 1/100th the API cost.

LLM-as-a-judge has become the default for automated evaluation in NLP and multimodal tasks, but the cost and opacity of repeated inference calls limit its use at scale. PAJAMA's distillation path offers a concrete alternative: after an initial synthesis phase, the programs run without network calls or token budgets. The paper reports that program outputs serve as effective routing signals, advancing the Pareto frontier for accuracy versus compute.

The next step is broader model coverage. The current evaluation spans five datasets, but production use cases often require domain-specific rubrics or multi-turn conversation scoring. Watch for extensions that handle structured outputs, multi-modal inputs, or safety-specific criteria — areas where transparent programmatic logic could replace expensive judge ensembles entirely.