CalibForge tunes 5,431 terminal tasks to solver difficulty, lifting agent benchmarks by 30 points
A new preprint introduces CalibForge, a system that synthesizes executable terminal tasks by calibrating them against solver behavior, yielding up to 30-point benchmark gains on agent training datasets.
CalibForge is a terminal-task synthesis system from researchers at Renmin University and the University of Chinese Academy of Sciences that generates training data by tuning task difficulty to match solver capability zones. Posted to arXiv on August 7, the preprint describes two calibration strategies—multi-solver and contrastive—that revise candidate tasks until they land in a "learnable zone" where some solvers succeed and others fail.
The system produced 5,431 calibrated tasks. Models trained on the full collection scored 32.58% and 47.57% on Terminal-Bench 2.0, with the largest improvement over baseline reaching 24.71 percentage points on Terminal-Bench 2.0, 27.68 points on SWE-bench Pro, and 30.04 points on Doc2Repo.
What stands out
- 01Multi-solver calibration targets disagreement within a heterogeneous solver pool—tasks are revised until different solvers produce conflicting pass/fail outcomes, signaling that the task sits at a useful difficulty threshold.
- 02Contrastive solver calibration enforces a strong-pass/weak-fail relation: a stronger solver must succeed while a weaker one fails. This operationalizes a solver-relative learnable zone anchored in demonstrated solvability.
- 03Both strategies outperform baseline pipelines. Ablation results show multi-solver and contrastive calibration beat authoring-and-validation alone or single-solver feedback loops. Executable validation establishes feasibility but does not reveal whether a task is appropriately challenging for learning.
- 04Benchmark transfers are substantial. The 30-point gain on Doc2Repo and 27-point gain on SWE-bench Pro suggest that solver-calibrated tasks generalize beyond the terminal domain they were synthesized for.
