CiteTracer reaches 97% accuracy detecting LLM-fabricated citations
Researchers introduce a multi-agent framework that classifies citations into Real, Potential, and Hallucinated categories, achieving 97% accuracy on synthetic and real-world fabrications from ICLR 2026 submissions.
CiteTracer, a cascading multi-agent detector from Mingzhe Li, Zhiqiang Lin, and Shiqing Ma, identifies fabricated citations in LLM-assisted scientific writing. The system extracts structured citations from PDF and BibTeX files, retrieves evidence through cache lookup, URL fetch, scholar connectors, and web search, applies deterministic field matching, and routes ambiguous cases to class-specialist judgers. Unlike binary found/not-found detectors, CiteTracer classifies citations into a 12-code taxonomy spanning Real, Potential, and Hallucinated categories, giving auditors field-level signal on what specifically fails verification—title, author list, venue, or year.
The researchers released a benchmark of 2,450 synthetic citations built from real seeds with controlled LLM mutations, paired with 957 real-world fabrications drawn from ICLR 2026 and desk-rejected submissions. On the synthetic benchmark, CiteTracer reaches 97.1% accuracy, with class-level F1 scores of 97.0 for Real, 95.8 for Potential, and 98.5 for Hallucinated citations. On the real-world fabrication set, the system detects 97.1% of fabricated references without abstaining, offering a concrete tool for conference organizers and journal editors managing LLM-generated submissions. Code and benchmarks are available on GitHub.