No. 034

PaperRaven, LLM peer review framing, AlphaGenome 2

A common design question appears across several items this week: at which point in the research pipeline should AI output be checked by an independent process? PaperRaven addresses it at the literature stage by refusing any citation the model proposes but cannot locate in OpenAlex or Crossref. The DIME team's code-review agent addresses it at the analysis stage, prompted by an audit finding coding errors in 26 of 110 published economics papers despite most being computationally reproducible. A NeurIPS benchmark addresses it at the review stage and finds that model selection alone produces a 50-percentage-point gap in error detection rates. The LLM framing experiment shows what is at stake when that check is absent: a single prompt change raised aggregate acceptance rates more than fourfold on the same set of published papers.

Literature and Data Infrastructure

Code Quality and Review

  • AI is writing more research code. Who is reviewing it?

    World Bank Development Impact (Ankriti Singh, MarĂ­a Reyes Retana Torre, Maria Jones), November 3 2026

    In an audit of 110 published economics and political science articles, 26 contained substantive coding errors despite 85 percent being computationally reproducible, and the team has released a public agent skill of more than 60 checks that runs a read-only code review in two phases and reports findings by severity.

  • Auditing AI peer reviewers: a benchmark on real scientific papers

    arXiv (Zubeldia, Bolliet, Villaescusa-Navarro, Villanueva-Domingo, Cranmer), November 12 2026

    Using 357 deliberate errors of four types injected into real astrophysics and machine learning papers, the benchmark found that model choice has the largest effect on detection rate: across the same papers and prompts, three frontier LLMs differed by roughly 50 percentage points in total detection.

Peer Review and Verification

  • Down the Rabbit Hole of LLM Peer Review

    Radiology: Artificial Intelligence (Bastien Le Guellec, Su Hwan Kim), November 11 2026

    A single sentence telling the model a manuscript had already been revised once raised aggregate acceptance rates from 4 to 18 percent across four models on 200 published papers, and Gemini 3.7 Flash's rate from 15.5 to 68.5 percent, leading the authors to argue that a reviewer must be accountable for the experiments it requests and that an LLM cannot meet that standard.

  • AI's ability to 'autoformalize' proofs is changing math

    Quanta Magazine (Konstantin Kakaes), November 11 2026

    Anthropic's agents formalized Fermat's Last Theorem in 30 hours, compressing what required 11 days in September, and formalization expert Kevin Buzzard now expects formalization of new results on the fly, shifting the question of mathematical correctness from peer review to machine-checkable verification.