Three unrelated items this week have the same shape. Each makes a substantial claim, each is honest about a gap in the evidence, and in each the gap is what determines how much the claim should change your plans.
Three versions of the same gap
- Evidence you cannot see. METR's assessment of AI acceleration at Anthropic, published November 5, is an unusually detailed outside look at a frontier lab. METR also says Anthropic redacted a majority of the evidence it found most informative, and that the public must lean on METR's judgment "much more than would be ideal."
- Artifacts you cannot yet test. Thinking Machines' Inkling 2 Small arrived November 5 with a full table of benchmark scores and closed weights, pending external safety tests. The numbers exist before the object that would let anyone reproduce them.
- Conditions that change the meaning. The PwnEscape write-up on Firecracker demonstrates root on a host, and states plainly that the setup had no jailer and ran as root. A reader who stops at the headline and a reader who reaches that sentence come away with different risk assessments.
Why this is becoming normal
There are reasons, and they are not all bad. Labs have real competitive and security concerns about internal data, and METR's arrangement gave it more access than outsiders usually get, in exchange for limits on publication. Staged weight release gives safety testers time to work. Security researchers often withhold exploit details on purpose. A world with perfect disclosure would be more dangerous in some respects.
The cost falls on people making decisions downstream. Anthropic's amended S-1, covered here on November 3, tells investors the company may soon meet its own threshold for AI systems that speed up AI research. Policymakers, customers and investors now hold a chain of claims with a broken link they cannot inspect.
Research shows the same pressure at smaller scale. The PACT-Check work found some gains in cross-model links could be reproduced with uninformative signals. BestRand showed that a usually adequate benchmark subset can occasionally be far off. Neither would be visible without a deliberate control.
What remains uncertain. It is not clear whether third-party review with redactions will mature into something more checkable, such as audited summaries that a second reviewer can confirm. We also cannot yet say whether Inkling's weights will arrive on schedule or how outside tests will compare.
Analysis: Treat the limitation as the first thing to read and write it next to the claim in your own notes. When Geetha evaluates a vendor's agent for Belgavi's Book Store, three questions cover most of this: what evidence is withheld and why, which artifact can we run ourselves, and what condition was true in the test that is not true for us. A claim that survives all three deserves a budget line. The rest deserve a pilot.
Sources and evidence
- Independent Assessment of AI R&D Acceleration at Anthropic in October 2026 (METR, November 5)
- Introducing Inkling 2 Small (Thinking Machines, November 5)
- Firecracker VM Escape: PwnEscape Part 2 (pwn.ai)
- On the Opportunities and Challenges of Prefill-to-Autoregressive Cross-Model Transfer (arXiv, November 4)
METR's redactions, Inkling 2 Small's pending weights and PwnEscape's no-jailer condition show one pattern: major claims now ship with part of the evidence withheld, untestable or conditional. Disclosure limits can be reasonable, but the cost lands on people making decisions, so record what is withheld, what you can run yourself, and what test condition differs from yours.