Research review / research

An Agent Hallucination Fix Weakens When the Action Format Changes

A November 13 preprint, surfaced in the November 16 research announcements, finds that an intervention for invalid web actions transfers imperfectly when the action and response formats change.

By Sandeep Belgavi · Published 2026-11-16

One legal action set is rendered as JSON or a numbered list and passed through the same intervention before separate evaluations.
Original conceptual diagram by Sandeep Belgavi: compare interface formats with the intervention held fixed. It illustrates the question, not measured timing or a reproduced result.

A reliability intervention can depend on the interface around the model. A preprint submitted November 13, 2026, included in the November 16 arXiv announcements, studies web agents that reference actions absent from their offered choices. Its useful finding is a transfer problem: changing the representation weakens a previously effective correction.

Detecting a mistake and correcting it are different tests

Aghzal, Geva, Stein and Yao examine Qwen2.5-7B-Instruct on Online-Mind2Web. Their paper reports 300 episodes and 2,071 actions, with 32.6% referencing an unavailable action. A probe at the last prompt token reaches 0.80 AUC before generation. These figures describe the authors' setup, not a general rate for web agents.

A detector ranks cases by estimated risk. A correction changes what the model does. A complete evaluation therefore needs separate questions: can a monitor distinguish invalid from valid actions, can an intervention repair invalid choices, and how often does it damage choices that were already valid? A high score for the first question cannot supply the answer to the others.

The same vector meets a different format

In the paper's Table 6, a fixed steering vector at strength 40 corrects 85.4% of invalid actions in the original format and 58.1% after a format change. Both conditions break 7.0% of previously valid actions. JSON actions become a numbered Markdown list, and the required response format also changes. This does not isolate input formatting alone.

The denominators matter. Correction is measured over an invalid subset; damage is measured over a previously valid subset. Neither number is the percentage of complete tasks solved. Combining them into a single headline improvement would hide both the population being helped and the population being harmed. A task can also fail after a locally valid action for reasons unrelated to this intervention.

Version the interface with the intervention

The engineering implication is to treat a monitor and its action schema as a coupled configuration. Record the offered-action renderer, the required output syntax and the intervention version together. A seemingly cosmetic prompt edit should trigger a transfer check if it changes one of those components. This is a proposed operational rule derived from the study, not a guarantee about another model.

There is also a distinction between recognizing a legal identifier and preserving the intent behind it. A format conversion should retain stable identifiers, ordering rules and the mapping from each displayed choice to its executable action. If that mapping changes, the test no longer asks whether the same intervention transfers across presentations of the same choice set.

A synthetic transfer test with separate outcomes

Proposed exercise, not executed: create a local interface offering only inspect_node and toggle_flag. Give each action an immutable identifier and a harmless deterministic effect. Present identical choices once as JSON and once as a numbered list. Use separate cases for changing only the input renderer, only the output contract, and both. This design separates factors that the cited transfer comparison changes together.

Freeze the model and intervention, preserve the original responses, and score legality before executing anything. Then classify the paired outcomes: invalid becomes valid, valid becomes invalid, or the action stays in its original category. Keep end-task success in a separate column. A refusal is another outcome to record, rather than silently treating it as a corrected action.

For a small teaching fixture, all effects can remain in memory. The interface validator can reject identifiers outside the offered set independently of the model's internal intervention. That validation tests the execution contract; it does not prove that the model chose the most useful legal action. Both checks are valuable because they answer different questions.

One setup is not a transfer guarantee

The cited results concern one model and evaluation setup. This review did not reproduce them, and a paper-specific code release was not established in the checked primary record. The probe's predictive score does not by itself explain the whole causal mechanism. The practical lesson is narrower and actionable: preserve the interface version, measure correction and damage separately, and rerun those measurements when the action contract changes.

Sources accessed November 16, 2026. Views are my own and do not represent any employer, past or present.

Sources and evidence

A reliability intervention has an interface contract. Test changes to action and response formats separately, and retain distinct measures for correction, damage and task completion.