Players, actions, and payoffs: the formal skeleton
A game in normal form is a triple (N, A, u): players N = {1, …, n}, an action set A_i per player, and a payoff u_i : A_1 × … × A_n → ℝ scoring each joint profile a = (a_1, …, a_n). Write a_-i for ‘everyone but i’, so u_i(a_i, a_-i) is i’s payoff given what the rest do.
For LLM agents the action set is astronomically large (every possible message is an action), so we never enumerate A_i. We abstract: in a negotiation the action is an offer, in a debate a claim plus argument, in a vote a ballot. A mixed strategy σ_i is a distribution over actions, mapping naturally onto sampling at temperature > 0, with expected payoff u_i(σ) = Σ_a (∏_j σ_j(a_j)) · u_i(a).
Nash equilibrium and best response
The central solution concept is the Nash equilibrium: a profile σ* where no player gains by deviating alone. Formally, for every i and every alternative σ_i, u_i(σ_i*, σ_-i*) ≥ u_i(σ_i, σ_-i*). Each agent plays a best response to the others, so the profile is self-enforcing: nobody regrets their choice.
This is the outcome a well-tuned multi-agent system converges to, and it need not be the one you wanted. If a debate’s scoring rewards confident assertion over calibrated hedging, the equilibrium is two agents over-claiming, because unilaterally hedging loses. Nash’s theorem guarantees at least one mixed equilibrium exists in any finite game, but says nothing about it being good, unique, or reachable. Designing payoffs so the equilibrium coincides with the behavior you want is the whole task: the subject of mechanism design below.
The negotiation game: bargaining and its equilibria
Put two LLM agents in a buyer-seller negotiation and you have a bargaining game. The surplus is the gap between the seller’s floor and the buyer’s ceiling; agents alternate offers, and each round of delay shrinks the pie by a discount factor δ < 1 (impatience, or the risk the deal dies). Rubinstein’s result says alternating-offer bargaining has a unique subgame-perfect equilibrium in which the first mover’s share is 1 / (1 + δ).
Agents must discover that a credible outside option and patience move the split: an agent that can walk away, or discounts the future less, commands a larger share. Prompting a model to ‘be a tough negotiator’ without giving it these levers is theater; the equilibrium is set by the payoff structure, not the persona. Negotiation benchmarks are really tests of whether a model reasons about the opponent’s incentives.
Debate as a two-player zero-sum game
AI debate frames truth-seeking as a game: two agents argue opposite sides before a judge who declares a winner. Because one’s win is the other’s loss, it is zero-sum: u_1(a) + u_2(a) = 0 for every outcome. Two-player zero-sum games are the best-behaved case in game theory: the minimax theorem gives a unique equilibrium value max_σ1 min_σ2 u_1(σ), a well-defined ‘correct’ value for the game.
The hope behind debate is that when a bounded judge watches two strong adversaries, the equilibrium strategy is to tell the truth, because a lie creates an opening the opponent exploits. That holds only if the judge can recognize a decisive rebuttal and the debate is long enough to expose sophistry. When those fail, the equilibrium can reward whichever falsehood is hardest to refute in the allotted turns: a persuasive-but-wrong Nash outcome, only as good as the judge’s verdict.
Self-play and fictitious play
Where do equilibrium strategies come from? Rarely from solving the game in closed form — the action space is too large. Agents learn them by playing against copies of themselves. Self-play is the engine behind superhuman game AI, and it transfers to LLMs: generate dialogues where the model argues or critiques itself, score the outcomes, and reinforce the winners.
A clean model of this is fictitious play: each agent assumes its opponent plays the empirical average of everything it has done, and best-responds to that belief. In two-player zero-sum games this converges to the minimax equilibrium, giving self-play on debate a firm footing. The catch is that self-play optimizes against the opponent it trains on; a model at equilibrium against copies of itself can still be exploited by a differently-distributed human. Robustness comes from population-based training against a diverse pool of opponents, not a single mirror match.
Regret minimization and correlated equilibria
The practical route to equilibrium in large games is no-regret learning. Define an agent’s average external regret after T rounds as R_i(T) = (1/T) · max_a Σ_t [ u_i(a, a_-i^t) − u_i(a_i^t, a_-i^t) ]: how much better it could have done, in hindsight, with a single best fixed action. An algorithm is no-regret if R_i(T) → 0 as T → ∞.
The key theorem: if every player uses a no-regret algorithm, the empirical distribution of joint play converges to the set of coarse correlated equilibria, a relaxation of Nash where a shared signal can correlate actions. This matters for LLM ensembles because you need not solve the game; each agent adapts with a simple regret-matching update and the collective drifts toward a stable outcome. In two-player zero-sum, the time-average converges all the way to Nash, tying regret minimization back to debate.
The consensus game: generation as equilibrium search
A striking recent idea treats a model’s output as the equilibrium of a game it plays with itself. In the consensus game (Jacob et al., 2023), a generator proposes an answer and a discriminator judges whether it is consistent with the question, and the two are trained to agree. A model can answer generatively (sample a response) or discriminatively (score candidates); these channels often disagree.
The consensus game defines a payoff rewarding a consistent, calibrated answer, then searches for the equilibrium with a no-regret procedure. The result reconciles both channels into one answer they endorse, and empirically it beats either alone. The deeper lesson is conceptual: ‘decoding’ need not be greedy or token-by-token. It can be cast as finding a fixed point where the model’s ways of assessing an answer stop contradicting each other, which is exactly what an equilibrium is.
Cooperation, defection, and the LLM prisoner dilemma
Not every setting is zero-sum. Many are mixed-motive, and the canonical warning is the prisoner’s dilemma: two agents choose cooperate or defect, mutual cooperation beats mutual defection, yet defection dominates for each individual, so the unique Nash equilibrium is the outcome both dislike. When LLM agents share a rate limit or a reward budget, this structure appears.
Say two agents decide whether to spend expensive tool calls on a shared task. Payoffs: mutual restraint yields (3, 3); one over-consuming while the other restrains, (5, 0); mutual over-consumption, (1, 1). Each agent’s best response to either opponent action is to over-consume — 5 > 3 and 1 > 0 — so both defect and get 1, worse than the available 3. The escape is repetition: in an iterated game, tit-for-tat makes cooperation an equilibrium, because today’s defection is punished tomorrow. Build long-lived agent systems as repeated games, not one-shots.