LLM Abuse Prevention: Design a Gateway, Budgets, and Incident Response, in depth

Start with a product-specific abuse model

Start from your product's capabilities, not from a generic list of bad content. For each capability, write the harm it enables, the evidence that would show it, and which service owns the decision. A content classifier can judge a piece of text. It cannot prove that an account owner consented to contacting a recipient, or that a workload breaches a contract. Those decisions need the data that lives in your identity, billing and messaging services.

Also write down the legitimate neighbours of each abuse: a trust-and-safety team summarising scam reports, a load test, a classroom of students hitting the same endpoint. They define the false positives you must not create.

Order the gateway from cheap to expensive

Order the gateway from cheap to expensive so that most abuse is rejected before it costs inference. Authentication and tenant resolution come first and assign a request identifier that every later stage logs. Shaping checks request size, rate and concurrency. Budget reservation happens before the model call. Input screening runs with a versioned policy and returns allow, block or review. Output screening runs last.

Every stage needs a written timeout behaviour, because the default in most code is whatever the exception handler happens to do. A useful rule: shaping and budgets fail closed, since they protect your infrastructure; input screening fails closed for high-risk capabilities like outbound messaging and may fail to a reduced mode for low-risk ones like private summarisation. Streaming forces a choice at the output stage. Buffering the full response before release allows complete screening but adds latency; releasing chunks immediately means a later block cannot retract what the user already saw. Pick per capability and measure the latency cost.

Price requests in cost units

OWASP lists Unbounded Consumption as LLM10 in its 2025 Top 10, and a request-per-minute limit is a weak answer to it, because requests differ by orders of magnitude. Convert every request into a single cost unit before you limit it. With illustrative prices of 3 units per thousand input tokens and 15 per thousand output tokens, a chat turn of 2,000 input and 300 output tokens costs 6 + 4.5 = 10.5 units. A long-document job with 150,000 input tokens and 4,000 output tokens costs 450 + 60 = 510 units, nearly fifty chat turns, yet it is one request. Add a fixed charge per tool call and per second of sandbox time so agent work is priced too.

Because output length is unknown up front, reserve the worst case (max_tokens) before the call and settle the actual figure afterwards, returning the difference. Without the reservation, a hundred parallel requests all pass a check against the same remaining balance and overspend it together.

Atomic reservations across replicas

The reservation must be atomic across replicas. A counter in process memory protects one pod; an attacker spreads load across all of them. A Lua script executed by Redis runs without interleaving, so the check and the debit cannot be split by a concurrent request.

-- KEYS[1] = budget key, e.g. "budget:tenant:42:2026-10-15"
-- ARGV[1] = units to reserve, ARGV[2] = daily limit, ARGV[3] = ttl seconds
local used = tonumber(redis.call("GET", KEYS[1]) or "0")
local want = tonumber(ARGV[1])
if used + want > tonumber(ARGV[2]) then
  return {0, used}
end
used = redis.call("INCRBY", KEYS[1], want)
if used == want then
  redis.call("EXPIRE", KEYS[1], tonumber(ARGV[3]))
end
return {1, used}

Settlement is a second call that applies DECRBY for the unused part of the reservation. If the worker crashes between reserve and settle, the reservation is simply kept, which errs towards the tenant paying for capacity it held; that is the safe direction. Check tenant, account and key budgets in the same script so one call decides all three.

Budget the whole job, including retries

Agents turn one user request into many model calls, tool calls and retries. If each call is limited independently, a retry loop multiplies cost while every individual attempt looks small. Give each job a budget at creation and pass a handle to it through every child task.

class BudgetExceeded(Exception):
    pass

class JobBudget:
    def __init__(self, store, job_id, units, tool_calls, deadline_s):
        self.store, self.job_id = store, job_id
        store.init(job_id, units=units, tool_calls=tool_calls, deadline_s=deadline_s)

    def reserve(self, units, tool_call=False):
        ok = self.store.reserve(self.job_id, units=units, tool_calls=int(tool_call))
        if not ok:
            raise BudgetExceeded(self.job_id)  # stable error, no automatic resubmit

    def settle(self, reserved, actual):
        self.store.refund(self.job_id, units=max(0, reserved - actual))

def call_model(budget, req, retries=2):
    for attempt in range(retries + 1):
        reserved = cost_units(req.input_tokens, req.max_tokens)
        budget.reserve(reserved)                # every attempt draws on the same job
        try:
            resp = model.generate(req)
        except TransientError:
            budget.settle(reserved, cost_units(req.input_tokens, 0))
            continue
        budget.settle(reserved, cost_units(resp.input_tokens, resp.output_tokens))
        return resp
    raise TransientError("retries exhausted")

Return a stable, typed error when a budget runs out, with a retry-after hint when the window will reset. Do not return a generic 500 that a client library will resubmit automatically. Termination failures in agents are covered in agent DoS in depth.

A graduated enforcement ladder

Signals such as repeated blocks, sudden volume changes or near-identical outputs across accounts justify closer inspection. They do not prove intent; load tests and incident investigations produce strange traffic too. Map risk to graduated actions, each reversible and each logged with a reason code.

Levels 1 and 2 should be cheap to apply and cheap to undo, so you can act early without harming legitimate customers much. Levels 4 and 5 need evidence that meets a written policy, an appeal path, and a reason a support engineer can read without opening raw conversations.

Design review as a bounded service

A review queue is a service with capacity, not a bin. Give it an owner, priorities, a service level, and an expiry rule that states what happens to unreviewed items. For outbound messages the only safe expiry is "not sent"; for a private summary, releasing in a reduced mode may be fine. If arrival rate exceeds review rate, the queue must shed load by tightening upstream limits, never by auto-approving. Show reviewers the proposed action, the evidence and the rule, not just a model's accusation, and restrict who can read raw content.

Incident response runbooks

Rehearse three incidents before launch, because each needs a control that must already exist.

• Leaked API key. Detect a geography or volume shift on one key. Revoke the key, cancel its queued jobs, issue a replacement through the owner's verified channel, and bill-protect the tenant for the abusive window if policy allows. Verify that revocation propagates to every gateway replica within your target time.
• Screening outage. The classifier times out. High-risk capabilities stop; low-risk ones switch to a reduced mode with a banner. Page the owner. When it recovers, re-screen what was released in reduced mode.
• Runaway agent loop. Cost per job alarms. Job budgets should already have stopped it; if not, use the tool-level and tenant-level kill switches, preserve the trace, then fix the budget propagation bug. Agent kill switch design covers stop levels.

After each incident, add the trace as a synthetic regression fixture and record time to detect and time to contain.

Measure effectiveness

Track confirmed abuse per thousand completed jobs, legitimate requests blocked, reviewer disagreement rate, queue age at the 95th percentile, time to containment, and cost per tenant against plan. Slice by language and task type, because classifiers are usually weaker outside the languages they were tuned on. A block rate on its own is uninterpretable: it rises with good detection, with a broken classifier, and with an attack wave.

Introduce every new rule or threshold in shadow mode first. The gateway evaluates it, logs what it would have done, and enforces nothing. After a week of traffic, sample the would-be blocks, have reviewers label them, and estimate the false positive rate before turning enforcement on. Keep the shadow decision in the log after launch too, so a later rule change can be replayed against recorded traffic. Pair each enforcing rule with an owner and a review date; rules that nobody owns accumulate until the product feels hostile to ordinary users and nobody can say which rule is responsible. Finally, publish limits and remaining balances to customers in response headers or a usage page, because a customer who can see their budget draining is your fastest detector of a leaked key.

Failure modes

• Per-pod counters. Limits hold in testing and fail under distributed load. Use the atomic store.
• Retry multiplication. Client SDK retries plus server retries plus agent retries each draw fresh budget. Charge every attempt to the original job.
• Timeout means allow. A screening timeout falls through to the happy path. Write and test each stage's timeout branch.
• Internal caller exemption. Agent workers call tools with a service identity that skips checks. Enforce at the tool.
• Silent auto-approve. An overloaded review queue starts releasing expired items.
• Unreadable enforcement. Support cannot explain a suspension, so appeals take days.

Trade-offs

Every control trades abuse for friction. Fail-closed screening protects users and costs availability during outages. Buffered output protects and adds latency. Tight free-tier budgets stop farming and slow down legitimate evaluation. Cost-based limits are fairer than request limits but harder for customers to predict, so publish them and return remaining balance in response headers. Retaining conversations helps investigations and increases breach impact; keep verdicts and reason codes long, raw content short.

What to do next

1. Write the capability-to-abuse table with evidence and owners, including legitimate neighbours.
2. Order gateway stages cheap to expensive and document each stage's timeout behaviour.
3. Price requests in cost units and reserve the worst case before inference.
4. Move all budget checks into one atomic script on a shared store.
5. Create job budgets and charge every retry and child task to them.
6. Define the enforcement ladder with reason codes and reversal rules.
7. Give the review queue an owner, a service level and a safe expiry rule.
8. Rehearse a leaked key, a screening outage and a runaway loop, and time each containment.