Fraud Prevention in LLM Applications: Verify Identity, Evidence, and Actions, in depth

Persuasive content is not verified authority

An assistant handling refunds will meet convincing claims: an urgent message that sounds like an executive, a screenshot of a bank transfer, a scanned identity document. None of these establish identity or entitlement. The FBI's Internet Crime Complaint Center published a public service announcement in December 2024 describing criminals using generative AI to produce synthetic text, images, audio and video for fraud, and advising people to verify identity through a contact channel they already know rather than one supplied in the request. It is a description of techniques, not a prevalence estimate, and the engineering lesson is direct: evidence that arrives inside the conversation can be fabricated cheaply, so authority must come from systems the attacker does not control.

That gives one rule for the whole design. A value may move money only if it was produced or confirmed by an authoritative system: your order database, your ledger, the payment processor's record, or an out-of-band confirmation through contact details you held before the request.

A worked case: refund to a new destination

A message arrives on an authenticated session: "Order 5531 arrived damaged. My card was cancelled, so please refund the 189.00 to my new bank account, details attached." There is a photo of a damaged parcel and a screenshot that appears to show the original payment. Four distinct things are being requested: identification of an order, a judgement that it is refundable, an amount, and a change of destination. Each has a different verifier.

Turn claims into evidence with explicit states

Let the model do what it is good at: extract the claimed order number, amount and requested destination, summarise the story, and list what is missing. Every extracted value starts in the state CLAIMED. The backend then tries to resolve each claim against the authenticated customer's records and records the source.

from dataclasses import dataclass
from enum import Enum
from typing import Optional

class State(Enum):
    CLAIMED = "claimed"        # extracted from the conversation, not checked
    VERIFIED = "verified"      # matched an authoritative record
    CONFLICT = "conflict"      # a record exists and disagrees with the claim
    UNKNOWN = "unknown"        # not stated, or could not be checked

@dataclass
class Evidence:
    field: str
    claimed: Optional[str]
    state: State
    source: Optional[str] = None      # e.g. "orders:5531", "ledger:txn/88f1"
    observed_at: Optional[str] = None

def verify_order(session, claim) -> Evidence:
    order = orders.get(claim.order_id) if claim.order_id else None
    if order is None or order.customer_id != session.customer_id:
        return Evidence("order_id", claim.order_id, State.UNKNOWN)
    return Evidence("order_id", claim.order_id, State.VERIFIED,
                    source=f"orders:{order.id}", observed_at=now_iso())

In the worked case the order resolves to customer 4411's session, so it is VERIFIED. The claimed 189.00 is compared with the ledger, which shows a 189.00 capture and a 40.00 partial refund last week; the claim is CONFLICT and the real refundable balance is 149.00. The screenshot is stored as a submitted artifact, never as a payment record. The new bank account is CLAIMED and stays that way in this workflow. The model must not fill gaps: a schema with an explicit unknown state stops downstream code from mistaking a plausible guess for data.

Compute eligibility and reserve under a lock

OWASP's LLM06 Excessive Agency entry recommends limiting tool permissions and enforcing authorization in downstream systems. Here that means the assistant can only call propose_refund(order_id, reason). An eligibility service computes the refundable amount from the ledger in integer minor units and reserves it inside a transaction, so two concurrent proposals cannot both claim the same balance.

-- Amounts in minor units (cents). One row per order holds the running totals.
BEGIN;
SELECT captured, refunded, reserved
  FROM order_balance
 WHERE order_id = :order_id AND customer_id = :customer_id
   FOR UPDATE;
-- application computes: available = captured - refunded - reserved
-- and rejects if :amount > available or :currency <> order currency
INSERT INTO refund_reservation (id, order_id, amount, currency, expires_at, status)
VALUES (:reservation_id, :order_id, :amount, :currency, now() + interval '15 minutes', 'held');
UPDATE order_balance SET reserved = reserved + :amount WHERE order_id = :order_id;
COMMIT;

An idempotency key alone does not give this guarantee. It deduplicates retries of the same request, but two different requests with different keys can each pass an unlocked balance check. The row lock, or an equivalent conditional update, is what keeps the total within the captured amount. Expired reservations are released by a sweeper.

Bind approval to the exact action

Whether approval comes from a policy rule, the customer, or a human agent, bind it to the exact action. The approval token is a MAC over a canonical encoding of every field that matters, with an expiry. The executor recomputes the MAC from the stored reservation; if any field changed, the token is invalid.

import hashlib, hmac, json, time

def canonical(action: dict) -> bytes:
    keys = ("reservation_id", "order_id", "amount_minor", "currency",
            "destination_id", "customer_id", "expires_at")
    return json.dumps({k: action[k] for k in keys}, sort_keys=True,
                      separators=(",", ":")).encode()

def sign(action: dict, key: bytes) -> str:
    return hmac.new(key, canonical(action), hashlib.sha256).hexdigest()

def check(action: dict, token: str, key: bytes) -> bool:
    if action["expires_at"] < time.time():
        return False
    return hmac.compare_digest(sign(action, key), token)

The executor also re-checks the customer's account status at execution time, because an account suspended after approval should not still be paid, and sends the processor an idempotency key derived from the reservation identifier. Agent permission prompts covers how to render the preview a human approves so it shows values, not prose.

Destination changes are a separate event

In the worked case the refund cannot go to the new bank account, and the reason is structural. Destination changes and account recovery are a separate workflow with a separate trigger. The service sends a confirmation through contact details it held before this conversation, never through an address or phone number in the request. Until that confirmation succeeds, the only valid destination is the original payment method; if that is unavailable, the refund waits.

Keeping the workflows apart closes a common takeover path, in which an attacker who has a session changes the destination and spends it in the same conversation. Depending on risk, add a hold period after a destination change, a second factor, or human review. Choose the controls and durations from your own loss data rather than copying another company's thresholds; strong authenticators like WebAuthn passkeys make the confirmation step much harder to phish.

Reconcile unknown outcomes instead of retrying

Money movement over a network has three outcomes, not two: succeeded, failed, and unknown. A timeout means unknown. Retrying blindly with a new key can pay twice; giving up can leave the customer unpaid. Model the state explicitly.

A reconciler polls unknown items using the same idempotency key, and an operator owns anything still unresolved after a set time. The model is not in this loop at all.

Use the model as an analyst aid, not a judge

An LLM is useful for analysts: summarising a case, listing which records disagree, drafting a note. Treat its fraud label as a fallible recommendation and require every statement in a summary to cite a record identifier the reviewer can open. The case material is untrusted input, and a fraudster can place instructions in it aimed at the summariser.

Base rates decide whether a label is useful. Suppose 1% of 10,000 refund requests are fraudulent and a model catches 90% of them with a 5% false positive rate. It flags 90 real frauds and 495 legitimate customers, so only 90 of 585 flags, about 15%, are fraud. Accuracy looks fine at about 95%, yet acting automatically on the label would wrong five legitimate customers for every fraudster caught. Evaluate on time-split data, report precision and recall at the operating threshold, and track reviewer workload and appeal outcomes. Scoring models themselves are covered in AI fraud detection in depth.

Test the failure paths before enabling payouts

Build a sandbox with synthetic customers, a fake ledger and a processor simulator that can succeed, fail, or hang on demand, then turn every failure mode into a test before payouts are enabled. The important cases are concurrency and timing, not clever wording.

• Two proposals for the same order submitted in parallel; the total reserved must never exceed the captured amount.
• An approval presented after its expiry, and one presented after the amount was edited; both must fail.
• An order number belonging to another customer; the claim must stay UNKNOWN and nothing must leak about the other account.
• A destination change followed by a refund request in the same session; the refund must use the original method or wait.
• A processor timeout followed by a retry; exactly one payout must exist after reconciliation.
• Case notes containing instructions aimed at the summariser; the analyst view must still cite only real records.

Re-run the suite whenever a tool schema, the approval code or the model changes. For live incidents, operators need a switch that pauses payout tools without stopping read-only support, a way to revoke sessions and API credentials, and a restricted audit trail that links proposal, evidence, approval and processor result under one identifier. Keep full payment credentials out of prompts, model logs and analytics entirely; the model never needs them to do its job.

Failure modes

• Model-filled amount. The executor pays the amount the model extracted. Fix: amounts come only from the eligibility service.
• Screenshot as record. A submitted image is stored in the same field as a ledger reference. Fix: separate artifact and record types.
• Unlocked balance check. Parallel requests each pass. Fix: lock or conditional update.
• Same-session destination change. Attacker changes payout and spends it at once. Fix: separate workflow and out-of-band confirmation.
• Timeout retry with new key. Double payment. Fix: explicit unknown state and reconciliation.
• Label as verdict. High-confidence model output auto-denies customers. Fix: advisory label, cited evidence, appeal path.

Trade-offs

Every verification step adds friction for honest customers, and a hold on new destinations delays real refunds. Strict separation of workflows means more support contacts. The alternative costs are higher but less visible: chargebacks, account takeover losses, and regulatory exposure. Tune friction by risk: small refunds to the original payment method can be nearly instant, while new destinations, high amounts or conflicting evidence take the slow path.

What to do next

1. List every money-moving or account-changing action your assistant can trigger.
2. Give extracted values an explicit state and record the source of every verified value.
3. Compute amounts in the eligibility service, in minor units, with a locked reservation.
4. Bind approvals with a MAC over the canonical action and check it at execution.
5. Move destination changes and recovery to a separate workflow confirmed through known channels.
6. Add an unknown state, a reconciler and an operator for processor timeouts.
7. Keep model fraud labels advisory, cite records, and measure precision at your real base rate.
8. Test parallel requests, expired approvals, changed destinations and timeouts in a sandbox before enabling payouts.