A log entry is an observation
An execution log identifies which specialist was selected. It does not establish that the choice was correct. Training directly on the router own selections can reinforce initial mistakes. Distinguish selected_agent from reviewed_agent and preserve outcome evidence separately. A correct final answer also does not establish an efficient initial route: unnecessary transfers may have rescued a poor first decision at substantial cost.
Design a row for its actual purpose
A compact routing row can contain task_id, group_id, category, previous_agent, reviewed_agent, review_status and evidence_reference. Exclude sensitive text unless the classifier needs it and its use is permitted. JSON Lines stores one JSON object per line. That format does not by itself make the data appropriate for LLM fine-tuning. Provider training schemas, model objectives and consent requirements are separate decisions from fitting a routing count table.
Split groups before creating variants
Twenty paraphrases from one support ticket should not be randomly scattered between training and evaluation. Assign the originating ticket or family to one split before deriving variants. Depending on the deployment, stronger separation by customer, repository or time period may be needed. State which leakage the grouping prevents and what it cannot detect. Similar wording across unrelated identifiers can still reveal a shared template or copied answer.
Quarantine invalid and unreviewed records
Reject duplicate task identifiers with conflicting labels. Exclude unreviewed records from supervised updates. Check that every reviewed destination belongs to the supported role catalogue and that the reviewer can inspect the evidence. Keep a rejection report with specific reasons. Silent dropping obscures dataset quality, while accepting every syntactically valid row permits contaminated labels and unknown roles to change future behavior.
Version the transformation as well as the input
Preprocessing can change effective training data without changing the original file. Record normalization rules, category mappings, split assignment, review policy and code revision with the generated table. If performance declines, this evidence distinguishes a modeling issue from a changed label interpretation. Retain an immutable accepted dataset manifest and a way to reproduce its transformation; do not depend on a mutable folder of session exports as the only record.
Connect checks to the release gate
Dataset validation should fail before any candidate is trained when groups overlap or labels violate the contract. A later metric cannot repair contaminated evaluation evidence. Current Arena routing datasets now check declared group overlap and train reviewed count tables; they do not discover undeclared semantic duplicates automatically. A stronger implementation would combine group checks, duplicate analysis and reviewer provenance, with each check producing independently inspectable evidence.
Run the exercise
Save the code as 10-routing-jsonl-and-leakage.py and run python 10-routing-jsonl-and-leakage.py with Python 3.11 or newer. This standard-library exercise uses synthetic data. It does not invoke a model or require an account, network connection or API key.
# Author: Sandeep Belgavi. Synthetic exercise; no network or provider calls.
import json
lines=['{"task_id":"a","group_id":"ticket-1","review_status":"accepted"}',
'{"task_id":"b","group_id":"ticket-2","review_status":"pending"}']
rows=[json.loads(line) for line in lines]
accepted=[r for r in rows if r["review_status"]=="accepted"]
assert len(accepted)==1
assert {"ticket-1"}.isdisjoint({"ticket-2"})
assert not {"ticket-1"}.isdisjoint({"ticket-1"})
print("one reviewed row accepted; overlapping group detected")
Expected output:
one reviewed row accepted; overlapping group detectedChange one assumption
Add two accepted records with the same task_id and conflicting reviewed_agent values, then implement rejection. Place two paraphrases from one ticket in different splits. The group validator must reject them even with different task identifiers. Finally record rejected row counts without retaining unnecessary sensitive request text in the report.
Implementation scope and further reading
This is an original AICassindra teaching design. The exercise demonstrates the stated invariant; it is not a complete production implementation. The local Harness Studio extension supports saved read-only agent teams, reviewed count-based routing and bounded sequential execution. Arbitrary imported graphs, effectful tools and automatic self-promotion remain unsupported.
All harness articles · Interactive boundary labs · Our public lab code