An unchanged prompt can behave differently
A team changes retrieval depth while leaving its agent instruction untouched. The model now sees different records, including a contradictory policy. Another team lowers a routing threshold and sends more work to a specialist. Neither change alters model weights, yet both change behavior. Define a release unit covering the agent definition, tool schemas, routing policy, harness limits, evaluator version and relevant model identifiers.
Compare candidates on identical evidence
Run baseline and candidate against the same immutable cases and policy expectations. Separate task success from safety failures, recording critical cases individually. An aggregate score can hide a serious regression in a rare category. Stochastic inference may require repeated trials and uncertainty analysis. One passing sample does not establish stable improvement. Make raw outcome references and derived metrics traceable to the exact configuration that produced them.
Keep the promotion policy independent
The candidate may propose a change but must not edit the grader or remove critical labels to pass. The exercise requires strict improvement, no passing baseline case becoming a failure and every critical candidate case passing. These are conservative teaching rules, not universal deployment criteria. Real applications may require minimum effect sizes, latency limits and reviewed tradeoffs. Establish the policy before inspecting candidate results to avoid changing the target afterward.
Activation is not cloud deployment
Current Harness Studio approval creates an account-owned immutable configuration. Activation chooses the private configuration to run. A hosted deployment additionally requires infrastructure, credentials, verified identity, monitoring, persistence, backups and operational ownership. Preserve these as separate acceptance gates in a release report. A successful local comparison cannot establish that live authentication, external provider integration or customer-facing recovery is working.
Rollback restores configuration, not history
Returning to an earlier release does not undo completed effects or erase newly collected data. Preserve the event ledger and identify outstanding operations before rollback. For adaptive routing, keep the previous table and dataset provenance: restoring only a prompt leaves learned routing behavior changed. After rollback, verify a controlled set of tasks and continue monitoring the triggering failure. Record who selected the previous version and why.
Make release evidence useful to an operator
A release summary should identify the exact candidate, evaluated cases, critical failures, changed constraints, known limitations and the recovery path. Include failed experiments instead of only the best run. Bind approval to a digest computed from the accepted server-side snapshot. An approval for one candidate must not authorize a later draft merely because the two share a friendly name. A reviewer should be able to reconstruct the decision without relying on a model narrative.
Run the exercise
Save the code as 14-agent-harness-release-evidence.py and run python 14-agent-harness-release-evidence.py with Python 3.11 or newer. This standard-library exercise uses synthetic data. It does not invoke a model or require an account, network connection or API key.
# Author: Sandeep Belgavi. Synthetic exercise; no network or provider calls.
def eligible(rows):
improved=sum(c for b,c,k in rows)>sum(b for b,c,k in rows)
no_regression=all(not b or c for b,c,k in rows)
critical_pass=all(c for b,c,k in rows if k)
return improved and no_regression and critical_pass
assert eligible([(False,True,True),(True,True,False)])
assert not eligible([(False,True,False),(False,True,False),(True,False,True)])
assert not eligible([(True,True,True)])
print("improvement accepted; critical regression and unchanged score rejected")
Expected output:
improvement accepted; critical regression and unchanged score rejectedChange one assumption
Build a candidate gaining two easy cases while losing a critical case and verify rejection despite its higher total score. Remove all critical flags and confirm the no-regression rule still rejects a formerly passing case that now fails. Explain which additional evidence would be required before a real deployment rather than a private configuration activation.
Implementation scope and further reading
This is an original AICassindra teaching design. The exercise demonstrates the stated invariant; it is not a complete production implementation. The local Harness Studio extension supports saved read-only agent teams, reviewed count-based routing and bounded sequential execution. Arbitrary imported graphs, effectful tools and automatic self-promotion remain unsupported.
All harness articles · Interactive boundary labs · Our public lab code