A leaderboard is not a workload specification
A small policy extraction task and a long code repair task can favor different models. Start with the output contract, context requirements, tolerated errors, data location and latency objective. Decide which constraints are mandatory. A model with strong generic benchmark results is not a candidate if it cannot process the data in an allowed environment. Do not hide hard exclusions inside a weighted score that can compensate for them.
Separate feasibility from preference
The exercise excludes models violating locality, quality or latency constraints, then chooses the least expensive feasible candidate. Every number is an invented fixture measurement. Real quality comes from representative reviewed cases, latency from a specified hardware and concurrency setup, and cost from usage accounting. A local model can have no provider token bill while still consuming hardware, power, operational effort and queue capacity. State those assumptions when comparing options.
Measure the complete harness path
Model response time omits retrieval, tool execution, retries, validation and waiting. A fast model with frequent schema failures may yield slower completed tasks. Measure time to a validated outcome and the rate of withheld responses. Match context and output limits to the workload. A tiny benchmark prompt does not establish capacity for long production histories, especially when local inference competes for memory with other applications.
Preserve constraints during fallback
Falling back from local inference to a remote provider may violate data policy despite improving availability. Reapply feasibility checks to every fallback. Preserve the original shared deadline and budget, recording the reason for the transition. Fallback does not create a new request with unlimited capacity. If no permitted model satisfies the task, abstain or ask for review rather than silently relaxing the data boundary.
Version evidence behind the choice
Record candidate identifiers, available model revisions, inference settings, hardware profile, evaluation dataset and run date. Provider aliases and local model tags can change. Reevaluate after material updates to weights, quantization, prompting, context preparation or adapters. A reproducible decision needs both the selected model and evidence explaining why alternatives were excluded. Keep obsolete measurements visible as historical evidence rather than presenting them as current results.
Investigate a selection failure
Suppose quality passes offline but live tasks fail under concurrent load. Check whether requests hit the intended model version, whether context was truncated and whether queueing caused deadline exhaustion. Do not immediately conclude that a larger model is necessary. The constraint layer may be functioning correctly while capacity assumptions are wrong. Current Harness Studio exposes extractive and installed Ollama workers; it does not yet implement this broader ranking and capacity service.
Run the exercise
Save the code as 13-model-selection-under-constraints.py and run python 13-model-selection-under-constraints.py with Python 3.11 or newer. This standard-library exercise uses synthetic data. It does not invoke a model or require an account, network connection or API key.
# Author: Sandeep Belgavi. Synthetic exercise; no network or provider calls.
models=[
{"name":"local-small","local":True,"quality":0.80,"latency":1.0,"cost":1},
{"name":"local-large","local":True,"quality":0.94,"latency":2.0,"cost":3},
{"name":"remote-fast","local":False,"quality":0.97,"latency":0.8,"cost":2}]
feasible=[m for m in models if m["local"] and m["quality"]>=0.90 and m["latency"]<=2.5]
assert min(feasible,key=lambda m:m["cost"])["name"]=="local-large"
assert not [m for m in feasible if m["latency"]<1.5]
print("local-large selected; stricter latency has no feasible candidate")
Expected output:
local-large selected; stricter latency has no feasible candidateChange one assumption
Permit remote processing for a synthetic public-data task and recompute the selection. Then set the budget below every feasible candidate cost. Return no feasible candidate rather than relaxing constraints. Add a capacity dimension and explain how a model can be eligible in principle but temporarily unavailable because its queue is full.
Implementation scope and further reading
This is an original AICassindra teaching design. The exercise demonstrates the stated invariant; it is not a complete production implementation. The local Harness Studio extension supports saved read-only agent teams, reviewed count-based routing and bounded sequential execution. Arbitrary imported graphs, effectful tools and automatic self-promotion remain unsupported.
All harness articles · Interactive boundary labs · Our public lab code