PROMPT OPTIMIZATION · SELECTION
DSPy-style optimization: do not select on training examples
Score candidate prompt programs, choose one by training or development accuracy, and compare the choice with held-out test accuracy.
Event trace
Computed data
Final metrics
Fictional, seeded fixture: each candidate answers an unseen example correctly with a fixed probability, and every outcome is a fixed pseudo-random draw. Demo examples are answered by copying when scored on training data. This is not the DSPy library or any Java port; it isolates the selection mistake.
Selection is part of the evaluation
Eight candidate prompt programs, P1 to P8, each answer unseen examples correctly with a fixed hidden probability, and each takes a few demonstrations from the twenty training examples. The lab scores every candidate on the training split, on a development split of up to forty examples, and on two hundred held-out test examples, and picks the winner by the split you choose. Every answer is a fixed seeded draw, so identical inputs give identical tables. The test column is revealed only at the end.
Training selection rewards copying
With four demos per candidate and selection by training accuracy, P6 wins with 0.8. On the held-out test it scores 0.575, the fourth-lowest of the eight. Part of its training score is copying: a demo that is also a training example is answered correctly whether or not the program could solve it, and the Copied column counts those answers. Switch Select the winner by to development accuracy: P4 wins with 0.775 and scores 0.7 on test, the best available.
Remove the leak, then shrink the dev set
Set Demos per candidate to 0. Nothing is copied, and training selection now picks P2, which scores 0.68 on test: better, but still not the best, because twenty examples is a small sample and the highest of eight noisy scores is optimistic. With eight demos, training selection picks P6 again at 0.85. Return to development selection and reduce Development examples to 5: P5 scores a perfect 1.0 on five examples and only 0.635 on test.
Fewer candidates, less optimism
Reduce Candidate programs searched to 3. Only P1 to P3 remain, and both splits pick P2, which scores 0.68 on test. Searching more candidates can find a better program, but it also gives chance more opportunities to produce a lucky score, so the selection split must be separate from the data used for demos and large enough to separate candidates. The selection score minus the test score is reported as the optimism of the choice.
Scope
This is a selection experiment, not the DSPy library or a Java port of it. Real optimizers propose instructions and bootstrap demonstrations with a metric, but the rule this lab isolates is the same: measure the final choice on data that neither produced the demos nor chose the winner. Play, Next event, Restart, Speed and Inspection time reveal the candidates one at a time over sixty seconds.