Benchmarking Prompt Optimization of Large Language Models With Chess

The x-axis is the recorded USD cost of running the selected prompt on the full held-out test set: 559 puzzles expanded into 1,439 solver positions per run. Each condition reports the mean of its retained evaluation costs. Optimization spending is excluded; no baseline cost is added.

Inference cost reflects input prompt length, output tokens, model-specific pricing and any recorded cache effects. It is not a direct measure of prompt length, especially across models. Evaluation ledgers take priority; otherwise complete per-position costs are summed. Missing costs remain unavailable. All six applicable Jev conditions have inference-cost observations.

Each point represents one model–method condition. Unchanged optimizer runs are excluded before calculating both cost and accuracy means. Baselines remain. The resulting view contains 144 runs across 51 conditions, drawn from 165 source evaluations. Retained counts range from 1 to 3 and are shown in the inspector and table.

An optimizer run counts as updated when it changed the instruction or added demonstrations; the paper’s Table 3 averages all runs, including unchanged ones. All source runs remain in the downloadable data; selecting an artifact seed does not change the aggregate point.

A puzzle is correct only if every solver move matches the reference. Accuracy is the percentage of 559 puzzles solved. Vertical error bars show ±1 sample standard deviation across retained runs when n ≥ 2, not confidence intervals. No SD is estimated for n = 1.

Horizontal position is proportional to the square root of inference cost; tick labels show USD. Equal distances represent neither equal dollar differences nor equal ratios. A point is Pareto-optimal when no other filtered condition has equal or lower cost and equal or higher accuracy, with at least one strict improvement. Dominance uses point estimates, not uncertainty intervals.

Drag a rectangle in empty plot space to zoom; use Reset view to restore the full plot. Zoom changes only the viewport. The frontier continues to use all filtered conditions, including those outside the view. Connecting segments are visual guides, not measured learning curves. Changing filters resets the viewport.

The inspector displays the compiled instruction and demonstrations of each retained run, as the optimizer produced them; the DSPy adapter adds the fixed input and output formatting at call time. Best-iteration indices are shown for optimizers that record them (Jev BRS, COPRO, GEPA and MIPROv2).

The exporter verifies the source CSV hashes, reconstructs puzzle scores, and checks coverage against the release manifest. The JSON retains source IDs, cost provenance and all 141 optimized-run artifacts. Transfer, appendix baselines, regret and rollouts are separate cohorts and are not pooled here.