Baseline overview
Each metric is the average Recommended result across all baseline scenarios, followed by the percent change from the average Control result in parentheses.
| Model | Effort | Tests passed | Turns | Output tokens | Premium requests | API time | Session time | Tool calls |
|---|---|---|---|---|---|---|---|---|
| claude-sonnet-5 | — | 4/7.8 (+100%) | 19.4 (-3%) | 4,473.2 (-20.4%) | 1 (0%) | 63.4s (-11.9%) | 81.8s (-11.3%) | 21 (0%) |
| gpt-5.6-sol | — | 3.4/7.8 (+240%) | 13.4 (+76.3%) | 4,966.6 (+47.1%) | 0 (0%) | 89.6s (+62.2%) | 110.6s (+72.2%) | 29.4 (+63.3%) |
| gemini-3.6-flash | — | 3/7.8 (+200%) | 43.6 (+54.6%) | 3,865 (+8.7%) | 14 (0%) | 219.6s (+26.8%) | 264.3s (+29.1%) | 44.4 (+50%) |
| gemini-3.1-pro-preview | — | 3/7.8 (+50%) | 18.2 (+35.8%) | 1,562.8 (+40.4%) | 1 (0%) | 93s (+37%) | 114s (+43.3%) | 19.4 (+31.1%) |
| gpt-5.6-terra | — | 2.4/7.8 (+71.4%) | 9 (+9.8%) | 3,656.8 (+12.6%) | 0 (0%) | 40.6s (+31.1%) | 56s (+40.2%) | 16.4 (-4.7%) |
| claude-opus-5 | — | 2.4/7.8 (-20%) | 29.6 (+49.5%) | 10,758.2 (+74.7%) | 1 (+25%) | 152s (+85.7%) | 219.2s (+117.3%) | 33.4 (+62.1%) |
Trends
Strong lines show Recommended results and muted lines show Control over time. The comparison table shows each Recommended result relative to Control.
Turns
Output tokens
Premium requests
API time
Session time
Tool calls
Change from Control
Recommended results are shown first, followed by the percent change from Control in parentheses.
| Model | ||||
|---|---|---|---|---|
| claude-opus-5 | 3.6/7.8 (0%) | 4/7.8 (+66.7%) | 5.4/7.8 (+80%) | 2.4/7.8 (-20%) |
| claude-sonnet-5 | 3.8/7.8 (+90%) | 3.6/7.8 (+28.6%) | 3.8/7.8 (+46.2%) | 4/7.8 (+100%) |
| gemini-3.1-pro-preview | 6/7.8 (+500%) | 6.2/7.8 (+181.8%) | 6/7.8 (+172.7%) | 3/7.8 (+50%) |
| gemini-3.6-flash | 3.4/7.8 (+54.5%) | 3.2/7.8 (-44.8%) | 3.2/7.8 (+45.5%) | 3/7.8 (+200%) |
| gpt-5.6-sol | 2.8/7.8 (+180%) | 5.8/7.8 (+262.5%) | 2.8/7.8 (+55.6%) | 3.4/7.8 (+240%) |
| gpt-5.6-terra | 2.6/7.8 (+116.7%) | 2.2/7.8 (+120%) | 2.8/7.8 (+55.6%) | 2.4/7.8 (+71.4%) |
View raw trend data
| Date | Model | Tests passed | Turns | Output tokens | Premium requests | API time | Session time | Tool calls |
|---|---|---|---|---|---|---|---|---|
| claude-opus-5 | 3.6/7.8 (0%) | 22.2 (+26.1%) | 7,209.2 (+26.9%) | 1 (0%) | 112s (+33.3%) | 140.7s (+36.4%) | 25.4 (+32.3%) | |
| claude-sonnet-5 | 3.8/7.8 (+90%) | 27.6 (+55.1%) | 7,093.6 (+41%) | 1 (0%) | 104.5s (+54.6%) | 142.8s (+71.3%) | 30.6 (+61.1%) | |
| gemini-3.1-pro-preview | 6/7.8 (+500%) | 13.8 (-12.7%) | 1,048.2 (-13.9%) | 1 (0%) | 228.3s (+259.3%) | 247.3s (+224.2%) | 14 (-13.6%) | |
| gemini-3.6-flash | 3.4/7.8 (+54.5%) | 38.4 (+48.8%) | 5,404.6 (+101.6%) | 14 (0%) | 112.5s (+47.3%) | 160s (+53.7%) | 41.8 (+57.1%) | |
| gpt-5.6-sol | 2.8/7.8 (+180%) | 12.4 (+82.4%) | 4,877 (+73%) | 0 (0%) | 112.9s (+75.9%) | 137.5s (-87.8%) | 29.6 (+94.7%) | |
| gpt-5.6-terra | 2.6/7.8 (+116.7%) | 9.6 (-17.2%) | 2,564.4 (-23.7%) | 0 (0%) | 55.1s (+36.6%) | 75.6s (+46.5%) | 22.4 (+7.7%) | |
| claude-opus-5 | 4/7.8 (+66.7%) | 32 (+42.9%) | 11,165 (+66.1%) | 1 (0%) | 161.9s (+63.8%) | 207.9s (+64.9%) | 35.4 (+45.1%) | |
| claude-sonnet-5 | 3.6/7.8 (+28.6%) | 23.2 (-7.9%) | 5,583.8 (-16.8%) | 1 (0%) | 83.1s (-9.4%) | 105.6s (-11.2%) | 25 (-8.8%) | |
| gemini-3.1-pro-preview | 6.2/7.8 (+181.8%) | 17.2 (+7.5%) | 1,488.2 (+5.2%) | 1 (0%) | 64.4s (+16%) | 85.2s (+19.1%) | 18.8 (+11.9%) | |
| gemini-3.6-flash | 3.2/7.8 (-44.8%) | 38.6 (+12.9%) | 5,694.2 (+144.2%) | 14 (0%) | 102.5s (+26.1%) | 152.3s (-17.4%) | 41 (+17.8%) | |
| gpt-5.6-sol | 5.8/7.8 (+262.5%) | 12.4 (+51.2%) | 4,489.6 (+52.2%) | 0 (0%) | 105.4s (+71.8%) | 126.4s (+76.7%) | 32.4 (+65.3%) | |
| gpt-5.6-terra | 2.2/7.8 (+120%) | 15.4 (+60.4%) | 5,422.2 (+131.8%) | 0 (0%) | 55.9s (+148.6%) | 93.3s (+167.4%) | 30.8 (+73%) | |
| claude-opus-5 | 5.4/7.8 (+80%) | 28.6 (+32.4%) | 8,079.6 (+27.2%) | 1 (0%) | 124.5s (+22.5%) | 160.9s (+29.6%) | 31.8 (+38.3%) | |
| claude-sonnet-5 | 3.8/7.8 (+46.2%) | 26.4 (-0.8%) | 7,090.8 (+2.1%) | 1 (0%) | 99.9s (-4.5%) | 145.4s (+1.4%) | 30 (+3.4%) | |
| gemini-3.1-pro-preview | 6/7.8 (+172.7%) | 16.4 (0%) | 1,087.2 (-31.1%) | 1 (0%) | 63s (+19.9%) | 80.8s (+23.9%) | 16.8 (+1.2%) | |
| gemini-3.6-flash | 3.2/7.8 (+45.5%) | 42.4 (+21.1%) | 7,939.4 (+55.3%) | 14 (0%) | 106.1s (+39.4%) | 153.1s (+23.7%) | 43.4 (+17.9%) | |
| gpt-5.6-sol | 2.8/7.8 (+55.6%) | 14.2 (+82.1%) | 3,897.6 (+26%) | 0 (0%) | 68.5s (+43.2%) | 90s (+48.4%) | 32 (+60%) | |
| gpt-5.6-terra | 2.8/7.8 (+55.6%) | 12.8 (+30.6%) | 4,060.8 (+27.2%) | 0 (0%) | 42.1s (+48%) | 62.6s (+57.1%) | 29.6 (+76.2%) | |
| claude-opus-5 | 2.4/7.8 (-20%) | 29.6 (+49.5%) | 10,758.2 (+74.7%) | 1 (+25%) | 152s (+85.7%) | 219.2s (+117.3%) | 33.4 (+62.1%) | |
| claude-sonnet-5 | 4/7.8 (+100%) | 19.4 (-3%) | 4,473.2 (-20.4%) | 1 (0%) | 63.4s (-11.9%) | 81.8s (-11.3%) | 21 (0%) | |
| gemini-3.1-pro-preview | 3/7.8 (+50%) | 18.2 (+35.8%) | 1,562.8 (+40.4%) | 1 (0%) | 93s (+37%) | 114s (+43.3%) | 19.4 (+31.1%) | |
| gemini-3.6-flash | 3/7.8 (+200%) | 43.6 (+54.6%) | 3,865 (+8.7%) | 14 (0%) | 219.6s (+26.8%) | 264.3s (+29.1%) | 44.4 (+50%) | |
| gpt-5.6-sol | 3.4/7.8 (+240%) | 13.4 (+76.3%) | 4,966.6 (+47.1%) | 0 (0%) | 89.6s (+62.2%) | 110.6s (+72.2%) | 29.4 (+63.3%) | |
| gpt-5.6-terra | 2.4/7.8 (+71.4%) | 9 (+9.8%) | 3,656.8 (+12.6%) | 0 (0%) | 40.6s (+31.1%) | 56s (+40.2%) | 16.4 (-4.7%) |