005-agent-enables-theme-switching
Recommended results are shown first, followed by the percent change from Control in parentheses. Positive means the raw value increased and negative means it decreased. Models are ranked by test pass rate within each scenario.
| Model | Effort | Tests passed | Turns | Output tokens | Premium requests | API time | Session time | Tool calls |
|---|---|---|---|---|---|---|---|---|
| gpt-5.6-sol | — | 0/18 (0%) | 11 (+57.1%) | 10,265 (+42%) | 0 (0%) | 135.7s (+52.3%) | 153.4s (+58.5%) | 24 (+60%) |
| gpt-5.6-terra | — | 0/18 (0%) | 11 (+22.2%) | 8,466 (+123.1%) | 0 (0%) | 79.6s (+153.1%) | 90.1s (+128.1%) | 20 (+17.6%) |
| gemini-3.1-pro-preview | — | 0/18 (0%) | 25 (+92.3%) | 1,664 (+138.7%) | 1 (0%) | 100.3s (+133.7%) | 126.8s (+143.6%) | 26 (+73.3%) |
| claude-sonnet-5 | — | 0/18 (0%) | 28 (+16.7%) | 7,565 (+9.5%) | 1 (0%) | 97.6s (+17.1%) | 118.7s (+21.1%) | 30 (+15.4%) |
| claude-opus-5 | — | 0/18 (0%) | 49 (+104.2%) | 25,957 (+149.2%) | 1 (0%) | 321.6s (+156.7%) | 463.8s (+214.5%) | 56 (+124%) |
| gemini-3.6-flash | — | 0/18 (0%) | 75 (+82.9%) | 7,503 (-15.4%) | 14 (0%) | 515.9s (+35.6%) | 619.7s (+34.2%) | 79 (+92.7%) |
Trends
Strong lines show Recommended results and muted lines show Control over time. The comparison table shows each Recommended result relative to Control.
Turns
Output tokens
Premium requests
API time
Session time
Tool calls
Change from Control
Recommended results are shown first, followed by the percent change from Control in parentheses.
| Model | ||||
|---|---|---|---|---|
| claude-opus-5 | 0/18 (0%) | 1/18 (—) | 15/18 (—) | 0/18 (0%) |
| claude-sonnet-5 | 0/18 (0%) | 0/18 (0%) | 0/18 (0%) | 0/18 (0%) |
| gemini-3.1-pro-preview | 14/18 (—) | 14/18 (—) | 14/18 (—) | 0/18 (0%) |
| gemini-3.6-flash | 0/18 (0%) | 0/18 (-100%) | 0/18 (0%) | 0/18 (0%) |
| gpt-5.6-sol | 1/18 (—) | 16/18 (—) | 1/18 (—) | 0/18 (0%) |
| gpt-5.6-terra | 0/18 (0%) | 0/18 (0%) | 0/18 (0%) | 0/18 (0%) |
View raw trend data
| Date | Model | Tests passed | Turns | Output tokens | Premium requests | API time | Session time | Tool calls |
|---|---|---|---|---|---|---|---|---|
| claude-opus-5 | 0/18 (0%) | 30 (+7.1%) | 15,853 (+23.7%) | 1 (0%) | 200.2s (+20%) | 226.2s (+19%) | 32 (0%) | |
| claude-sonnet-5 | 0/18 (0%) | 30 (+50%) | 8,532 (+7.8%) | 1 (0%) | 130.1s (+29.9%) | 154s (+32.3%) | 32 (+45.5%) | |
| gemini-3.1-pro-preview | 14/18 (—) | 14 (-44%) | 1,008 (-30%) | 1 (0%) | 72.6s (-32.6%) | 88.5s (-29.1%) | 15 (-40%) | |
| gemini-3.6-flash | 0/18 (0%) | 104 (+205.9%) | 19,265 (+269.6%) | 14 (0%) | 325.1s (+146.8%) | 442.8s (+178.2%) | 110 (+223.5%) | |
| gpt-5.6-sol | 1/18 (—) | 23 (+283.3%) | 14,215 (+102.6%) | 0 (0%) | 296.7s (+126.8%) | 327.1s (+133.7%) | 51 (+218.8%) | |
| gpt-5.6-terra | 0/18 (0%) | 10 (+25%) | 3,019 (-48.4%) | 0 (0%) | 43.5s (-9.6%) | 65.3s (+16.2%) | 22 (+37.5%) | |
| claude-opus-5 | 1/18 (—) | 80 (+300%) | 34,928 (+240.7%) | 1 (0%) | 464.3s (+281.7%) | 569.3s (+290.6%) | 83 (+219.2%) | |
| claude-sonnet-5 | 0/18 (0%) | 31 (+10.7%) | 9,368 (-2%) | 1 (0%) | 128.4s (+6.2%) | 150.8s (+7.9%) | 33 (+10%) | |
| gemini-3.1-pro-preview | 14/18 (—) | 33 (+32%) | 3,579 (+146.1%) | 1 (0%) | 132.4s (+64.4%) | 172.4s (+72.3%) | 36 (+38.5%) | |
| gemini-3.6-flash | 0/18 (-100%) | 65 (+14%) | 18,000 (+306.9%) | 14 (0%) | 202.3s (+59.2%) | 309.6s (-40.8%) | 71 (+24.6%) | |
| gpt-5.6-sol | 16/18 (—) | 26 (+271.4%) | 13,065 (+192.2%) | 0 (0%) | 298.1s (+251.8%) | 332.7s (+259.3%) | 72 (+300%) | |
| gpt-5.6-terra | 0/18 (0%) | 16 (+45.5%) | 9,250 (+134.3%) | 0 (0%) | 81.4s (+171%) | 108.7s (+104.5%) | 31 (+63.2%) | |
| claude-opus-5 | 15/18 (—) | 48 (+100%) | 18,445 (+52.7%) | 1 (0%) | 261.5s (+47.4%) | 311.2s (+59.1%) | 49 (+96%) | |
| claude-sonnet-5 | 0/18 (0%) | 32 (+18.5%) | 10,176 (+47.3%) | 1 (0%) | 116.4s (+8.9%) | 141.1s (-4.1%) | 34 (+21.4%) | |
| gemini-3.1-pro-preview | 14/18 (—) | 17 (-19%) | 1,083 (+9.6%) | 1 (0%) | 56.2s (-5.9%) | 70.8s (-1.8%) | 17 (-19%) | |
| gemini-3.6-flash | 0/18 (0%) | 125 (+78.6%) | 33,077 (+113%) | 14 (0%) | 345.8s (+113.4%) | 480.5s (+82.3%) | 125 (+78.6%) | |
| gpt-5.6-sol | 1/18 (—) | 25 (+257.1%) | 7,326 (+113.3%) | 0 (0%) | 104.4s (+138.5%) | 126.7s (+141.3%) | 45 (+164.7%) | |
| gpt-5.6-terra | 0/18 (0%) | 8 (-20%) | 4,629 (-19.6%) | 0 (0%) | 40.3s (+5.8%) | 53.4s (+15.3%) | 21 (+10.5%) | |
| claude-opus-5 | 0/18 (0%) | 49 (+104.2%) | 25,957 (+149.2%) | 1 (0%) | 321.6s (+156.7%) | 463.8s (+214.5%) | 56 (+124%) | |
| claude-sonnet-5 | 0/18 (0%) | 28 (+16.7%) | 7,565 (+9.5%) | 1 (0%) | 97.6s (+17.1%) | 118.7s (+21.1%) | 30 (+15.4%) | |
| gemini-3.1-pro-preview | 0/18 (0%) | 25 (+92.3%) | 1,664 (+138.7%) | 1 (0%) | 100.3s (+133.7%) | 126.8s (+143.6%) | 26 (+73.3%) | |
| gemini-3.6-flash | 0/18 (0%) | 75 (+82.9%) | 7,503 (-15.4%) | 14 (0%) | 515.9s (+35.6%) | 619.7s (+34.2%) | 79 (+92.7%) | |
| gpt-5.6-sol | 0/18 (0%) | 11 (+57.1%) | 10,265 (+42%) | 0 (0%) | 135.7s (+52.3%) | 153.4s (+58.5%) | 24 (+60%) | |
| gpt-5.6-terra | 0/18 (0%) | 11 (+22.2%) | 8,466 (+123.1%) | 0 (0%) | 79.6s (+153.1%) | 90.1s (+128.1%) | 20 (+17.6%) |