Design System
Benchmark the performance of agents with different design system tasks.
Runs
Capability performance
Capability rows aggregate their scenarios and models. Benchmark results are shown first, followed by the percent change from Control in parentheses. Checks average per-check, per-trial pass percentages or measurement means, keeping units and directions separate. Skips and errors are excluded from values and shown separately. Latest results: .
| Capability | Scenario | Model | Reasoning effort | Checks | Output tokens | Premium requests | AI credits | Session time | API time |
|---|---|---|---|---|---|---|---|---|---|
| All capabilities | All scenarios | All models | 49.1% (+8.9%) | 110,551 (+9.4%) | 95 (0%) | 769.657 (+28.8%) | 34m 3.8s (+22.8%) | 26m 27.7s (+32.4%) | |
| Uses Octicons by default for iconography | All scenarios | All models | 100.0% (0%) | 3,693 (-23.5%) | 19 (0%) | 57.227 (-15.4%) | 1m 43.1s (-13.8%) | 1m 9.7s (-20.9%) | |
| 002-agent-uses-octicon-from-primer | All models | 100.0% (0%) | 3,693 (-23.5%) | 19 (0%) | 57.227 (-15.4%) | 1m 43.1s (-13.8%) | 1m 9.7s (-20.9%) | ||
| claude-opus-5.5 | medium | 100.0% (0%) | 558 (-38.5%) | 15 (0%) | 8.234 (-56.3%) | 11.7s (-35.3%) | 7.3s (-29.7%) | ||
| gpt-6-astra | medium | 100.0% (0%) | 663 (-9.8%) | 1 (0%) | 32.04 (+13.1%) | 28.9s (-2.1%) | 22.8s (-8.6%) | ||
| gpt-6-luna | medium | 100.0% (0%) | 675 (-25.8%) | 1 (0%) | 0.398 (-19.0%) | 20.7s (-14.0%) | 14.0s (-29.5%) | ||
| claude-sonnet-5 | medium | 100.0% (0%) | 892 (-41.0%) | 1 (0%) | 7.043 (-36.0%) | 15.9s (-39.7%) | 10.4s (-45.8%) | ||
| gpt-6-sol | medium | 100.0% (0%) | 905 (+18.3%) | 1 (0%) | 9.51 (+5.7%) | 26.0s (+19.9%) | 15.3s (+10.4%) | ||
| Uses appropriate components by default | All scenarios | All models | 41.4% (+43.2%) | 32,470 (+12.6%) | 38 (0%) | 304.22 (+52.6%) | 9m 52.1s (+30.3%) | 7m 27.8s (+28.3%) | |
| 001-agent-uses-button-from-primer | All models | 40.0% (+166.7%) | 7,097 (+16.9%) | 19 (0%) | 91.448 (+18.3%) | 2m 56.6s (+37.2%) | 2m 10.0s (+38.7%) | ||
| claude-opus-5.5 | medium | 100.0% (N/A) | 1,680 (+94.0%) | 15 (0%) | 16.751 (+77.3%) | 30.6s (+94.8%) | 18.3s (+106.2%) | ||
| gpt-6-sol | medium | 100.0% (N/A) | 1,952 (+154.2%) | 1 (0%) | 13.571 (+114.6%) | 48.8s (+139.7%) | 33.8s (+171.8%) | ||
| gpt-6-astra | medium | 0.0% (0%) | 846 (+32.4%) | 1 (0%) | 51.812 (+22.1%) | 43.2s (+39.0%) | 35.7s (+55.5%) | ||
| claude-sonnet-5 | medium | 0.0% (-100.0%) | 1,242 (-52.2%) | 1 (0%) | 8.922 (-52.2%) | 20.2s (-49.0%) | 14.9s (-52.6%) | ||
| gpt-6-luna | medium | 0.0% (0%) | 1,377 (+14.8%) | 1 (0%) | 0.393 (+0.4%) | 33.9s (+53.3%) | 27.3s (+51.1%) | ||
| 003-agent-uses-form-from-primer | All models | 42.9% (0%) | 25,373 (+11.4%) | 19 (0%) | 212.771 (+74.3%) | 6m 55.5s (+27.6%) | 5m 17.8s (+24.5%) | ||
| gpt-6-astra | medium | 100.0% (+600.0%) | 4,739 (+0.8%) | 1 (0%) | 105.835 (+106.3%) | 2m 17.3s (+53.6%) | 1m 47.5s (+32.4%) | ||
| claude-sonnet-5 | medium | 85.7% (0%) | 7,448 (+5.8%) | 1 (0%) | 44.109 (+41.2%) | 1m 39.6s (+7.0%) | 1m 15.3s (+9.0%) | ||
| gpt-6-sol | medium | 14.3% (0%) | 2,535 (+2.4%) | 1 (0%) | 14.331 (+51.9%) | 36.7s (+14.9%) | 26.8s (+10.4%) | ||
| gpt-6-luna | medium | 14.3% (0%) | 3,711 (+14.0%) | 1 (0%) | 0.503 (-14.3%) | 46.1s (+10.6%) | 37.5s (+15.0%) | ||
| claude-opus-5.5 | medium | 0.0% (-100.0%) | 6,940 (+31.0%) | 15 (0%) | 47.992 (+62.4%) | 1m 35.8s (+37.9%) | 1m 10.7s (+47.3%) | ||
| Sets up new projects with Primer | All scenarios | All models | 31.3% (-7.4%) | 74,388 (+10.4%) | 38 (0%) | 408.211 (+23.4%) | 22m 28.6s (+23.7%) | 17m 50.2s (+40.4%) | |
| 004-agent-setup-nextjs | All models | 62.5% (-7.4%) | 35,250 (+31.7%) | 19 (0%) | 190.986 (+12.8%) | 12m 33.5s (+32.5%) | 9m 48.3s (+78.3%) | ||
| claude-opus-5.5 | medium | 75.0% (0%) | 4,311 (+46.6%) | 15 (0%) | 31.372 (+53.6%) | 1m 2.3s (+15.0%) | 41.9s (+34.7%) | ||
| claude-sonnet-5 | medium | 75.0% (-14.3%) | 7,339 (-26.1%) | 1 (0%) | 42.159 (-28.4%) | 1m 55.2s (-46.4%) | 1m 24.3s (-19.7%) | ||
| gpt-6-luna | medium | 62.5% (-16.7%) | 14,752 (+99.8%) | 1 (0%) | 3.93 (+116.0%) | 4m 0.6s (+98.9%) | 3m 6.0s (+130.0%) | ||
| gpt-6-astra | medium | 50.0% (0%) | 3,880 (+37.6%) | 1 (0%) | 86.146 (+30.1%) | 2m 18.6s (+79.9%) | 1m 57.6s (+89.1%) | ||
| gpt-6-sol | medium | 50.0% (0%) | 4,968 (+34.2%) | 1 (0%) | 27.379 (+24.8%) | 3m 16.8s (+94.2%) | 2m 38.6s (+212.0%) | ||
| 005-agent-enables-theme-switching | All models | 0.0% (0%) | 39,138 (-3.6%) | 19 (0%) | 217.224 (+34.5%) | 9m 55.1s (+14.1%) | 8m 1.9s (+11.5%) | ||
| claude-opus-5.5 | medium | 0.0% (0%) | 3,126 (-8.8%) | 15 (0%) | 17.795 (-12.5%) | 37.0s (-18.0%) | 26.6s (-13.3%) | ||
| gpt-6-luna | medium | 0.0% (0%) | 4,584 (-53.6%) | 1 (0%) | 0.582 (-49.9%) | 54.4s (-42.4%) | 44.5s (-46.2%) | ||
| claude-sonnet-5 | medium | 0.0% (0%) | 9,788 (+2.0%) | 1 (0%) | 33.735 (-13.7%) | 1m 57.6s (+0.4%) | 1m 39.3s (+1.1%) | ||
| gpt-6-sol | medium | 0.0% (0%) | 10,552 (-15.2%) | 1 (0%) | 22.263 (-14.7%) | 1m 40.6s (-14.8%) | 1m 26.2s (-12.5%) | ||
| gpt-6-astra | medium | 0.0% (0%) | 11,088 (+111.3%) | 1 (0%) | 142.849 (+91.0%) | 4m 45.5s (+94.8%) | 3m 45.3s (+84.1%) |