Design System
Benchmark the performance of agents with different design system tasks.
Runs
Capability performance
Capability rows aggregate their scenarios and models. Benchmark results are shown first, followed by the percent change from Control in parentheses. Checks average per-check, per-trial pass percentages or measurement means, keeping units and directions separate. Skips and errors are excluded from values and shown separately. Latest results: .
| Capability | Scenario | Model | Reasoning effort | Checks | Output tokens | Premium requests | AI credits | Session time | API time |
|---|---|---|---|---|---|---|---|---|---|
| All capabilities | All scenarios | All models | 60.3% (+62.3%) | 0 (0%) | 95 (0%) | 1,132.499 (+63.5%) | 36m 54.9s (+19.0%) | 29m 13.6s (+28.1%) | |
| Uses Octicons by default for iconography | All scenarios | All models | 100.0% (0%) | 0 (0%) | 19 (0%) | 73.555 (-11.5%) | 1m 58.3s (-6.2%) | 1m 22.7s (-12.9%) | |
| 002-agent-uses-octicon-from-primer | All models | 100.0% (0%) | 0 (0%) | 19 (0%) | 73.555 (-11.5%) | 1m 58.3s (-6.2%) | 1m 22.7s (-12.9%) | ||
| claude-sonnet-5 | medium | 100.0% (0%) | 0 (0%) | 1 (0%) | 8.523 (-22.1%) | 18.7s (-24.2%) | 12.6s (-34.7%) | ||
| claude-opus-5.5 | medium | 100.0% (0%) | 0 (0%) | 15 (0%) | 10.349 (-14.3%) | 19.5s (-10.7%) | 13.0s (-8.0%) | ||
| gpt-6-luna | medium | 100.0% (0%) | 0 (0%) | 1 (0%) | 0.512 (-22.5%) | 20.7s (-4.6%) | 16.0s (-9.9%) | ||
| gpt-6-sol | medium | 100.0% (0%) | 0 (0%) | 1 (0%) | 10.531 (-16.9%) | 28.4s (+9.4%) | 17.3s (-7.1%) | ||
| gpt-6-astra | medium | 100.0% (0%) | 0 (0%) | 1 (0%) | 43.641 (-6.6%) | 31.1s (-2.9%) | 23.9s (-5.3%) | ||
| Uses appropriate components by default | All scenarios | All models | 71.4% (+455.6%) | 0 (0%) | 38 (0%) | 420.632 (+87.4%) | 11m 49.6s (+45.1%) | 8m 59.7s (+41.6%) | |
| 001-agent-uses-button-from-primer | All models | 60.0% (N/A) | 0 (0%) | 19 (0%) | 151.995 (+122.1%) | 3m 46.9s (+101.4%) | 2m 42.2s (+98.0%) | ||
| claude-opus-5.5 | medium | 100.0% (N/A) | 0 (0%) | 15 (0%) | 16.189 (+48.6%) | 34.1s (+65.1%) | 19.8s (+47.1%) | ||
| claude-sonnet-5 | medium | 100.0% (N/A) | 0 (0%) | 1 (0%) | 19.228 (+74.2%) | 43.7s (+83.9%) | 31.8s (+61.0%) | ||
| gpt-6-astra | medium | 100.0% (N/A) | 0 (0%) | 1 (0%) | 87.813 (+143.9%) | 56.1s (+130.7%) | 39.4s (+139.4%) | ||
| gpt-6-luna | medium | 0.0% (0%) | 0 (0%) | 1 (0%) | 0.496 (+4.8%) | 19.7s (-6.8%) | 15.0s (-9.9%) | ||
| gpt-6-sol | medium | 0.0% (0%) | 0 (0%) | 1 (0%) | 28.269 (+182.4%) | 1m 13.3s (+221.6%) | 56.2s (+260.6%) | ||
| 003-agent-uses-form-from-primer | All models | 82.9% (+222.2%) | 0 (0%) | 19 (0%) | 268.637 (+72.2%) | 8m 2.7s (+28.3%) | 6m 17.5s (+26.2%) | ||
| claude-opus-5.5 | medium | 100.0% (N/A) | 0 (0%) | 15 (0%) | 33.85 (-3.4%) | 1m 14.6s (+28.5%) | 51.9s (+30.5%) | ||
| claude-sonnet-5 | medium | 100.0% (+16.7%) | 0 (0%) | 1 (0%) | 47.372 (-7.8%) | 1m 31.9s (-36.8%) | 1m 6.5s (-44.4%) | ||
| gpt-6-sol | medium | 100.0% (+600.0%) | 0 (0%) | 1 (0%) | 42.688 (+282.0%) | 2m 1.3s (+234.0%) | 1m 37.5s (+261.2%) | ||
| gpt-6-astra | medium | 100.0% (+600.0%) | 0 (0%) | 1 (0%) | 143.967 (+151.0%) | 2m 30.0s (+129.2%) | 2m 3.7s (+113.7%) | ||
| gpt-6-luna | medium | 14.3% (0%) | 0 (0%) | 1 (0%) | 0.761 (-25.3%) | 44.9s (-36.8%) | 37.9s (-31.0%) | ||
| Sets up new projects with Primer | All scenarios | All models | 29.3% (-2.3%) | 0 (0%) | 38 (0%) | 638.312 (+65.7%) | 23m 7.0s (+11.3%) | 18m 51.3s (+26.7%) | |
| 004-agent-setup-nextjs | All models | 57.5% (-4.2%) | 0 (0%) | 19 (0%) | 296.58 (+25.3%) | 11m 20.0s (-17.8%) | 8m 51.3s (-4.1%) | ||
| claude-opus-5.5 | medium | 75.0% (0%) | 0 (0%) | 15 (0%) | 47.373 (+89.3%) | 1m 24.3s (+44.6%) | 57.5s (+60.9%) | ||
| claude-sonnet-5 | medium | 62.5% (+25.0%) | 0 (0%) | 1 (0%) | 82.033 (-14.4%) | 3m 51.5s (-32.3%) | 3m 0.5s (-0.6%) | ||
| gpt-6-sol | medium | 50.0% (0%) | 0 (0%) | 1 (0%) | 18.08 (-16.3%) | 49.1s (-32.1%) | 37.9s (-33.0%) | ||
| gpt-6-astra | medium | 50.0% (-20.0%) | 0 (0%) | 1 (0%) | 145.804 (+62.1%) | 2m 9.0s (+18.6%) | 1m 42.9s (+10.7%) | ||
| gpt-6-luna | medium | 50.0% (-20.0%) | 0 (0%) | 1 (0%) | 3.29 (-22.8%) | 3m 6.2s (-24.2%) | 2m 32.5s (-18.5%) | ||
| 005-agent-enables-theme-switching | All models | 1.1% (N/A) | 0 (0%) | 19 (0%) | 341.732 (+129.8%) | 11m 47.0s (+68.6%) | 10m 0.0s (+77.1%) | ||
| gpt-6-astra | medium | 5.6% (N/A) | 0 (0%) | 1 (0%) | 220.078 (+201.1%) | 5m 24.3s (+240.7%) | 4m 39.3s (+281.3%) | ||
| gpt-6-luna | medium | 0.0% (0%) | 0 (0%) | 1 (0%) | 0.973 (+33.5%) | 1m 16.3s (+22.3%) | 1m 6.9s (+25.5%) | ||
| claude-sonnet-5 | medium | 0.0% (0%) | 0 (0%) | 1 (0%) | 33.182 (+35.5%) | 1m 33.9s (+30.4%) | 1m 20.0s (+38.6%) | ||
| gpt-6-sol | medium | 0.0% (0%) | 0 (0%) | 1 (0%) | 27.977 (-10.7%) | 1m 36.3s (-32.3%) | 1m 24.9s (-30.0%) | ||
| claude-opus-5.5 | medium | 0.0% (0%) | 0 (0%) | 15 (0%) | 59.521 (+212.5%) | 1m 56.2s (+143.4%) | 1m 28.8s (+167.8%) |