primer/agent-eval

Agent eval navigation

Design System

Benchmark the performance of agents with different design system tasks.

Runs

DateResultsChecks
5049.0%
5047.1%
5046.3%
5050.2%
5048.7%
5055.8%
4050.1%
4056.9%
4052.1%

Capability performance

Capability rows aggregate their scenarios and models. Benchmark results are shown first, followed by the percent change from Control in parentheses. Checks average per-check, per-trial pass percentages or measurement means, keeping units and directions separate. Skips and errors are excluded from values and shown separately. Latest results: .

CapabilityScenarioModelReasoning effortChecksOutput tokensPremium requestsAI creditsSession timeAPI time
All capabilitiesAll scenariosAll models55.3% (+30.0%)122,092 (+21.7%)95 (0%)837.832 (+43.8%)32m 15.8s (+21.0%)23m 33.1s (+21.2%)
Uses Octicons by default for iconographyAll scenariosAll models100.0% (0%)4,838 (-3.0%)19 (0%)78.383 (+23.8%)2m 2.7s (-0.7%)1m 23.6s (-4.7%)
002-agent-uses-octicon-from-primerAll models100.0% (0%)4,838 (-3.0%)19 (0%)78.383 (+23.8%)2m 2.7s (-0.7%)1m 23.6s (-4.7%)
gpt-6-astramedium100.0% (0%)651 (-32.5%)1 (0%)33.411 (+6.4%)28.3s (-6.6%)21.3s (-12.2%)
claude-opus-5.5medium100.0% (0%)925 (+49.0%)15 (0%)19.058 (+129.7%)20.4s (+33.8%)10.6s (+42.8%)
gpt-6-solmedium100.0% (0%)938 (+4.0%)1 (0%)12.246 (+10.6%)29.9s (+11.1%)19.7s (+5.8%)
gpt-6-lunamedium100.0% (0%)946 (+80.5%)1 (0%)0.439 (+47.0%)22.3s (+18.3%)16.1s (+11.8%)
claude-sonnet-5medium100.0% (0%)1,378 (-30.3%)1 (0%)13.23 (+8.2%)21.9s (-32.3%)15.9s (-30.9%)
Uses appropriate components by defaultAll scenariosAll models50.0% (+133.3%)36,593 (+5.4%)38 (0%)303.594 (+45.3%)10m 42.2s (+17.4%)7m 41.8s (+16.9%)
001-agent-uses-button-from-primerAll models40.0% (N/A)6,085 (+21.9%)19 (0%)75.686 (+60.6%)2m 34.9s (+32.8%)1m 41.2s (+21.9%)
gpt-6-solmedium100.0% (N/A)1,507 (+76.1%)1 (0%)11.695 (+71.1%)47.2s (+110.2%)28.8s (+119.9%)
claude-opus-5.5medium100.0% (N/A)1,547 (+63.5%)15 (0%)15.954 (+72.5%)32.6s (+77.5%)17.5s (+78.1%)
gpt-6-astramedium0.0% (0%)740 (+13.7%)1 (0%)35.675 (+67.5%)32.8s (+79.2%)23.6s (+66.3%)
gpt-6-lunamedium0.0% (0%)1,084 (-8.5%)1 (0%)0.405 (+1.8%)22.8s (-40.2%)17.2s (-43.6%)
claude-sonnet-5medium0.0% (0%)1,207 (-10.9%)1 (0%)11.957 (+27.7%)19.7s (+0.5%)14.1s (-8.4%)
003-agent-uses-form-from-primerAll models60.0% (+40.0%)30,508 (+2.6%)19 (0%)227.907 (+40.9%)8m 7.3s (+13.2%)6m 0.6s (+15.6%)
gpt-6-astramedium100.0% (+600.0%)3,739 (-34.5%)1 (0%)87.937 (+27.1%)1m 45.0s (+37.4%)1m 14.3s (+9.3%)
gpt-6-solmedium100.0% (+600.0%)5,405 (+105.4%)1 (0%)29.714 (+179.8%)1m 39.5s (+171.7%)1m 16.6s (+186.7%)
claude-sonnet-5medium85.7% (0%)7,803 (-33.0%)1 (0%)42.37 (-22.4%)1m 43.3s (-46.3%)1m 19.2s (-34.7%)
gpt-6-lunamedium14.3% (0%)3,595 (-36.3%)1 (0%)0.708 (+8.6%)46.1s (-30.6%)35.1s (-38.9%)
claude-opus-5.5medium0.0% (-100.0%)9,966 (+143.5%)15 (0%)67.178 (+151.3%)2m 13.4s (+127.8%)1m 35.5s (+147.8%)
Sets up new projects with PrimerAll scenariosAll models38.3% (+9.5%)80,661 (+33.0%)38 (0%)455.856 (+46.8%)19m 30.9s (+26.1%)14m 27.6s (+27.0%)
004-agent-setup-nextjsAll models60.0% (-14.3%)30,246 (+14.1%)19 (0%)165.528 (+5.5%)8m 27.2s (-1.3%)6m 20.8s (+9.7%)
claude-opus-5.5medium75.0% (0%)4,533 (+14.0%)15 (0%)30.617 (+22.8%)1m 1.2s (-3.8%)41.6s (+0.4%)
gpt-6-astramedium62.5% (0%)2,349 (-31.3%)1 (0%)62.13 (-17.7%)1m 8.9s (-13.9%)52.9s (-11.9%)
claude-sonnet-5medium62.5% (-28.6%)10,016 (+43.2%)1 (0%)49.255 (+50.2%)2m 6.8s (+3.0%)1m 37.3s (+29.3%)
gpt-6-solmedium50.0% (0%)3,854 (+22.2%)1 (0%)21.235 (-1.4%)1m 21.6s (+18.3%)1m 1.2s (+40.1%)
gpt-6-lunamedium50.0% (-33.3%)9,494 (+6.0%)1 (0%)2.291 (+6.7%)2m 48.7s (-5.4%)2m 7.8s (+0.9%)
005-agent-enables-theme-switchingAll models16.7% (N/A)50,415 (+47.7%)19 (0%)290.328 (+88.9%)11m 3.7s (+60.0%)8m 6.8s (+44.9%)
claude-opus-5.5medium77.8% (N/A)9,054 (+208.3%)15 (0%)46.176 (+173.5%)1m 46.7s (+213.8%)1m 21.0s (+218.7%)
gpt-6-astramedium5.6% (N/A)10,765 (+27.6%)1 (0%)172.661 (+94.8%)3m 39.6s (+53.4%)2m 24.5s (+27.3%)
gpt-6-lunamedium0.0% (0%)4,635 (+2.9%)1 (0%)0.725 (+8.0%)1m 3.1s (+5.9%)45.8s (-11.0%)
claude-sonnet-5medium0.0% (0%)9,321 (+31.7%)1 (0%)32.24 (+37.2%)1m 47.8s (+31.0%)1m 30.1s (+35.0%)
gpt-6-solmedium0.0% (0%)16,640 (+48.8%)1 (0%)38.525 (+60.6%)2m 46.4s (+73.9%)2m 5.4s (+59.2%)