primer/agent-eval

Agent eval navigation

002-agent-uses-octicon-from-primer

Recommended results are shown first, followed by the percent change from Control in parentheses. Positive means the raw value increased and negative means it decreased. Models are ranked by test pass rate within each scenario.

ModelEffortTests passedTurnsOutput tokensPremium requestsAPI timeSession timeTool calls
claude-sonnet-5—2/2 (0%)6 (-14.3%)818 (-13.9%)1 (0%)16s (+5.7%)24.5s (+31.8%)7 (0%)
gpt-5.6-terra—2/2 (0%)6 (-14.3%)538 (-36.9%)0 (0%)13.6s (-2.1%)25.7s (+18.9%)9 (-35.7%)
gpt-5.6-sol—2/2 (0%)7 (0%)908 (-0.2%)0 (0%)26.7s (+46.9%)37.4s (+41.7%)15 (-11.8%)
claude-opus-5—2/2 (0%)8 (-27.3%)1,173 (-25.7%)1 (0%)21.7s (-15.7%)31.7s (-6.7%)9 (-18.2%)
gemini-3.1-pro-preview—2/2 (0%)11 (0%)419 (-19.7%)1 (0%)35.9s (-70.3%)39.1s (-69.6%)11 (0%)
gemini-3.6-flash—2/2 (0%)23 (+64.3%)831 (+31.5%)14 (0%)75.4s (+86%)93s (+75.5%)23 (+27.8%)

Strong lines show Recommended results and muted lines show Control over time. The comparison table shows each Recommended result relative to Control.

Tests passed

2.221.807-2908-0308-1008-17Run date

Turns

24.714.54.307-2908-0308-1008-17Run date

Output tokens

2,498.81,300101.207-2908-0308-1008-17Run date

Premium requests

15.47.7007-2908-0308-1008-17Run date

API time

13266007-2908-0308-1008-17Run date

Session time

139.972.75.507-2908-0308-1008-17Run date

Tool calls

24.6155.407-2908-0308-1008-17Run date

Change from Control

Recommended results are shown first, followed by the percent change from Control in parentheses.

Model
claude-opus-52/2 (0%)2/2 (0%)2/2 (0%)2/2 (0%)
claude-sonnet-52/2 (0%)2/2 (0%)2/2 (0%)2/2 (0%)
gemini-3.1-pro-preview2/2 (0%)2/2 (0%)2/2 (0%)2/2 (0%)
gemini-3.6-flash2/2 (0%)2/2 (0%)2/2 (0%)2/2 (0%)
gpt-5.6-sol2/2 (0%)2/2 (0%)2/2 (0%)2/2 (0%)
gpt-5.6-terra2/2 (0%)2/2 (0%)2/2 (0%)2/2 (0%)
View raw trend data
DateModelTests passedTurnsOutput tokensPremium requestsAPI timeSession timeTool calls
claude-opus-52/2 (0%)9 (-25%)1,239 (-24.5%)1 (0%)25.6s (-20.8%)38.3s (-7.1%)9 (-25%)
claude-sonnet-52/2 (0%)7 (-41.7%)965 (-38.4%)1 (0%)18.7s (-33.6%)25.5s (-31.7%)8 (-33.3%)
gemini-3.1-pro-preview2/2 (0%)8 (-11.1%)301 (-14%)1 (0%)26.8s (-9%)34.2s (-10.7%)8 (-11.1%)
gemini-3.6-flash2/2 (0%)13 (-27.8%)459 (-35.2%)14 (0%)23.7s (-29.9%)37.5s (-34.8%)13 (-35%)
gpt-5.6-sol2/2 (0%)8 (+14.3%)755 (+5.2%)0 (0%)37.6s (+46.1%)50.9s (+47.8%)12 (-7.7%)
gpt-5.6-terra2/2 (0%)7 (+16.7%)975 (+58%)0 (0%)23.1s (+66.1%)34.7s (+94.2%)14 (+40%)
claude-opus-52/2 (0%)9 (-25%)1,134 (-39.6%)1 (0%)28.5s (-10.3%)42.4s (+5.2%)9 (-30.8%)
claude-sonnet-52/2 (0%)7 (-22.2%)923 (-29.1%)1 (0%)17.4s (-33.8%)25.6s (-30.8%)8 (-20%)
gemini-3.1-pro-preview2/2 (0%)10 (+42.9%)421 (+39.4%)1 (0%)28.7s (+24.7%)38.8s (+31.3%)11 (+57.1%)
gemini-3.6-flash2/2 (0%)15 (-16.7%)495 (-19.5%)14 (0%)26.2s (-14.2%)42s (-2.6%)15 (-21.1%)
gpt-5.6-sol2/2 (0%)6 (0%)576 (-13.5%)0 (0%)25.1s (+30.7%)32.9s (+34.9%)10 (-23.1%)
gpt-5.6-terra2/2 (0%)7 (0%)841 (-5%)0 (0%)12.1s (-0.6%)25.4s (+51.9%)14 (+7.7%)
claude-opus-52/2 (0%)9 (-40%)1,199 (-47.8%)1 (0%)24.7s (-43.1%)36s (-39.4%)9 (-40%)
claude-sonnet-52/2 (0%)7 (-36.4%)941 (-47.8%)1 (0%)19.6s (-39.7%)26.6s (-35.7%)8 (-33.3%)
gemini-3.1-pro-preview2/2 (0%)10 (+25%)444 (+41%)1 (0%)28s (+29.1%)37.9s (+49.9%)12 (+50%)
gemini-3.6-flash2/2 (0%)16 (+6.7%)521 (-16.9%)14 (0%)26.7s (+25.3%)41.9s (-31.2%)16 (0%)
gpt-5.6-sol2/2 (0%)8 (+14.3%)902 (+7.8%)0 (0%)41.5s (+106.2%)53.3s (+85.3%)16 (+6.7%)
gpt-5.6-terra2/2 (0%)6 (0%)686 (+3.2%)0 (0%)11.3s (+17.7%)23.5s (+24.3%)10 (0%)
claude-opus-52/2 (0%)8 (-27.3%)1,173 (-25.7%)1 (0%)21.7s (-15.7%)31.7s (-6.7%)9 (-18.2%)
claude-sonnet-52/2 (0%)6 (-14.3%)818 (-13.9%)1 (0%)16s (+5.7%)24.5s (+31.8%)7 (0%)
gemini-3.1-pro-preview2/2 (0%)11 (0%)419 (-19.7%)1 (0%)35.9s (-70.3%)39.1s (-69.6%)11 (0%)
gemini-3.6-flash2/2 (0%)23 (+64.3%)831 (+31.5%)14 (0%)75.4s (+86%)93s (+75.5%)23 (+27.8%)
gpt-5.6-sol2/2 (0%)7 (0%)908 (-0.2%)0 (0%)26.7s (+46.9%)37.4s (+41.7%)15 (-11.8%)
gpt-5.6-terra2/2 (0%)6 (-14.3%)538 (-36.9%)0 (0%)13.6s (-2.1%)25.7s (+18.9%)9 (-35.7%)