Baseline
Each value is the baseline delta from Control. A positive value means the baseline performed better for that metric; a negative value means it performed worse. An em dash means the result was not recorded.
| Scenario | Model | Effort | Test pass rate | Turns | Output tokens | Premium requests | API time | Session time | Tool calls |
|---|---|---|---|---|---|---|---|---|---|
| 001-agent-uses-button-from-primer | gpt-5.6-sol | — | 0% | +28.6% | -10.1% | 0% | +40.6% | +99.3% | +28.6% |
| 001-agent-uses-button-from-primer | gpt-5.6-terra | — | 0% | +16.7% | -16.9% | 0% | -49.6% | -58.6% | 0% |
| 001-agent-uses-button-from-primer | claude-opus-5 | — | +33.3% | -63.6% | -40.7% | 0% | -56.4% | -89.1% | -66.7% |
| 001-agent-uses-button-from-primer | claude-sonnet-5 | — | +100% | -128.6% | -79.8% | 0% | -121.7% | -173.8% | -100% |
| 001-agent-uses-button-from-primer | gemini-3.1-pro-preview | — | +100% | +14.3% | -29.2% | 0% | -594.4% | -520.3% | +14.3% |
| 001-agent-uses-button-from-primer | gemini-3.6-flash | — | +100% | +5% | -2.6% | 0% | +12% | -15.2% | 0% |
| 002-agent-uses-octicon-from-primer | gpt-5.6-sol | — | 0% | -14.3% | -5.2% | 0% | -46.1% | -47.8% | +7.7% |
| 002-agent-uses-octicon-from-primer | gpt-5.6-terra | — | 0% | -16.7% | -58% | 0% | -66.1% | -94.2% | -40% |
| 002-agent-uses-octicon-from-primer | claude-opus-5 | — | 0% | +25% | +24.5% | 0% | +20.8% | +7.1% | +25% |
| 002-agent-uses-octicon-from-primer | claude-sonnet-5 | — | 0% | +41.7% | +38.4% | 0% | +33.6% | +31.7% | +33.3% |
| 002-agent-uses-octicon-from-primer | gemini-3.1-pro-preview | — | 0% | +11.1% | +14% | 0% | +9% | +10.7% | +11.1% |
| 002-agent-uses-octicon-from-primer | gemini-3.6-flash | — | 0% | +27.8% | +35.2% | 0% | +29.9% | +34.8% | +35% |
| 003-agent-uses-form-from-primer | gpt-5.6-sol | — | +600% | -120% | -40.5% | 0% | -37.3% | -50.5% | -233.3% |
| 003-agent-uses-form-from-primer | gpt-5.6-terra | — | +500% | -33.3% | +37.5% | 0% | +10.7% | -22.7% | -28.6% |
| 003-agent-uses-form-from-primer | claude-opus-5 | — | 0% | -52.6% | -27.4% | 0% | -41% | -40.5% | -80% |
| 003-agent-uses-form-from-primer | claude-sonnet-5 | — | +16.7% | -19.2% | -4.2% | 0% | -14.9% | -21% | -42.9% |
| 003-agent-uses-form-from-primer | gemini-3.1-pro-preview | — | +600% | -85.7% | -23% | 0% | -88.1% | -88.7% | -85.7% |
| 003-agent-uses-form-from-primer | gemini-3.6-flash | — | 0% | -9.1% | +2.5% | 0% | -16.4% | -19.8% | -12.5% |
| 004-agent-setup-nextjs | gpt-5.6-sol | — | +100% | -66.7% | -68.6% | 0% | -76.6% | -93% | -66.7% |
| 004-agent-setup-nextjs | gpt-5.6-terra | — | +66.7% | +51.7% | +0.6% | 0% | -81.6% | -62.9% | +16.7% |
| 004-agent-setup-nextjs | claude-opus-5 | — | -16.7% | -38.9% | -45.4% | 0% | -61.1% | -54.7% | -50% |
| 004-agent-setup-nextjs | claude-sonnet-5 | — | +200% | -125% | -132.7% | 0% | -138.9% | -176.9% | -128% |
| 004-agent-setup-nextjs | gemini-3.1-pro-preview | — | +50% | +8.3% | +37.6% | 0% | -543.3% | -475.3% | +15.4% |
| 004-agent-setup-nextjs | gemini-3.6-flash | — | +100% | +8.6% | +3.6% | 0% | +6.8% | +7.3% | -11.4% |
| 005-agent-enables-theme-switching | gpt-5.6-sol | — | +100% | -283.3% | -102.6% | 0% | -126.8% | -133.7% | -218.8% |
| 005-agent-enables-theme-switching | gpt-5.6-terra | — | 0% | -25% | +48.4% | 0% | +9.6% | -16.2% | -37.5% |
| 005-agent-enables-theme-switching | claude-opus-5 | — | 0% | -7.1% | -23.7% | 0% | -20% | -19% | 0% |
| 005-agent-enables-theme-switching | claude-sonnet-5 | — | 0% | -50% | -7.8% | 0% | -29.9% | -32.3% | -45.5% |
| 005-agent-enables-theme-switching | gemini-3.1-pro-preview | — | +100% | +44% | +30% | 0% | +32.6% | +29.1% | +40% |
| 005-agent-enables-theme-switching | gemini-3.6-flash | — | 0% | -205.9% | -269.6% | 0% | -146.8% | -178.2% | -223.5% |
Scenarios
| ID | Prompt |
|---|---|
| 000-nextjs-template | Example prompt |
| 000-vite-template | Example prompt |
| 001-agent-uses-button-from-primer | Update the index page to use a primary button with the text 'Submit' |
| 002-agent-uses-octicon-from-primer | Update the index page to use a Search icon |
| 003-agent-uses-form-from-primer | Update the index page to render a sign-up form. The form does not need to post to an endpoint, I am only working on the UI for now. |
| 004-agent-setup-nextjs | Setup this project to work with Primer. Including a default page layout in app.tsx. |
| 005-agent-enables-theme-switching | Enable support for switching between all available color schemes. |
Experiments
| Name | Description | Models | Scenarios |
|---|---|---|---|
| Baseline | Baseline experiment to evaluate the performance of the agent with our recommended setup. | 6 | 5 |
| MCP with server instructions | Comparing `@primer/mcp` with and without server instructions to determine if server instructions improve the model's ability to follow instructions and complete tasks effectively. The experiment will involve two groups: one using `@primer/mcp` with server instructions and another using `@primer/mcp` without server instructions. | 1 | 2 |
| MCP | Compare MCP versus local instructions performance for Primer usage. | 3 | 2 |