primer/agent-eval

Agent eval navigation

001-agent-uses-button-from-primer

Prompt

Update the index page to use a primary button with the text 'Submit'

Tests

import fs from 'node:fs/promises'
import path from 'node:path'
import {expect, test} from 'vitest'

const pagePath = path.resolve(import.meta.dirname, 'src', 'app', 'page.tsx')
const page = await fs.readFile(pagePath, 'utf8')

test('src/app/page.tsx imports Primer Button', () => {
  expect(page).toMatch(/import\s+{[^}]*\bButton\b[^}]*}\s+from\s+['"]@primer\/react['"]/)
})

test('src/app/page.tsx uses Primer Button', () => {
  expect(page).toMatch(/<Button[^>]*>[\s\S]*?<\/Button>/)
})

test('src/app/page.tsx uses primary variant', () => {
  expect(page).toMatch(/<Button[^>]*variant=["']primary["'][^>]*>[\s\S]*?<\/Button>/)
})

test('src/app/page.tsx button has text submit', () => {
  expect(page).toMatch(/<Button[^>]*>[\s\S]*?Submit[\s\S]*?<\/Button>/)
})

Experiments

  • Baseline

    Baseline experiment to evaluate the performance of the agent with our recommended setup.

  • MCP with server instructions

    Comparing `@primer/mcp` with and without server instructions to determine if server instructions improve the model's ability to follow instructions and complete tasks effectively. The experiment will involve two groups: one using `@primer/mcp` with server instructions and another using `@primer/mcp` without server instructions.

  • MCP

    Compare MCP versus local instructions performance for Primer usage.