primer/agent-eval

Agent eval navigation

002-agent-uses-octicon-from-primer

Prompt

Update the index page to use a Search icon

Tests

import fs from 'node:fs/promises'
import path from 'node:path'
import {expect, test} from 'vitest'

const pagePath = path.resolve(import.meta.dirname, 'src', 'app', 'page.tsx')
const page = await fs.readFile(pagePath, 'utf8')

test('src/app/page.tsx imports Primer SearchIcon', () => {
  expect(page).toMatch(/import\s+{[^}]*\bSearchIcon\b[^}]*}\s+from\s+['"]@primer\/octicons-react['"]/)
})

test('src/app/page.tsx uses Primer SearchIcon', () => {
  expect(page).toMatch(/<SearchIcon(?:\s[^>]*)?\/>|<SearchIcon(?:\s[^>]*)?>[\s\S]*?<\/SearchIcon>/)
})

Experiments

  • Baseline

    Baseline experiment to evaluate the performance of the agent with our recommended setup.

  • MCP with server instructions

    Comparing `@primer/mcp` with and without server instructions to determine if server instructions improve the model's ability to follow instructions and complete tasks effectively. The experiment will involve two groups: one using `@primer/mcp` with server instructions and another using `@primer/mcp` without server instructions.

  • MCP

    Compare MCP versus local instructions performance for Primer usage.