Skip to content

GitHub Copilot Evals enters public preview

GitHub's new eval tool lets you write repeatable tests for skills, custom agents and MCP tools in your Copilot setup, then rerun them to catch regressions.

Draftship Editorial4 min read

GitHub Copilot Evals is now in public preview, GitHub said in a changelog entry dated November 17, 2026. The tool targets a problem anyone who customises Copilot runs into: after you change a prompt, a model, a skill or a tool, how do you know the setup still works as well as it did? GitHub Copilot Evals is meant to answer that with repeatable tests instead of a gut feeling.

GitHub frames the release as a way to optimize Copilot for yourself, and names skills, custom agents and MCP tools as the things it evaluates. According to the announcement, the preview is available on Copilot Individual, Copilot Business and Copilot Enterprise plans.

What GitHub counts as an eval

GitHub defines an eval as a repeatable test for AI behaviour. You give an agent a realistic task, describe what a good result looks like, and then check two things: the result itself, and how the agent got there.

The reasoning is that AI output varies between runs, so a single good answer proves little about whether a customisation works for you. Rerun the same eval after editing a skill, agent or MCP and you can see whether the behaviour you care about held, or whether something regressed.

Creating, running and comparing evals

The source lists four things you can do, either by asking Copilot in your own words or by using the /eval command:

  • Create an eval for a skill, custom agent or MCP tool in your project.
  • Run it and see which checks passed and which failed. You approve each run before it starts.
  • Find out why a check failed, then decide what to change in your Copilot setup.
  • Compare runs from before and after a customisation change to see whether it helped.

GitHub says the evaluation engine is Vally, described as an extensible local eval tool. Installation depends on where you already use Copilot: a GitHub Copilot Evals extension from the Visual Studio Marketplace for VS Code, a plugin for Copilot CLI, and for the GitHub Copilot app you open Customize, add the github/evals marketplace and install the plugin.

Where it fits in a working setup

The workflow the source describes is small and concrete. You change a skill or swap an MCP tool, rerun the eval you wrote earlier, and compare the new run against the old one. The failure explanation is part of the output, so the next step is a decision about the setup rather than another round of manual prompting.

This is likely most useful to people who keep several custom agents or MCP tools in a project and change them often, since those are exactly the pieces the preview covers. The source does not say what teams did before this, but it suggests that judging a customisation by trying a few prompts and eyeballing the answers is the behaviour being replaced.

Limits and open questions

Several things are not settled by the announcement:

  • How checks are scored. The source says you describe what a good result looks like but does not explain how that description is turned into a pass or fail.
  • Whether results are shared. Vally is described as local, and the source does not say whether a team can see each other's eval runs.
  • Cost. Nothing is said about how many eval runs a plan includes or whether runs consume usage.
  • Timeline. It is a public preview with no stated date for general availability, so behaviour may change.
  • Scope. The source only mentions skills, custom agents and MCP tools, not other parts of a Copilot setup.

The broader idea, that agent behaviour needs a test suite rather than a one-off spot check, is likely to travel well beyond Copilot. GitHub's own documentation is linked from the public preview announcement, and feedback goes through GitHub issues or the community discussion.

What this means for building apps with AI

The pattern is worth borrowing. When you build an app by describing it in plain language and then refining that description, the output changes with every edit, and it is easy to lose track of whether the latest version is better than the one before. Writing down a few realistic tasks and the result you expect, then rerunning them after each change, is the same discipline Copilot Evals applies to agent customisations. If you are iterating that way in Draftship, the guide to writing effective prompts covers how to keep those descriptions specific enough to test against. Draftship is not affiliated with GitHub and does not support or integrate with Copilot Evals.

Primary sources

This article was drafted with AI assistance from the primary sources listed below and checked automatically against them. An editor hasn't reviewed it yet.