Overview
Imagine you’re an AI researcher sifting through hundreds of thousands of lines of JSON every day, each file representing the step-by-step journey of a coding agent attempting a benchmark task. This is the reality for teams evaluating agent performance on standardized tests like TerminalBench2 or SWEBench-Pro. The sheer volume of data makes manual analysis impossible, yet the patterns hidden within are crucial for improvement.
This guide walks you through the process that led one Copilot Applied Science researcher to create eval-agents — a system that automates the intellectual toil of trajectory analysis. By following this approach, you can apply agent-driven development to your own workflows, enabling faster iteration, easier collaboration, and a shift from reactive analysis to proactive innovation.
The core principles are simple:
- Make agents easy to share and use within a team.
- Make authoring new agents straightforward.
- Treat agents as the primary vehicle for contributions.
Whether you’re an experienced engineer or a curious beginner, this guide will help you unlock a new level of productivity with GitHub Copilot.
Prerequisites
Before diving in, ensure you have the following:
- GitHub Copilot – installed and activated in your preferred IDE (VS Code, JetBrains, etc.).
- Basic understanding of coding agents – familiarity with concepts like trajectories, benchmark evaluations, and agent loops.
- A target evaluation dataset – e.g., TerminalBench2 or SWEBench-Pro (or any JSON-based trajectory data).
- Python (3.8+) – for scripting and data analysis.
- Git – for version control and collaboration.
- Optional: Experience with function calling APIs or custom Copilot extensions, though not required.
Step-by-Step Instructions
1. Analyze the Problem: Understanding Trajectory Data
Start by examining a typical trajectory file. Each task in a benchmark generates a JSON file that lists the agent’s thoughts and actions. For example:
{
"task_id": "swebench-pro_00123",
"steps": [
{
"thought": "I need to find the file that contains the bug...",
"action": "cat src/main.py",
"observation": "File content..."
},
...
],
"final_result": "pass"
}
Your goal is to identify common failure patterns, successful strategies, or performance bottlenecks. With dozens of tasks and multiple runs, manual inspection is impractical.
2. Using Copilot to Surface Patterns
Open one trajectory file in your IDE. Let GitHub Copilot help you by typing comments that describe what you want to extract. For instance:
# Load trajectory JSON
# Find all steps where the agent made an error
# Count how many steps involved file reading vs. editing
Copilot will suggest code snippets. Accept or modify them. This interactive loop reduces the lines you need to read from thousands to dozens. Document these patterns in a shared note — they’ll feed into your agent logic later.
3. Automate the Loop with eval-agents
Now, turn your ad‑hoc Copilot interactions into a reusable agent. The eval-agents system is essentially a framework that:
- Takes a set of trajectory files as input.
- Applies a series of analysis functions (written by you).
- Outputs a summary report.
Here’s a minimal example in Python:
import json
import os
def analyze_trajectory(file_path):
with open(file_path, 'r') as f:
data = json.load(f)
# Your analysis logic (initially developed with Copilot)
failures = [step for step in data['steps'] if 'error' in step.get('observation', '')]
return {
"task": data['task_id'],
"num_failures": len(failures),
"result": data['final_result']
}
# Run on all trajectories
results = []
for traj in os.listdir('./trajectories/'):
results.append(analyze_trajectory(f'./trajectories/{traj}'))
print(json.dumps(results, indent=2))
This script is your first agent. Extend it by making it configurable — e.g., accept a list of analysis functions as arguments.
4. Make It Shareable and Extensible
To achieve the team goals, package your code as a CLI tool or a Python package. Structure your repository like this:
eval-agents/
├── agents/
│ ├── __init__.py
│ ├── failure_patterns.py
│ └── success_analysis.py
├── data/
│ └── trajectories/
├── tests/
├── README.md
└── setup.py
Each file in agents/ exports a function. Let teammates add new agents by simply adding a new module. Use GitHub Copilot to help document and test these modules — it will suggest docstrings and test cases as you write.
5. Author New Agents Using Existing Ones
Encourage team members to create custom agents by forking the repository or contributing a pull request. The key is to keep the interface simple: each agent receives a trajectory object and returns a result dict. Example:
# agents/failure_patterns.py
def analyze(data):
# reuse logic from step 3
...
return {"pattern": "file_not_found", "count": 5}
Then, a master agent runs all registered agents and merges results. This modularity enables collaboration and rapid experimentation.
Common Mistakes
- Over‑automation too early: Don’t try to build the perfect system in one go. Start with manual Copilot‑assisted analysis, then automate only when the pattern becomes repetitive.
- Ignoring trajectory structure: Standardize the JSON format across benchmarks. Without consistency, your agents will break.
- Neglecting error handling: Trajectories can be malformed or missing fields. Always include try‑except blocks and log warnings.
- Forgetting to share: The whole point is collaboration. Use Git from day one, write clear commit messages, and invite feedback.
- Assuming Copilot is the final answer: Copilot is a powerful assistant, but your domain knowledge guides it. Always review generated code for correctness.
Summary
By combining GitHub Copilot’s on‑the‑fly pattern recognition with the automation power of custom agents, you can eliminate the intellectual toil of analyzing massive evaluation datasets. The eval-agents approach reduces the challenge from reading hundreds of thousands of lines to maintaining a small collection of shared, reusable analysis scripts. Your team gains speed, consistency, and the freedom to focus on creative problem‑solving. Start small, iterate quickly, and let Copilot handle the boilerplate — you handle the breakthroughs.