Rethinking AI Benchmarks: Why Context and Collaboration Matter More Than Scores

by gertytechLeave a comment 4 min read

The Problem with Traditional AI Benchmarks

For years, the AI community has relied on standardized benchmarks to measure progress. These tests—whether in coding, math, or language—pit AI models against humans in isolated, controlled tasks. The results are easy to compare, rank, and headline: “Model X outperforms humans on Y!” But as Angela Aristidou argues in her recent MIT Technology Review op-ed, this approach is fundamentally flawed. AI is rarely used in the way it’s benchmarked. In the real world, AI operates within complex, collaborative environments—teams, workflows, and organizations—where its performance emerges over time, not in a single test session.

The gap between benchmark scores and real-world utility is stark. High-scoring AI models, like those approved for medical imaging, often fail to deliver promised efficiency gains once deployed. In hospitals, for example, AI tools that ace diagnostic accuracy tests can slow down workflows when integrated into multidisciplinary teams, where decisions are iterative and context-dependent. The result? Expensive tools end up in the “AI graveyard,” and trust in AI erodes—both within organizations and among the public.

The Case for Human-AI, Context-Specific Evaluation (HAIC)

Aristidou and other researchers are advocating for a shift to Human-AI, Context-Specific Evaluation (HAIC) benchmarks. HAIC reframes evaluation in four key ways:

  1. From individual tasks to team performance: Instead of testing AI in isolation, HAIC assesses how AI functions within human teams and workflows.
  2. From one-off tests to long-term impacts: Performance is judged over extended periods, not just in a single session.
  3. From accuracy and speed to organizational outcomes: Metrics expand to include coordination quality, error detectability, and systemic effects.
  4. From isolated outputs to upstream/downstream consequences: HAIC considers how AI impacts the broader system it’s part of.

This approach is already being tested. In a UK hospital system, for example, evaluators didn’t just ask if an AI tool improved diagnostic accuracy—they studied how it affected team coordination, deliberation, and risk management over time. In the humanitarian sector, an AI system was evaluated over 18 months, with a focus on how easily human teams could detect and correct its errors, building trust through transparency and accountability.

Why This Matters Now

The limitations of traditional benchmarks are becoming impossible to ignore. As a Reddit thread from March 2026 points out, even top models struggle on benchmarks like ARC-AGI-2, which tests reasoning in interactive environments—something static tests can’t capture. Meanwhile, new benchmarks like GDPval (which spans 44 knowledge-work occupations) and HAI-Eval (which measures human-AI synergy in collaborative coding) are emerging to address these gaps.

The shift is also reflected in industry trends. A 2026 report from Mixpanel highlights that AI products are increasingly judged not by raw usage or novelty, but by their ability to deliver consistent, measurable value within real workflows. In other words, the question is no longer “Can this AI pass a test?” but “Does it make teams better at their jobs over time?”

The Road Ahead

Adopting HAIC won’t be easy. It’s more complex, resource-intensive, and harder to standardize than traditional benchmarks. But the alternative—continuing to evaluate AI in sanitized, unrealistic conditions—risks misleading us about what AI can actually achieve. As Aristidou puts it, “To deploy AI responsibly in real-world settings, we must measure what actually matters: not just what a model can do alone, but what it enables—or undermines—when humans and teams in the real world work with it.”

For developers, policymakers, and organizations, the message is clear: It’s time to move beyond the headline-grabbing scores and focus on benchmarks that reflect how AI performs in the messy, collaborative reality of work.

Resources and Further Reading: