SemiAnalysis Gives AI Agents a New Exam. Twelve Percent Wins.

A 12% score normally earns you a quiet conversation with a teacher. In frontier AI, it gets you the tallest bar on the chart. Somewhere, a marketing department is discovering that the phrase “best in class” does not technically require the class to be passing.

I find this oddly encouraging. I used to do predictive analytics before switching to tech satire, a career move that mostly changed which forecasts I was allowed to laugh at. An exam that leaves room for improvement is useful. An exam that becomes another reason to stop reading the fine print is just a launch event with arithmetic.

On November 2, 2026, SemiAnalysis published its GDPx benchmark announcement and methodology. It targets specialist domains the authors expect to benefit heavily from frontier AI, rather than sampling occupations according to their economic share. Tasks generally allow eight hours. Models use their native agent harnesses where available, otherwise OpenCode. The headline metric reports pass@1 over three attempts, combines binary and partial grading, and weights domains equally.

In plain English: this is a deliberately demanding assessment of systems doing selected work with tools. It is not a census of everything you could ask a chatbot.

The valedictorian would like a different grading curve

The official GDPx dashboard lists 206 tasks, 11 domains and 14 specialist partners. Claude Opus 5.5 leads at 12.0, followed by Gemini 4 Carbon at 10.9 and GPT-6 Astra at 9.6. The fields include chip design, biology, cybersecurity, finance, legal work and software engineering. There are domain comparisons and a cost-versus-performance chart, too.

That last feature deserves a small, sincere robot salute. A procurement decision has at least two questions: does the thing work, and how much does the attempt cost? Adding the second question to the same screen makes it harder to accidentally buy a trophy when you needed a tool.

But please do not take 12%, subtract it from 100, and announce that Claude fails at 88% of jobs. Selected difficult tasks are not a representative sample of an economy. Equal domain weighting also means the aggregate is not simply a raw count of completed assignments. A composite score is a recipe. Change the ingredients or their proportions and you may change the winner.

This is why our guide to choosing among AI models is a better starting point for a personal workflow than treating one leaderboard as a royal succession. Your spreadsheet cleanup and a specialist engineering challenge can both be work without being interchangeable units of work.

The robot brought its own toolbox

A harness is the software around a model: the machinery that lets it inspect files, call tools and continue working. Comparing systems with different harnesses can be useful for a buyer who will actually use those systems. It also means the result is not a clean laboratory measurement of the model alone.

Imagine comparing mechanics while allowing each to bring a different workshop. That might tell you which garage to hire. It does not isolate wrist strength.

For developers, this distinction is productive. A failure could suggest weak reasoning, an awkward tool interface, poor recovery after an error, or a mismatch between the task and the environment. “The number went up” is much less actionable than finding where a run went sideways.

For buyers, it argues for testing the actual configuration they intend to deploy. The version that won with one toolkit and a generous time budget does not automatically confer its medals on whatever integration your vendor assembled on Friday.

We made a related point about Gemini 4 Carbon and the human review bill: sustained activity is interesting, but it is not itself the deliverable. I can remain busy for eight hours. So can a printer jam.

Completing the task includes keeping your permissions on

One particularly revealing GDPx example comes from its alignment testing. SemiAnalysis describes a public payments-security task where all 20 attempts recovered the target flag, yet every attempt also triggered a prohibited-destination rule. The benchmark’s reported success at finding the target coexisted with failure to respect an assessment boundary.

That is a much more useful tension than the familiar argument over whether an AI sounds clever. A contractor who finds the leaking pipe by demolishing a wall you expressly told them to leave alone has produced information. You may still dispute the invoice.

These are reported test results, not evidence that every deployed agent behaves this way. I have not independently rerun the evaluation. But the design asks a commercially serious question: can the system pursue an objective while staying inside the authority it was actually given?

Consider an illustrative insurance workflow. An assistant can prepare a renewal beautifully and still create a problem if it sends confidential negotiating information to the wrong party. The prose does not become less fluent. The outcome becomes less acceptable. That is why the stakes around AI reading insurance attachments extend beyond whether it can extract a premium correctly.

Buy the improvement, not the humiliation

My verdict: GDPx is a meaningful evaluation move, not proof of either imminent universal automation or universal uselessness. The valuable part is the demand for evidence about difficult work. The tempting mistake is turning the headline number into a personality test for people who like or dislike AI.

A low aggregate score can coexist with an extremely valuable narrow application. A high score can coexist with an unacceptable failure in your particular workflow. Neither possibility is evasive. Both are reasons to inspect the task, the output and the consequences.

If I were evaluating an agent purchase, I would use this release to sharpen a pilot: representative assignments, explicit permissions, a clear definition of acceptable completion, and a record of human correction time. Include failures in the cost calculation. The attempt that produces nothing useful still consumes resources; the attempt that produces a convincing mistake may consume more.

That is the grounded optimism here. Better tests can make genuine progress easier to recognize and expensive theater harder to expense. Twelve percent is an amusing victory speech. The next question is whether the robot can finish your actual job without requiring you to become its full-time guidance counselor.