
A set of three unmarked mechanical key switches sits in a precision test fixture on a workshop bench. AI-generated editorial illustration.
Cline-Bench puts AI coding agents through the same software task three times. The test covers 47 requests drawn from real work. Its results show how often an agent got the job done on the first try and how often it could do so every time.
The new results come from the people building Cline, an AI coding tool. The project starts with one request at a time in an existing codebase. That gives the test a clear job and a way to check whether the changes broke things the software already did.
Why Cline-Bench runs the same job three times
The team’s benchmark report gives three scores. The first is the share of runs that pass. Another measures how many tasks passed at least once in three runs. A third counts tasks that passed all three times.
One way to read the scores is to imagine three runs of a small bug fix. If two runs pass, the task counts toward the score for getting it right at least once. It does not count toward the score for getting it right on all three tries. The two numbers answer different things a team might want to know.
The results show why that matters. Gemini 4 Argon had the highest mean success rate in the table at 51.1 percent. Opus 5.5 had a higher share of tasks it passed on all three runs, at 42.6 percent compared with Argon’s 40.4 percent.
How the test checks an agent’s work
Each task starts with a user’s request and a version of the code that can run. There are checks for the change the user asked for and for parts of the software that should keep working. The team checks the submitted work in a fresh environment and counts missing or skipped required tests as failures.
That setup matters for a project that wants to measure more than a nice-looking patch. A change can seem to work from a short read of the code. A test can find a case it missed. Tests for older behavior can also catch a fix that solves one problem and creates another.
The Cline-Bench project repository gives more background on where the requests came from. There are still limits to what one set of projects can show. The work is a test of the agents, tasks, and setup in the report.
What a coding team can take from the results
The report covers one turn per task. A developer may work in several turns, asking the agent to inspect a failure or explain a choice before making a new attempt. Cline says it plans to test more of those cases in future work.
A team can also decide what makes a pass in its own setting. A short patch may need a test for how it handles old data or for what a user sees when a step fails.
For teams choosing a model, the scores offer a reason to look at repeat runs on their own code. A tool that works well on the second try may be fine when a developer has time to check it. More repeatable work may matter for jobs that run often. See our comparison of DeepSeek and Claude and our guide to agent design and review for the choices around the model itself.
Read also
Popular news

AWS lets AI agents buy select tools on the developer’s cloud bill

Human clues helped AI agents mimic supply-chain decisions

ElevenLabs moves to buy Kotoba for its Japanese voice AI

Claude Code vs Cursor: Try the Same Fix in Both

US House lawmakers launch a bipartisan caucus on AI safety


