
AI-generated editorial illustration of practical cybersecurity tools.
Google has added HelpBench to its Gemini 4 Carbon safety tests. The test checks AI cybersecurity advice. It looks at the tips models give about risks online.
Google's November 17 update revisits work with University College London. The first paper came out in June. It checked 18 models and 40,500 answers.
HelpBench draws on real questions
The test has 450 questions on how to stay safe online. They cover scams, lost accounts, device concerns and more.
The paper says the team drew questions from Reddit posts. It changed details to help protect the people who wrote them.
Experts wrote rules to score each answer. The rules check the facts and how well the advice fits. An answer can lose points for harmful or needless steps. It can also lose points for too much detail.
The rules ask if the answer gives the facts a user needs. They also check for steps that could cause harm. The test looks at how well the answer sets out what to do first.
The questions use English text and need just one turn. The team plans tests in more languages and with images.
The same question gets different answers
Models scored 82% on average against the expert rules. About one in ten answers scored less than 65%. The team linked low scores to poor or harmful advice.
Google's new post also describes wide swings between repeated answers. The team asked the same question five times. In some cases, the spread was large. The standard deviation reached 12 points.
One weak answer told a user to discard a device. The question was about how to deal with malware. Experts gave steps that would cost less time and money.
A test for the newer model
Google says Carbon scores better than Argon and Gemini 3.1 Pro. The new report uses HelpBench to check those gains. The earlier study supplies the test and its scoring rules.
Our Carbon report covers the model's wider agent features. The data and safety guide explains data and account controls. HelpBench checks how well the advice meets a user's needs.
Read also
Popular news

Google brings Gemini to Android Go phones with 2GB RAM

XPeng Bets Everything on Physical AI and Robotics at Upcoming Tech Summit

AI Boom Will Push Data Center Carbon Emissions Up 70 Percent by 2030

Microsoft recruits more than 40 groups to make AI agents auditable

Replit Vibe Coding Startup Fuels Campaign to Reinstate Ousted Nvidia CEO


