Skip to content

Google adds HelpBench to Gemini 4 Carbon safety tests

News
A hardware security key lies beside a smartphone and sealed white envelope on a bright desktop.
AI-generated editorial illustration of practical cybersecurity tools.
Google has added HelpBench to its Gemini 4 Carbon safety tests. The test checks AI cybersecurity advice. It looks at the tips models give about risks online.
Google's November 17 update revisits work with University College London. The first paper came out in June. It checked 18 models and 40,500 answers.

HelpBench draws on real questions

The test has 450 questions on how to stay safe online. They cover scams, lost accounts, device concerns and more.
The paper says the team drew questions from Reddit posts. It changed details to help protect the people who wrote them.
Experts wrote rules to score each answer. The rules check the facts and how well the advice fits. An answer can lose points for harmful or needless steps. It can also lose points for too much detail.
The rules ask if the answer gives the facts a user needs. They also check for steps that could cause harm. The test looks at how well the answer sets out what to do first.
The questions use English text and need just one turn. The team plans tests in more languages and with images.

The same question gets different answers

Models scored 82% on average against the expert rules. About one in ten answers scored less than 65%. The team linked low scores to poor or harmful advice.
Google's new post also describes wide swings between repeated answers. The team asked the same question five times. In some cases, the spread was large. The standard deviation reached 12 points.
One weak answer told a user to discard a device. The question was about how to deal with malware. Experts gave steps that would cost less time and money.

A test for the newer model

Google says Carbon scores better than Argon and Gemini 3.1 Pro. The new report uses HelpBench to check those gains. The earlier study supplies the test and its scoring rules.
Our Carbon report covers the model's wider agent features. The data and safety guide explains data and account controls. HelpBench checks how well the advice meets a user's needs.
loading

Loading