Gemini 4 Carbon: what evidence is still missing between test scores and real-world defence?

Google announced Gemini 4 Carbon on October 28, planning access for selected cyberdefenders this week; its self-reported unguardrailed SEC-bench Pro score was 88.2%.¹

Archival photograph: Googleplex plaza, Mountain View, California, United States, July 18, 2014. Photograph: Runner1928 / Wikimedia Commons; CC BY-SA 3.0.

A test score cannot simply become a real-world success rate

A result of this kind first answers a specific question: how did the model perform on a defined set of tasks, under particular scoring rules and operating conditions? It cannot be converted directly into the proportion of real-world vulnerabilities discovered, let alone automatically cover different codebases, permission configurations and every attack scenario. The boundaries of an experimental task differ from those of the systems real organisations must manage.

An evaluation should therefore retain both the result and its provenance. A vendor's disclosure provides a starting point to test, not proof that an independent team has reproduced it or that users have achieved the same result in their own systems. Without field validation, the most accurate approach is to preserve the questions still to be answered, rather than turn test figures into defensive benefits already achieved.

Moving from benchmarks to deployment requires another layer of validation

The US National Institute of Standards and Technology (NIST) offers a concrete approach to this distinction in its AI Risk Management Framework. MEASURE 2.1 calls for documentation of test plans, metrics and tools; 2.3 emphasises measuring performance under conditions as close as possible to deployment; 2.4 calls for monitoring behaviour in operation; and 2.5 calls for documenting how far results can be generalised and what their limitations are.²

When using this framework to understand a new model, the point is not to list more leaderboards but to check whether the test conditions match the intended use. Whether a finding can be reproduced, whether a proposed fix introduces new problems, and which systems the model can access all need assessment within a specific workflow. These are questions for validation, not assertions that Carbon has produced false positives, introduced vulnerabilities or passed a certification process.

The framework is a voluntary approach published in 2023, not a new regulatory approval for this model. Citing it does not mean that NIST has evaluated, approved or endorsed the product.²³

Limited access must still be distinguished from general availability

In security, stronger discovery or validation capabilities have a dual use: the same capability may help defenders identify weaknesses while also increasing the risk of misuse. Access permissions, the operating environment and continuous monitoring therefore cannot be treated as peripheral matters unrelated to performance.

For general readers, the distinction for now is between arrangements set out in an announcement and something already available in their own account. This review did not test access or establish an independent record of deployment outcomes. Meaningful new evidence would include a verifiable scope of access, reproducible third-party evaluations and actual performance in defensive work—not the same score repeated in more reports as though it were a real-world result.

Source notes

1. 2026-10-28 · Gemini 4 Carbon: our next frontier model · Google. ↩

2. 2023 framework · 5 AI RMF Core, MEASURE 2.1, 2.3, 2.4 and 2.5 · NIST; an explanatory paraphrase of the framework's methods. Republished courtesy of the National Institute of Standards and Technology. ↩ ↩²

3. 2023-01-26 · Artificial Intelligence Risk Management Framework (AI RMF 1.0) · NIST; historical methodological context, not an evaluation of this product. ↩