Chaen.ai Work with me

AI news

Gemini 4 Carbon Benchmarks: Long Context and Security Lead

🗓 ⏱ 6 min read

中文日本語

Gemini 4 Carbon Benchmarks: Long Context and Security Lead

Gemini 4 Carbon benchmarks: it tops AA Index v5.0 at 59 and leads on long context, automation and security, but trails on coding and computer use.

Google has announced its next-generation model, Gemini 4 Carbon. It takes first place on the composite index, and it arrived just two weeks after the previous model, Argon. I haven’t been able to try it yet, so nothing below is a hands-on impression. This is a reading of the published Gemini 4 Carbon benchmarks: where it is genuinely ahead, where it still loses, and how a company should decide what to verify first — including what the release looks like from Japan, where I work.

What Google announced about Gemini 4 Carbon

The headline figures:

  • AA Index v5.0 composite score: 59, first place
  • GraphWalks long-context reading: 96.2%
  • AutomationBench business automation: 53.9%
  • SEC-bench Pro vulnerability discovery: 88.2%

Three details matter as much as the scores.

First, access is staged. Cyber-defense specialists get it first, with developers, enterprises and general users following in stages. This is not a model that anyone can open today.

Second, pricing. During the introductory period it is $2 per million input tokens and $10 per million output tokens — the same as Argon. After that period it moves to $4 and $20.

Third, two quality claims beyond raw capability: ungrounded assertions are down to roughly half of Argon’s level, and the reported success rate of prompt injection attacks is 0%.

And one caveat Google states itself: the figures for other companies’ models are those vendors’ self-reported numbers, not independent re-runs under identical conditions.

Honestly, the number that sticks with me is not any benchmark. It is two weeks. A frontier model replaced its predecessor in two weeks. If your organisation evaluates AI on a six-month cycle and buys on an annual one, the model you were evaluating will have been replaced twice before the paperwork clears. That gap — between release cadence and decision cadence — is the real story for most companies.

Where Gemini 4 Carbon pulls ahead: long context, automation, security

Put the three models side by side (Carbon / Opus 5.5 / Astra):

  • Composite AA Index: 59 / 58 / 57
  • Long-context reading up to 1M tokens: 96.2% / 66.8% / 71.8%
  • Zapier business automation: 53.9% / 42.5% / 41.4%
  • Terminal-based coding: 65.2% / 59.6% / 59.1%
  • ExploitGym attack validation: 54.8% / 34.5% / 42.4%

Look at the composite first: 59, 58, 57. A headline that says “world-class best model” implies a gap. The composite gap is one point. If you only read the ranking, you will over-read it.

The separation lives in individual tasks, and the largest by far is long-context reading: 96.2% against 66.8% and 71.8%. That is close to a thirty-point spread on documents up to a million tokens. Of everything announced, this is the number most likely to change what you can actually ship.

Security is the second standout. Carbon is reported first in all three security categories — finding vulnerabilities, validating exploits, and fixing them. On ExploitGym it leads second place by more than twelve points. The 0% prompt injection figure is hard to interpret without the test conditions, but read together with the rollout order, it tells you exactly where this release is aimed.

Business automation is third: 53.9% against 42.5% and 41.4% on Zapier workflows. Worth noting that a bit over half still means close to one task in two does not complete. This is a step forward, not an autopilot.

Where Gemini 4 Carbon loses: hard coding and computer use

It is not a clean sweep, and any write-up that omits this part is not worth much.

  • FrontierSWE, hard real-world coding: GPT-6 Astra 65.5% vs Carbon 61.2%
  • OSWorld, operating a PC: Claude Opus 5.5 87.2% vs Carbon 81.5%

The interesting part is that “coding” has stopped being one capability. Carbon is first on terminal-based coding at 65.2% and behind on FrontierSWE, which is closer to real engineering work. Calling a model “good at coding” is now too coarse to make a decision with; you have to say which kind of coding.

The same applies to agents that drive a computer. Opus 5.5 leads OSWorld by about six points, so if your use case is clicking through a desktop application, the ranking you care about is not the composite one.

Add the self-reported caveat and the practical conclusion is simple: treat these rankings as a shortlist generator, not a verdict. Choosing different models for different jobs will stay normal for a while yet.

Pricing and rollout order: what they actually tell you

The introductory price holds Argon’s rate, then doubles. That matters more than it sounds. If you size a budget on $2/$10 and then ship into the standard period, your cost doubles while your product stays identical. Run evaluations while it is cheap, and build the business case at $4/$20.

The rollout order is itself a signal. Security specialists first, everyone else in stages. The positioning and the distribution agree with each other. For planning, it also means the capability you are most excited about may not reach you for a while — so spend the waiting time preparing inputs rather than refreshing the announcement page.

What Gemini 4 Carbon means in Japan

AI adoption in Japan is heavily document-driven: internal rules, contracts, meeting minutes, specifications, and long approval trails. That makes the long-context result unusually relevant here. A thirty-point gap on million-token reading maps directly onto the work Japanese companies actually hand to AI today.

For Japanese enterprises

Start where the gap is largest. Pick one workflow that involves reading long, cross-referenced documents and run it against your current model under the same conditions. That is the cheapest way to find out whether the benchmark gap shows up in your own work.

Be careful with the automation number. It was measured on Zapier. Japanese operations more often sit on core systems and layered approval flows, so a score from a SaaS-connector benchmark will not transfer unchanged. Build your own set instead: ten representative tasks, fixed, each with a known good answer. Re-run the same ten every time a model ships. Teams that have this do not panic at a two-week release cycle; teams that don’t will re-litigate vendor choice from scratch every time.

For security use cases, accept that you are in a queue. The useful work right now is preparation: decide which repositories and assets you would point it at, and secure permission to test, so that when access arrives you can measure in days instead of months.

For global AI companies announcing in Japan

  • Lead with tasks, not rank. The composite gap is one point; “what gets faster in which job” travels much further in Japanese media than “number one”.
  • Say who can use it today. Staged rollouts generate confused inbound when the first paragraph does not name the eligible group.
  • Publish introductory and standard pricing together. Japanese internal approval needs an annual cost figure, and an intro-only price will not clear review.
  • State plainly that competitor figures are self-reported. In this market that one line buys more trust than a stronger claim does.

Understated, verifiable messaging lasts longer here than a big number.

How to apply this inside your own company

Three concrete moves. Choose one long-document task and compare Carbon against what you use now, as soon as you have access. Freeze a ten-task evaluation set and run it on every model release, so comparison costs you an afternoon rather than a quarter. Budget at the standard price, not the introductory one. When the frontier moves every two weeks, the ability to measure quickly is worth more than the ability to pick correctly once.

FAQ

Can I use Gemini 4 Carbon today?

Not necessarily. Access starts with cyber-defense specialists, and developers, enterprises and general users follow in stages. It is not open to everyone at announcement.

How much does Gemini 4 Carbon cost?

During the introductory period it is $2 per million input tokens and $10 per million output tokens, the same as Argon. After that it moves to $4 and $20, so budget at the standard price.

Is Gemini 4 Carbon better than GPT-6 Astra and Claude Opus 5.5 at everything?

No. GPT-6 Astra leads FrontierSWE at 65.5% against Carbon's 61.2%, and Claude Opus 5.5 leads OSWorld at 87.2% against 81.5%. Carbon is first on long context, business automation and security, and the competitor figures are vendor self-reported.

Based on a post on XOriginal post ↗

Share this article

XFacebookLINEはてブ

About the author

Masahiro Chaen
Masahiro Chaen Founder & CEO, DigiRise Inc.

CEO of DigiRise Inc. Has supported 600+ companies and 50,000+ people with AI training, consulting and development. GMO AI & Crypto advisor, Google Gemini AI advisor. 200K+ followers on X.