Chaen.ai Work with me

AI news

Gemini 4 Carbon Benchmark: No.1, but No Public API

🗓 ⏱ 6 min read

中文日本語

Gemini 4 Carbon Benchmark: No.1, but No Public API

Gemini 4 Carbon benchmark: Google scored 59 to top Artificial Analysis's Intelligence Index, one point over Opus 5.5. No public API yet — how I read it.

Google’s Gemini 4 Carbon (high) scored 59 on the Artificial Analysis Intelligence Index and took the top spot. Claude Opus 5.5 (max) sits one point behind at 58. But on the chart Gemini 4 Carbon is marked as having no public API, and the announcement says nothing about availability or pricing. So my read is less “the leaderboard changed” and more “this starts to matter when you can actually call it.” Below: how I read the spread, how I use a composite index when choosing models, and what teams — especially teams in Japan — can usefully do while the API is still missing.

Gemini 4 Carbon benchmark: how the top scores line up

The top of the index looks like this:

  • Gemini 4 Carbon (high) — 59
  • Claude Opus 5.5 (max) — 58
  • GPT-6 Astra (max) — 57
  • GPT-6.1 Sol (max) — 55

First, second and third land on 59, 58 and 57 — one point apart each. Fourth is two points below third. Google is in front, but the distance between the leaders is almost nothing. Demis Hassabis’s post about it was a single phrase: “No.1.” That is the whole message; there is nothing else in it to read.

Why I don’t read a one-point gap as a win

I described the top of this list as a three-way tie, and the reason is plain arithmetic. A composite index is built by combining several separate evaluations into one number. If any one of those components shifts slightly on the next run, the order at the top can flip. Treating a single point as “Google won” leaves you with nothing to say internally when the next update swings the other way.

The useful signal is not the order; it is that the leaders are bunched. Whichever vendor you pick, the raw capability of the frontier model now sits in roughly the same band. That moves the deciding factors somewhere else: can you call it, what does it cost, where does it run, and how is your data handled. This announcement answers none of those — and the fact that it answers none of them is itself part of the information.

Every score carries a setting

One detail that is easy to skip: each score comes with a setting attached. Gemini 4 Carbon is reported at (high), while Opus 5.5, GPT-6 Astra and GPT-6.1 Sol are at (max). What those labels mean in practice, and how cost differs between them, is not something this post tells us. But it does mean that when you compare numbers you need to check which setting each was measured at. Strip that away and you are no longer comparing like with like.

“No public API” is the heaviest line in the chart

For anyone running AI inside a company, the most important detail here is not the 59. It is that Gemini 4 Carbon is treated as having no public API. A model you cannot call is a model you cannot put into a workflow, and with no stated availability date or price it cannot go into a plan either.

Meanwhile the highest-scoring Gemini you can actually reach through an API is Gemini 4 Argon (high) at 50 points — seventh overall. Fifty-nine versus fifty. There is a gap between the number in the headline and the number you can call today, and mixing the two inside a company causes concrete confusion. If “Gemini is number one” is the only line that circulates internally, it will not match what your engineers feel when they test the model they can actually reach. I keep two lists separate:

  • the best model among the ones we can call today
  • announced scores for models we cannot call yet

That single habit removes a surprising number of pointless meetings.

How to read the Artificial Analysis Intelligence Index v5.0

Artificial Analysis is a third-party benchmark organisation; it does not build the models, which is what makes its numbers easier to compare across vendors than first-party claims. Version 5.0 of the Intelligence Index is a composite of nine evaluations, including Terminal-Bench 4.0, FrontierCode 2.0, HLE-Diamond and GDP.pdf.

A composite is convenient and it has a personality. The moment nine evaluations become one score, strengths and weaknesses dissolve into an average: a terminal-operation result and a document-processing result are added as the same kind of point. So I use the index to narrow the field, then read the components. If your work is mostly code generation, the component that matters is not the one that matters for long-document reading. The model that ranks first overall is not automatically first for your job.

The order I work in

  1. Use the composite to shortlist a handful of models.
  2. Read the individual evaluations closest to your actual workload.
  3. Check whether there is a public API.
  4. Run a short test on your own data.

Skip step three and you end up building a plan around something you cannot buy. Skip step four and “it ranked first” becomes your only justification — which is exactly the kind of reasoning that falls apart one month later.

What this means in Japan — and for global AI companies selling here

For people driving AI adoption inside Japanese companies, the immediate practical impact of this news is small: a model without an API does not enter a procurement discussion. It is still worth filing away, for one reason. If the frontier keeps clustering within a point or two, the value of long internal debates about which model is smartest goes down, and the value of being able to measure quickly goes up. Build the evaluation harness first, and the day a new model arrives with a public API your comparison takes days rather than months. The gap between a company that restarts the debate every time the ranking moves and a company that can simply measure on its own data shows up right there.

This also matters for how approvals work here. In Japanese companies a written proposal often has to survive several layers of review, and “ranked number one on a benchmark” is an easy line to put in a deck — but it is weak support, because the ranking can change before the project ships. I advise putting results measured on your own operational data in the main argument, and leaving external benchmarks as supporting context.

For AI and tech companies outside Japan that want traction here, the shape of this announcement is itself a lesson. A one-word “No.1” travels well in English-language social media; it generates coverage with no explanation attached. But a Japanese buyer cannot move an organisation on a ranking. The questions that actually decide things are: when can we use it, what does it cost, how does it behave in Japanese, and where does our data sit. A score answers none of them, and until they are answered you tend to get articles without evaluations. The inverse is the opportunity: an announcement that lands availability, pricing and access at the same time is unusually strong in this market. If you must split the news into a capability announcement and an availability announcement, put the Japan-specific detail into the second one and treat that second one as the real launch.

Putting it to work

Three things you can do with this news, none of which require the model itself.

  • Keep one table of the models you can actually call today mapped against your use cases, with announced-but-unavailable models in a separate column. It stops a headline from distorting your roadmap.
  • Build a small evaluation set out of your own work — ten or twenty real tasks is enough. The point is to be able to re-run the same tasks the day something new ships.
  • Write internal updates starting from “available / not available,” not from rank.

The leaderboard will move again. Being able to measure within a week is worth more than knowing who is one point ahead this month.

FAQ

Can I use Gemini 4 Carbon right now?

At the time of this announcement it is marked on the chart as having no public API, and no availability date or pricing is given. The highest-scoring Gemini you can reach through an API is Gemini 4 Argon (high) at 50 points, seventh overall.

What is the Artificial Analysis Intelligence Index?

It is a composite score published by Artificial Analysis, a third-party benchmark organisation that does not build models itself. Version 5.0 combines nine evaluations, including Terminal-Bench 4.0, FrontierCode 2.0, HLE-Diamond and GDP.pdf.

Does a one-point lead mean Gemini is actually better?

The top three land on 59, 58 and 57 — one point apart each — so I read them as effectively tied. Because the index is a composite of several evaluations, a small shift in any component can reorder the top.

Based on a post on XOriginal post ↗

Share this article

XFacebookLINEはてブ

About the author

Masahiro Chaen
Masahiro Chaen Founder & CEO, DigiRise Inc.

CEO of DigiRise Inc. Has supported 600+ companies and 50,000+ people with AI training, consulting and development. GMO AI & Crypto advisor, Google Gemini AI advisor. 200K+ followers on X.