Artificial Analysis Finally Scored Calibration. OpenAI's Decisions API Lands 13th of 17 Decision Models, Below Jev, While the Same Luna Run as a Plain LLM Scores 76.
Artificial Analysis released Decision Index v2 on November 8 with a metric that scores correctness and confidence together, 50 times one plus accuracy minus Brier, across six evaluations, 45% of them private. GPT-6 Luna through OpenAI's Decisions API scores 61.7, thirteenth of seventeen decision models and behind TypeSafe's Jev 1.14 at 67.0; the same GPT-6 Luna run as an ordinary LLM at max effort scores 76.4, second overall behind Claude Haiku 5.5 at 77.1. The top decision models, Drex Max at 73.4, GLiDE at 72.8, and Mammoth 27B at 72.4, cost a fraction of the LLMs per task and run in milliseconds; one of the six evaluations is built by TypeSafe, which sells Jev. v1 and v2 scores are not comparable. We read AA's article and evaluation page.
By T.W. Ghost
Three times this fall we wrote that OpenAI publishes no calibration data for its Decisions API, the typed-answer endpoint that returns a probability with every choice. Today, November 8, 2026, Artificial Analysis published a board that measures it. Decision Index v2 scores each answer on correctness and confidence together: 50 times one plus the correctness minus the squared gap between the stated probability and the outcome, which works out to 50 times one plus accuracy minus Brier. A model that is right and sure scores high; a model that is right and hedging, or wrong and sure, scores lower.
The result for OpenAI's product is the catch. GPT-6 Luna through the Decisions API scores 61.7, thirteenth of seventeen dedicated decision models, below TypeSafe's Jev 1.14 at 67.0 and below Microsoft-Decision-1 at 62.8. The same GPT-6 Luna, run as an ordinary language model at max effort, scores 76.4, second on the whole board behind Claude Haiku 5.5 at 77.1. The model is not the problem. The packaging is, or the way the Decisions API reports its probabilities. The other catch is money and time: the LLMs that top the board cost fifteen to sixty-one times more per task than the best dedicated decision model and run forty to one hundred twenty times slower. We read AA's article and its evaluation page. Here is what held up.
The Board, by Who and What
| Model | Decision Index v2 | Cost per 1M tasks | Time per task |
|---|---|---|---|
| Claude Haiku 5.5 (max), LLM | 77.1 | $6,240 | 32.65 s |
| GPT-6 Luna (max), LLM | 76.4 | $1,530 | 11.34 s |
| Drex Max v1.1 (Nace AI) | 73.4 | $102 | 275 ms |
| GLiDE v1.2 (Fastino) | 72.8 | $780 | |
| Mammoth 27B (Base Labs) | 72.4 | ||
| GPT-6 Luna (low), LLM | 69.2 | ||
| Jev 1.14 (TypeSafe) | 67.0 | 92 ms | |
| Claude Haiku 5.5 (low), LLM | 66.7 | ||
| Microsoft-Decision-1 | 62.8 | ||
| GPT-6 Luna via Decisions API | 61.7, 13th of 17 decision models | ||
| d1; Decider Flash; Clef; Clef-Omni | 61.6; 61.6; 60.2; 59.3 | Decider v1.2 $68; Clef $859 | Celeris-1.1 $14 at 56 ms; Elephant 9B 70 ms |
The product sits near the bottom and its own model sits near the top. Luna as an LLM at 76.4 and Luna through the Decisions API at 61.7 are on the same board, measured the same way. That gap is the number OpenAI should be asked about.
Calibration costs money in both directions. Haiku 5.5 at max is the most calibrated thing on the board and costs $6,240 per million tasks; Drex Max scores 73.4 at $102 and 275 milliseconds. For a routing layer that is the whole trade.
Billing shape beats list price. GLiDE bills the document input on every question, so it costs about eight times Drex Max per task despite a list price only about a third higher. AA's own note, and the kind of thing a per-token comparison hides.
What Is in the Index
| Evaluation | Weight | Note |
|---|---|---|
| AA-Paperclip | 30% | 1,500 private documents; a document counts only if all 15 answers are right; pass rates Haiku 5.5 max 92%, Luna max 86%, Drex Max 63%, GLiDE 62%, Decider 55% |
| EvalSafe-O*NET-AA | 15% | Built by TypeSafe, which sells Jev |
| When2Call | 10% | |
| AA-Omniscience-Decision v1.1 | 15% | |
| HLE-MultiChoice v1.1 | 15% | |
| AA-Symmetry | 15% | Rewards abstention; AA notes Base Labs models lead partly by confidently saying they do not know |
| WildGuardTest | removed | |
| Private share | 45% | Up from 25% in v1 |
Three honest observations:
- v1 and v2 do not compare. Luna through the Decisions API scored 48.5 and eleventh of thirteen on v1 in October, and Jev 57.1. New metric, new weights, more private data; "Luna rose from 48.5 to 61.7" is a sentence that should not be written.
- One benchmark belongs to a vendor on the board. TypeSafe built EvalSafe-O*NET-AA and sells Jev. AA discloses it; so do we. It is 15% of the score.
- Forty-five percent private is the same trade as Intelligence Index v5. Harder to contaminate, impossible to reproduce. Readers get AA's judgment on data they cannot see, which is the same caveat we gave the main index.
Verified vs Unconfirmed: The Scorecard
| Claim | Verdict |
|---|---|
| Decision Index v2 released Nov 8, 2026; metric definition; weights; 45% private | Verified (AA article; evaluation page) |
| Scores for decision models and LLMs as listed | Verified (AA evaluation page) |
| Luna via Decisions API 13th of 17 decision models | Our count from AA's page data |
| Cost per 1M tasks and time per task figures | Verified (AA article) |
| GLiDE per-document billing makes it about 8x Drex Max per task | Verified (AA's note) |
| Paperclip pass rates | Verified (AA article) |
| v1 figures: Luna-Decisions 48.5, Jev 57.1 | Our October log; not comparable to v2 |
| EvalSafe-O*NET-AA built by TypeSafe | Verified (AA evaluation page) |
Who Should Use It
Use a dedicated decision model if your routing layer needs calibrated probabilities at scale: Drex Max scores 73.4 against the best LLM's 77.1, at $102 per million tasks against $6,240 and in 275 milliseconds against 32 seconds, by AA's measurement.
Skip the Decisions API for calibration until OpenAI explains why its packaging scores so far below its own model, and skip any comparison that mixes v1 and v2 numbers.
The frontier is elsewhere: the most calibrated model on the board is a general LLM at max effort, and the second most calibrated is the one OpenAI wraps in the product that lands thirteenth. Our model comparison has the current standings side by side.
Sources
- Artificial Analysis: Decision Index v2
- Artificial Analysis: Decision Index evaluation page
- Artificial Analysis on X: Decision Index v2
*Not sure whether your branching logic needs a decision model or a calibrated LLM? Take the free 2-minute quiz and get matched. Then read the beta post where we first asked for the calibration number: OpenAI's Decisions API.*