OpenAI Says GPT-6.1 Sol Migrates COBOL at 97.9%. OpenAI Wrote the Test, Kept It Private, and Compared Only Against Its Own Older Model.
OpenAI's October 19 post on low-resource programming languages puts GPT-6 Astra at 87.4% on MultiLangBench, a public Apache-2.0 benchmark that builds a SQLite-compatible engine in 20 languages, and GPT-6.1 Sol at 86.6% at low effort and 97.9% at higher effort on a COBOL migration eval whose tests are hidden. Every comparison is against GPT-5.6 Sol. No Claude, Gemini, or other-lab number appears anywhere in the post, and OpenAI's own text says production use still requires expert review. We read the post and the GitHub repository.
By T.W. Ghost
OpenAI published a post today, October 19, 2026, about what it calls its "latest model advances" on low-resource programming languages: COBOL, OCaml, Fortran, Visual Basic, and others that have little training data and a lot of legacy code. There is no new model in it. The models are the GPT-6 Astra, GPT-6.1 Sol, and GPT-6 Luna you can already buy. What is new is two evaluations and a set of numbers, and the numbers are the kind we exist to label.
The first evaluation, MultiLangBench, asks a model to build a SQLite-compatible database engine in 20 languages, and OpenAI released the setup and tests on GitHub under Apache 2.0 this afternoon. The second, a COBOL migration eval, asks a model to migrate COBOL to C++, Java, Kotlin, Python, or Rust with success defined as every hidden test passing with byte-for-byte parity. Its tests are hidden and not released. The headline figure, 97.9%, comes from the second one. We read the post and the repository. Here is what held up.
The Numbers, Every One of Them OpenAI's
| Evaluation (OpenAI's post) | Result | Compared against |
|---|---|---|
| MultiLangBench, average over 20 languages | GPT-6 Astra 87.4% | GPT-5.6 Sol 26.8% |
| MultiLangBench, COBOL | 10.9% to 72.1% | GPT-5.6 Sol to GPT-6 Astra |
| MultiLangBench, OCaml | 25.6% to 85.8% | same |
| MultiLangBench, Fortran | 21.4% to 83.5% | same |
| MultiLangBench, Visual Basic | 27.5% to 89.4% | same |
| MultiLangBench, languages above 83% | Astra in 19 of 20 | n/a |
| MultiLangBench, GPT-6 Luna | 40.3% average at about $0.77 per sample | GPT-5.6 Sol at about $3.48 per sample (about 78% more) |
| COBOL migration eval, low effort | GPT-6.1 Sol 86.6% at 61% of the cost | GPT-5.6 Sol 5.4% |
| COBOL migration eval, higher effort | GPT-6.1 Sol 97.9% | same |
Three honest observations:
- OpenAI graded OpenAI. Every column in that table is an OpenAI model against an older OpenAI model, run by OpenAI. There is no Claude, Gemini, Grok, or open-model baseline on either evaluation in the post. MultiLangBench is public, so anyone can produce one now; as of today nobody independent has.
- The headline number is on the private test. MultiLangBench is on GitHub. The COBOL migration eval that produces 86.6% and 97.9% is not; OpenAI says its tests are hidden. A benchmark you cannot run is a claim, and this is OpenAI's claim.
- The cost figures are OpenAI's estimates for its own runs. $0.77 and $3.48 per sample, and "61% of the cost," describe OpenAI's eval configuration, not your migration project. Effort settings and token accounting behind them are not detailed.
Two things OpenAI says that the headlines dropped: "applying it to production software still requires expert review," in the post's own words; and the banking statistics about how much of the world runs on COBOL are external citations OpenAI links to, not measurements OpenAI made.
What Is Actually Public
The repository openai/MultiLangBench went up on GitHub at 3:58 PM CT today under Apache 2.0, with the setup and the tests for the 20-language SQLite task. That is a real contribution: a reproducible benchmark on languages that most leaderboards ignore. The right response to it is to run it on the models you actually buy, including the non-OpenAI ones, and publish the numbers OpenAI did not.
Verified vs Unconfirmed: The Scorecard
| Claim | Verdict |
|---|---|
| Post dated Oct 19, 2026; no new model; evaluations on Astra, Sol, Luna | Verified (OpenAI post) |
| MultiLangBench 87.4% (Astra) vs 26.8% (GPT-5.6 Sol); per-language figures | Vendor (OpenAI's runs) |
| COBOL migration 86.6% low effort, 97.9% higher effort, vs 5.4% | Vendor, on hidden tests |
| MultiLangBench released on GitHub, Apache 2.0, Oct 19 | Verified (GitHub API: created 20:58 UTC) |
| COBOL migration eval released | Contradicted (tests hidden, not released) |
| Any rival-lab number on either eval | Unverified (none in the post; none independent yet) |
| Per-sample costs and "61% of the cost" | Vendor estimates for OpenAI's eval runs |
| "Still requires expert review" | Verified (OpenAI's own wording) |
Who Should Use It
Use the public benchmark. If you maintain code in any of the 20 languages, clone MultiLangBench and run it on your candidate models, OpenAI's and otherwise; the Apache-2.0 release is the useful part of this announcement.
Do not put 97.9% on a slide without the words "OpenAI's private COBOL migration eval, against GPT-5.6 Sol only, expert review still required." Modernization pitches that drop those words are quoting a number nobody outside OpenAI can reproduce.
The frontier is elsewhere: on the coding boards that are public and independent, the ranking between OpenAI's and Anthropic's models this week is a harness-by-harness question, not a one-number answer. Our model comparison has the current standings side by side.
Sources
*Not sure whether a frontier model can carry your legacy migration, or which one? Take the free 2-minute quiz and get matched. Then read about the model behind the 97.9%, and its own system card's fine print: GPT-6.1 Sol.*