OpenAI reports big gains on COBOL migration
OpenAI says its latest models raised COBOL migration accuracy from 5.4% to 97.9% and released a 20-language coding benchmark, targeting legacy code that still runs banking and other critical infrastructure.
OpenAI has published results from two evaluations it built to measure coding in low-resource languages such as COBOL, Fortran and Visual Basic. These languages have far less public code and documentation to train on than Python or JavaScript, but they still run much of banking, weather forecasting and nuclear safety software.
MultiLangBench asks a model to build a SQLite-compatible database engine from scratch in a given language. Each program is checked against tests adapted from SQLite's own suite, using database files as a shared interface so the same tests apply in every language. At maximum reasoning effort, GPT‑6 Astra averaged an 87.4% test pass rate across 20 languages, up from 26.8% for GPT‑5.6 Sol. Astra scored above 83% in 19 of the 20 languages. COBOL was the hardest at 72.1%, up from 10.9%. The cheaper GPT‑6 Luna averaged 40.3% at an estimated $0.77 per sample, compared with $3.48 for GPT‑5.6 Sol. OpenAI is releasing the benchmark setup, test suite and per-language results on GitHub.
The COBOL migration evaluation has models port programs to C++, Java, Kotlin, Python or Rust. A migration only counts if it passes hidden tests requiring byte-for-byte matching output, which catches subtle errors such as rounding where COBOL truncates. GPT‑6.1 Sol scored 86.6% at low reasoning effort, compared with 5.4% for GPT‑5.6 Sol, at 61% of the API cost. At higher effort it reached 97.9%, which OpenAI says nearly saturates the test.
OpenAI designed and scored both evaluations itself, and the models involved are already available. The company also notes that production migrations still need expert review.