Microsoft’s AI got 83% of medicine’s hardest cases right in a Science study, more than double the doctors’ score

Microsoft says its AI diagnostic system got the right answer on 83% of medicine’s hardest published cases, more than double the score of doctors given the same cases. The results, for a system called MAI-DxO, appeared on Thursday in a peer-reviewed paper in Science, and Microsoft presents it as a step towards “medical superintelligence”, the goal its AI chief Mustafa Suleyman has set for the company.

MAI-DxO, short for Microsoft AI Diagnostic Orchestrator, works like a virtual panel of doctors. Several AI agents each propose diagnoses, pick questions to ask or tests to order, and challenge the panel’s emerging view over several rounds of debate, weighing what each test is likely to reveal against what it costs.

How the AI and the doctors were tested

Microsoft built its own benchmark, SDBench, from 304 complex case reports published in The New England Journal of Medicine and a second set of more ordinary emergency department visits from MIMIC-IV-ED, a database of admissions at Beth Israel Deaconess Medical Center in Boston. Each case starts with a short description of the patient. The AI, or the doctor, then has to ask for each piece of information, from the patient’s history to a scan, and every test adds to an estimated bill.

The paper reports these results:

• Complex NEJM cases: MAI-DxO 83.2%; generalist physicians 36.1% with textbooks and the internet, 22.3% without them.
• Emergency department cases: MAI-DxO 80.3%; physicians 46.8%.
• Cost: on the NEJM cases its most accurate setting averaged $5,431 in tests per case, against $4,724 for the doctors using references. A setting capped at a $1,000 budget scored 79.6% at $1,303 per case.

The doctors were primary care physicians and general internists working alone. Cases this rare would normally go to specialists or teams of doctors, and none were compared. The authors also note the limits of the set-up in the paper’s summary: “Although these findings were obtained in a controlled benchmark setting, they highlight the promise of language models for supporting more accurate, lower-cost diagnoses.”

How much of it is GPT-5?

The headline system runs on OpenAI’s GPT-5, and much of the gain comes from that model. Microsoft’s own figures show GPT-5 alone, with no orchestration, scored 78.9% on the NEJM cases at $8,409 per case. MAI-DxO added about four points and cut the bill by about $3,000. Other models on their own trailed: Grok 4 scored 76.6%, Claude 4.1 Opus 74.0% and Gemini 2.5 Pro 73.0%.

The orchestrator helps most with weaker models and harder cases. TechCrunch reports that it lifted Gemini 2.5 Flash from 46.4% to nearly 69%, and in Microsoft’s later tests took the newer GPT-5.5 from 72% to nearly 87% on the NEJM cases. On the emergency cases GPT-5.5 scored about 85% with or without it.

Why the numbers changed since 2025

Microsoft first posted the work as a preprint in June 2025, when MAI-DxO ran on OpenAI’s o3 and scored up to 85.5% against 20% for 21 doctors from the US and UK. Those doctors weren’t allowed to look anything up. The published version runs on GPT-5, adds a group of doctors allowed textbooks and the internet, and adds the emergency department cases, which is why the comparison Microsoft now leads with is 83% against 36%.

‘It needs clinical validation’

MAI-DxO has only been tested on past cases, and Microsoft says the real test comes next. With Beth Israel Deaconess, it plans a study in which consenting patients get normal care and also talk to the system, whose recommendations will be compared with their doctors’ independent assessments. Dominic King, Microsoft AI’s vice president of health and a co-author of the paper, told The Verge it will likely “get underway in the next couple of months”. Suleyman told The Verge:

It needs clinical validation. That is a leap, but this is a major milestone along that journey.

Mustafa Suleyman, CEO, Microsoft AI, to The Verge

Suleyman also told TechCrunch that Microsoft is working with Mayo Clinic on a healthcare foundation model planned for next year, part of a push that started with Microsoft training its own frontier models. The company is open-sourcing SDBench for a selection of the NEJM cases so other researchers can test their own systems and check its results. Governments are moving too: Britain this month agreed to bring AI into NHS hospitals sooner.

Why it matters

A peer-reviewed paper in Science gives Microsoft’s claim more weight than a company benchmark, and it already reads like a pitch for clinical decision tools. But the doctors it beat were generalists working alone on cases built to be hard, and the hospital study is what will show whether any of this helps real patients beyond one-off stories of chatbots spotting rare conditions.

Sources: Nori et al., “Sequential diagnosis with language models”, Science (October 22, 2026); Microsoft AI, “Building Medical Superintelligence”; arXiv preprint (June 2025); The Verge and TechCrunch interviews.