🔍 Read the full analysis: How Microsoft’s AI Was Evaluated Against Doctors on ThorstenMeyerAI.com
Get business pricing on monitors, keyboards and dev gear
- Business-only prices and quantity discounts
- Tax-exempt purchasing
- Multiple users, one account, clear invoices
TL;DR
Microsoft’s MAI-DxO diagnostic system scored 83.2% on 304 difficult historical case records, compared with 36.1% for generalist physicians using reference materials. The peer-reviewed study tested a simulated, text-based benchmark—not diagnosis of real patients—and did not compare the system with specialists or clinical teams.
Microsoft’s diagnostic AI system MAI-DxO scored 83.2% on 304 difficult medical case records, compared with 36.1% for generalist physicians using textbooks and the internet, according to a study published October 22 in the peer-reviewed journal Science. The results concern historical, text-only cases in a simulator—not care delivered to real patients—and the comparison did not include specialists or physician teams.
The paper, titled Sequential diagnosis with language models, also reports results from 325 emergency department cases: MAI-DxO scored 80.3%, while physicians scored 46.8%. Microsoft described the system as more accurate than physicians in its accompanying account of the research. These figures come from Microsoft’s study; publication in a peer-reviewed journal means the work underwent review, but does not by itself show that independent researchers have reproduced the results.
The benchmark, called the Sequential Diagnosis Benchmark, presents a short case summary first. The system or physician must request additional information, such as answers to questions or test results, from a simulated gatekeeper holding the case file. The evaluation counts both the final diagnosis and the estimated cost of obtaining information. MAI-DxO uses a language model configured as a panel of virtual physicians that propose diagnoses and select further questions or tests.
The strongest score was not unique to the full orchestration system. Microsoft’s chart gives GPT-5 alone a 78.9% score on the complex cases, compared with 83.2% for MAI-DxO. The source report says the orchestration layer added about four percentage points and reduced estimated testing costs; the reported GPT-5 estimate was $8,409 in tests per case. Those cost figures are modeled estimates, not observed hospital spending.
Infographic 01 · The scoreboard
83% vs. 36%: the number, and who it was measured against.
Microsoft’s diagnostic system beat individual generalist physicians by a wide margin on a simulated, step-by-step test. The comparison group decides how much that says.
Hard cases (NEJM)
Diagnostic accuracy, 304 case records
Physician averages rest on 662 (open book) and 764 (closed book) case attempts, per Microsoft’s chart data.
Who was compared
The fine print behind “beats doctors”
- Generalists only: primary care doctors and hospital generalists. No specialists, no teams.
- Working alone: no colleagues; in the preprint also no search engines or language models.
- Text-only, historical cases: no real patients, no physical exam, no imaging.
- Hard by design: rare teaching cases; no healthy patients, so false alarms were not measured.
- Self-reported: all 15 authors are listed with Microsoft affiliations; peer-reviewed, not independently replicated.
Sources: Science 394, 423–428 (22 Oct 2026); Microsoft AI, “Building Medical Superintelligence” (22 Oct 2026) and its charts; physician cohort details from the arXiv 2506.22405 preprint (2025).
Powered by Thorsten Meyer AI
What the Scores Mean for Care
The results suggest that a language model can perform strongly on a structured task requiring it to gather information and revise a diagnosis, rather than simply choose an answer from a fully supplied case. That could inform future clinical decision-support research, particularly on how software selects tests and weighs their cost against their value.
But the reported accuracy figures do not establish that MAI-DxO is ready to diagnose patients or improve outcomes. The benchmark used historical written cases and simulated access to case details. It did not test bedside examination, real-world records, communication with patients, or the consequences of acting on a diagnosis. Hospitals, insurers and patients should treat the results as evidence about performance on this evaluation, not as a measure of clinical effectiveness.
AI diagnostic tools for healthcare
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
How the Doctor Comparison Was Set Up
The comparison group consisted of primary care doctors and hospital generalists working individually, rather than specialist teams. The source account says the cases were difficult examples drawn from the New England Journal of Medicine, a set likely to emphasize diagnostic challenges rather than the full range of routine presentations. In the Science results, physicians scored 22.3% without references and 36.1% with textbooks and internet access on the complex cases.
The study builds on an earlier, non-peer-reviewed version posted on arXiv in June 2025. That preprint reported a more dramatic comparison, with an earlier version of the AI scoring 80% against 20% for doctors. The later Science paper reports different figures and a broader comparison. The source account also notes methodological details from the preprint, including model-based grading and estimated test prices; it cautions that details may differ in the final paper.
According to that account, the preprint’s physician participants were 21 generalists from the United States and United Kingdom. The doctors were evaluated on a subset of cases, while the AI was scored across all 304. The source describes the test set as containing no healthy patients or harmless explanations, limiting what it can say about false alarms in ordinary practice.
“I really believe that in 2027, we will be on the verge of a coding style breakthrough for medical records.”
— Microsoft AI chief executive Mustafa Suleyman, speaking to TechCrunch
As an affiliate, we earn on qualifying purchases.
Limits Beyond the Benchmark
The results do not establish how MAI-DxO performs with live patients, incomplete or messy clinical records, physical examinations, or diagnoses outside the selected case sets. The comparison also leaves open how the system would fare against specialists, multidisciplinary teams, or doctors using their usual clinical resources and workflows.
Some elements of the benchmark rely on estimates or model-generated judgments. The source account says a judge model scored diagnoses, missing test results could be generated synthetically, and costs were mapped to billing codes and a 2023 U.S. hospital price list. The authors described those prices as first-order approximations. The available source material does not provide independent replication or patient-outcome evidence, and it is not clear whether all those implementation details are unchanged in the final paper.
As an affiliate, we earn on qualifying purchases.
Planned Study With Patients
The next reported step is a prospective study at Beth Israel Deaconess Medical Center in Boston. According to the source account, consenting patients will interact with the system alongside their usual care. That setting could provide evidence about how the tool behaves in a clinical workflow, though the study’s design, timeline, measures and results have not been specified in the material provided.
Until those findings are available, the Science paper should be read as a benchmark evaluation of diagnostic reasoning on selected cases. Whether MAI-DxO can safely assist clinicians, improve patient outcomes or reduce costs remains to be tested.
As an affiliate, we earn on qualifying purchases.
Key Questions
What is MAI-DxO?
MAI-DxO is Microsoft’s diagnostic AI system. In the study, it used a language model in a panel-like setup to propose diagnoses and request further case details, including tests.
Did Microsoft test the system on real patients?
No. The reported evaluation used historical, text-based medical cases in a simulator. A separate prospective study involving consenting patients at Beth Israel Deaconess Medical Center is planned, according to the source account.
How did it compare with doctors?
On 304 complex cases, MAI-DxO scored 83.2%, compared with 36.1% for generalist physicians using textbooks and the internet and 22.3% without those resources. On 325 emergency department cases, the scores were 80.3% and 46.8%, respectively.
Does the study show the system is better than specialists?
No. The comparison was with generalist physicians working individually; the study did not include specialists or physician teams. The results therefore do not establish how MAI-DxO compares with those groups.
Has the system been shown to improve patient outcomes?
Not in the evidence described here. The paper reports performance on a benchmark, while effects on patient outcomes, clinical safety and real-world costs remain unestablished.
Source: ThorstenMeyerAI.com
Halloween Picks
halloween
As an affiliate, we earn on qualifying purchases.
