Physicians Score Higher With GPT-4o Access in Trial

Access to GPT-4o (OpenAI), a large language model (LLM), improved physicians’ performance on simulated primary care vignettes in Indonesia, Kenya, and the Netherlands, according to a parallel-group randomized controlled trial published in npj Digital Medicine.

Investigators enrolled 249 physicians: 81 in Indonesia, 60 in Kenya, and 108 in the Netherlands. Participants were randomly assigned to GPT-4o access or control and completed the same four clinical vignettes in English. The intervention group could use GPT-4o through an interface integrated into the study platform, while controls were instructed not to use internet searches, clinical guidelines, or other traditional resources.

The primary outcome was the average percent-correct vignette score, calculated from weighted, locally adapted rubrics. Two reviewers independently graded open-ended responses and were blinded to treatment assignment. Compared with controls, mean scores were 10.7 percentage points higher with GPT-4o access in Indonesia (95% CI, 5.7-15.7; P < 0.001), 18 percentage points higher in Kenya (95% CI, 12.7-23.2; P < 0.001), and 7.2 percentage points higher in the Netherlands (95% CI, 3.7-10.7; P < 0.001).

The authors cautioned that controls could not use resources ordinarily available to physicians, likely making the estimated effect of GPT-4o an upper bound relative to real-world practice. The study was conducted in controlled settings, evaluated four globally prevalent conditions with established guideline-based management, and did not assess harms. Randomization in Kenya was not stratified by physician type, creating an imbalance across study arms.

The investigators concluded that LLM access may improve physician performance across economically diverse settings but cautioned that the results cannot be assumed to apply to all medical conditions or routine clinical practice. They called for further research into potential undesirable effects and sources of cross-country variation.

Source: Rounding N, Arif LS, Berg J, et al. Impact of LLM assistance on physician decision-making: a multi-country randomized controlled trial. npj Digit Med. Published online September 9, 2026. doi:10.1038/s41746-026-03111-5