TrialsgrambetaRCTs are beautiful

Goh GPT-4 (management reasoning)

2025

GPT-4 Assistance for Improvement of Physician Performance on Patient Care Tasks: A Randomized Controlled Trial

In 92 physicians answering clinical vignettes, GPT-4 plus conventional resources improved management reasoning scores compared with conventional resources alone.

Result

Primary endpoint: Difference in total score between groups on expert-developed management reasoning rubrics

Total score mean difference +6.5% with GPT-4 (95% CI 2.7–10.2; P<0.001); LLM users spent 119.3 s longer per case (95% CI 17.4–221.2; P=0.02). LLM-augmented physicians vs LLM alone −0.9% (95% CI −9.0 to 7.2; P=0.8).

Original publication
Subtopic
Physician decision-making
Design
RCT, physicians randomised, simulated vignettes
Relevance
Practice-defining: a trial the field needs to know.
Source
Goh E, et al. Nat Med. 2025;31(4):1233-1238.doi 10.1038/s41591-024-03456-y

Discussion & critique

The positive counterpart to the diagnostic-reasoning trial (Goh LLM (diagnostic reasoning)): on open-ended management tasks physicians with GPT-4 scored higher, and no better than the LLM alone. Vignettes in a simulated setting graded by rubric, with no patient outcomes; the authors themselves call for validation in real clinical practice.