Goh GPT-4 (management reasoning)
GPT-4 Assistance for Improvement of Physician Performance on Patient Care Tasks: A Randomized Controlled Trial
Patient / Population Intervention / Exposure Comparison Outcome
In 92 practising physicians randomised to answer five clinical vignettes in a simulated setting, GPT-4 plus conventional resources improved management reasoning scores compared with conventional resources alone.
N
92
Design
RCT, physicians randomised, simulated vignettes
Endpoint
Difference in total score between groups on expert-developed management reasoning rubrics
Relevance
1
ResultTotal score mean difference +6.5% with GPT-4 (95% CI 2.7–10.2; P<0.001); LLM users spent 119.3 s longer per case (95% CI 17.4–221.2; P=0.02). LLM-augmented physicians vs LLM alone −0.9% (95% CI −9.0 to 7.2; P=0.8).
Goh E, et al. Nat Med. 2025;31(4):1233-1238. 10.1038/s41591-024-03456-y
Discussion & critique
The positive counterpart to the diagnostic-reasoning trial (Goh LLM (diagnostic reasoning)): on open-ended management tasks physicians with GPT-4 scored higher, and no better than the LLM alone. Vignettes in a simulated setting graded by rubric, with no patient outcomes — the authors themselves call for validation in real clinical practice.