TrialsgrambetaRCTs are beautiful

Goh LLM (diagnostic reasoning)

2024

Large Language Model Influence on Diagnostic Reasoning: A Randomized Clinical Trial

In 50 physicians working up clinical vignettes, access to a large language model alongside conventional resources did not improve diagnostic reasoning scores compared with conventional resources alone.

Result

Diagnostic reasoning score per case on a standardised rubric (differential accuracy, supporting/opposing factors, next diagnostic steps), graded by blinded expert consensus
Absolute difference
2 points
Effect estimate

adjusted difference295% CI -4 to 8 · P = 0.60

The absolute difference is derived from the event rates.

Also reported: Time per case 519 vs 565 s (P=0.20). The LLM alone scored 16 points higher than the conventional-resources group (95% CI 2–30; P=0.03).

Original publication
Subtopic
Physician decision-making
Design
Single-blind RCT, physicians randomised, multicentre
Relevance
Important: one of several pillars.
Source
Goh E, et al. JAMA Netw Open. 2024;7(10):e2440969.doi 10.1001/jamanetworkopen.2024.40969

Discussion & critique

The first randomised test of whether handing physicians an LLM improves their reasoning: it did not, yet the LLM alone outperformed both physician groups, exposing a human–AI collaboration gap rather than a model deficit. Small (50 physicians), vignette-based, one month of recruitment and no patient outcomes. Its companion trial on management reasoning (Goh GPT-4 (management reasoning)) found a benefit.