TrialsgrambetaRCTs are beautiful

Agweyu LLM

2026

Generative AI-Enabled Clinical Decision Support System in Primary Care: A Pragmatic, Cluster-Randomized Trial

In 9,691 patients at primary care facilities in Kenya, an LLM decision support system in the medical record did not reduce treatment failure within 14 days compared with the electronic medical record alone.

Result

Efficacy: Expert-adjudicated composite of treatment-failure events within 14 days of enrolmentaOR 0.77 (95% CI 0.55-1.08) · P = 0.13

Below 1 favours an LLM decision support system in the medical record, above 1 favours the electronic medical record alone.

Event rates
Absolute difference
0.2 points

The absolute difference is derived from the event rates.

Also reported: No serious adverse events judged related to the intervention and no safety signal on independent review.

Original publication
Subtopic
Standalone consultation
Design
Pragmatic cluster RCT, clinical officers randomised, 16 primary care facilities, Kenya
Relevance
Practice-defining: a trial the field needs to know.
Source
Agweyu A, et al. Nat Med. 2026;32(8):3032-3039.doi 10.1038/s41591-026-04503-6

Discussion & critique

The first large pragmatic randomised trial of an LLM in real-world, low-resource primary care, and a null result on the patient-level primary outcome, with any benefit 'probably modest' in the authors' words. Reassuring safety, but a 2% event rate leaves little power, follow-up was 14 days, and the randomised units were clinicians rather than patients. A caution against extrapolating vignette-based LLM gains (Goh GPT-4 (management reasoning)) to patient outcomes.