This was a live alert trial, not a mock-up
Researchers ran a pragmatic, parallel-arm randomized trial in a large urban academic health system. The participants being randomized were clinicians: 150 physicians and nurse practitioners who had signed an eligible antipsychotic prescription for an outpatient with dementia between January 1, 2019, and April 30, 2021. When a clinician assigned to the intervention initiated another eligible prescription, the EHR displayed a clinical decision-support alert. Control clinicians did not receive it.
The alert stated that antipsychotic medicines increase mortality risk, linked to an after-visit handout about non-drug approaches, and defaulted the order to a lower dose and 30 pill-days if the clinician continued. The study therefore tested an intervention inside an operating EHR and a real prescribing moment. It did not test a consultant-pharmacist recommendation queue, a nursing-facility order process, or a general warning on every antipsychotic renewal.
The cancellation count and the primary result answer different questions
The alert appeared in 58 intervention encounters, and clinicians canceled 12 of those orders—21%. That is a clear interaction result: it tells us what happened immediately after the alert appeared. It does not establish why each order was canceled, what happened next, or whether the trial changed prescribing across the full follow-up period.
The prespecified primary outcome was mean total antipsychotic pill-days per provider over 19 months, adjusted for provider characteristics including prior prescribing. Raw mean pill-days were 126 in the intervention group and 225 in the control group. The adjusted difference was -113 pill-days, but the 95% confidence interval ran from -256 to +30 and the p value was .12. The trial therefore did not find a statistically significant reduction in its primary outcome. A result can look directionally encouraging and still remain compatible with no reduction under the study's analysis.
Question 1: Who was eligible, and who actually encountered the tool?
All 150 randomized clinicians had prescribed an eligible antipsychotic during the January 2019 through April 2021 eligibility window. During the study, however, only 28 intervention clinicians and 21 control clinicians initiated an eligible prescription. Those encounters enrolled 139 patients, whose mean age was 83; 67% were women. The alert fired 58 times across the 28 active intervention clinicians.
That funnel is the first thing to request after an alert demo: population eligible for the rule, users assigned to receive it, users who reached the trigger, alert exposures, unique people affected, and actions recorded. Without it, a cancellation percentage can float free of the opportunity count. A rule may perform exactly as designed and still reach too few relevant decisions to change the intended outcome.
Question 2: Which outcome was chosen before anyone saw the result?
The researchers also explored whether baseline prescribing behavior changed the effect. Their post hoc analysis suggested a reduction among clinicians with average baseline prescribing, and the authors described targeted future alerts as a question worth testing. That finding is exploratory. It can guide another confirmatory study; it should not replace the nonsignificant prespecified primary result in a product claim.
For software evidence, ask for the protocol or analysis plan, the named primary outcome, follow-up period, comparison group, effect estimate, confidence interval, and missing-data approach. Then separate prespecified secondary outcomes from exploratory subgroup findings. A vendor may reasonably report engagement measures, but clicks, dismissals, and order cancellations should remain labeled as intermediate behavior unless the evidence connects them to the promised operational or clinical outcome.
Question 3: What work did the alert add, move, or leave behind?
This alert targeted initiation: an eligible patient had no antipsychotic prescription recorded in the EHR during the prior 12 months. The trial does not tell a small consultant pharmacy practice how the same design would behave for renewals, gradual dose-reduction follow-up, facility documentation, after-hours orders, or recommendations that still need an authorized response.
A useful demonstration should follow both branches. When a user continues, where are the rationale, dose, duration, monitoring plan, and later reassessment recorded? When a user cancels, is there a replacement plan, communication task, or unresolved status, and who owns it? Also count the interruption: how often the alert appears, which roles see it, how long it takes, whether the same issue repeats, and how exceptions are reviewed. The goal is not to turn one trial into a universal design rule. It is to keep the screen event connected to the work that follows.
Carry the trial's limits into the next demo
This was one academic outpatient system with a smaller effective clinician sample than planned. Prescribing initiation fell during the study, and the authors noted that pretrial email and other priming may have influenced behavior before an alert appeared. The study covered new orders, not the broader antipsychotic lifecycle, and it did not evaluate a U.S. nursing-facility or consultant-pharmacist workflow. It does not prove that an alert improves resident outcomes.
For the next software review, ask the supplier to walk one synthetic case from eligibility through trigger, exposure, decision, downstream task, and follow-up. Request the denominator funnel and the prespecified outcome from any evaluation they cite. An alert can be thoughtfully designed, visible in practice, and associated with real cancellations while still missing the outcome the trial was built to test. That is not a reason to ignore the tool. It is a reason to ask a better second question after the demo looks convincing.