Start with the task the model actually performed
The open-access Drug Safety study, published September 3, evaluated 850 discharge summaries for patients aged 65 or older across six public hospitals in New South Wales, Australia. The 2022 records covered stays of at least 48 hours. All sites used the same electronic medical-record platform but included acute, regional, rehabilitation, and transitional-care settings.
The system did not generate deprescribing advice. A rule-based first stage extracted discharge medicines and sentences containing words such as “stop,” “cease,” or “wean,” retaining a sentence only when it linked to a listed medicine. A locally run Llama 3.3 model classified each candidate, and researchers mapped its five categories to a present-or-absent flag.
Finding an instruction that already exists is narrower than deciding whether a resident's regimen should change. The study did not test a nursing-facility MRR workflow, send a recommendation, reconcile an order, record an authorized response, or measure whether anyone acted on the flag.
Read the 144-case test beside the 850-summary headline
Researchers used 680 summaries to develop the rules and prompt, then reserved 170 summaries for testing. The final medication-level test contained 144 medications represented in 197 candidate sentences. On that test, the binary classifier produced an F1 score of 0.91 and accuracy of 0.90. Those are useful feasibility results, but they are not 850 independent tests of the final system.
Performance moved by site. Hospital-level F1 ranged from 0.67 to 0.97; the two lowest results came from hospitals with only 12 and nine test cases. Across the full candidate set, drug-class F1 ranged from 1.00 for constipation medicines to 0.73 for diabetes medicines. Human agreement was lowest for “dose reduction without aim to cease.” One dashboard score can hide variation by site, drug class, and definition.
The error path is the workflow test
Across development and test records, researchers counted 79 misclassifications: 66 false positives and 13 false negatives. In 23 cases, the system treated an action completed in hospital as a recommendation for after discharge. False negatives involved missing context, including language where the medicine and action were too far apart for the line-by-line method to connect.
The model's self-reported confidence was also moderately miscalibrated. The authors concluded that its confidence score could not support a simple rule that automatically accepts high-confidence classifications and sends only low-confidence ones to a person. They describe the current system as a screening tool for clinician review, not an autonomous extraction process. That is the most useful implementation finding in the paper.
Make a vendor or pilot prove one complete handoff
A small consultant pharmacy practice does not need to reproduce the research to ask a better demonstration question. Use synthetic or properly de-identified examples shaped like the records you actually receive, and include the exceptions that challenged the study:
- Put an inpatient medication discontinuation beside a separate post-discharge taper instruction. Require the system to show which is completed history and which remains pending.
- Separate the medication name from the action, and add an abbreviation or class term. Ask the system to display the exact text and medication link behind every flag.
- Include one explicit recommendation, one implicit recommendation, one ambiguous dose adjustment, and one record with no recommendation. Record false positives and false negatives separately.
- Require a human reviewer to confirm, correct, or reject the classification, then verify that the decision, source text, reviewer, time, and correction history survive export.
- Follow one accepted flag through assignment, communication, authorized response, medication reconciliation, and later review. Classification speed is not evidence that the care loop closed.
Local processing is an architecture choice, not a completed privacy review
The researchers ran the open-source model on Windows 11 with an RTX 6000 Ada GPU and 128 GB of memory, and present local deployment as cost-efficient and privacy compliant. The prompt is supplementary; custom code and data are available on reasonable request and subject to conditions. The paper reports no production integration, independent security assessment, U.S. privacy analysis, access-control test, incident process, or small-practice deployment.
Keeping data inside an organization's own environment may change the exposure path, but it does not by itself settle authorization, minimum-necessary use, logging, model updates, retention, backups, or who can inspect the source text and output. Ask for the actual data flow and controls rather than accepting “local” as the whole answer.
The authors call for validation across countries, hospitals, and specialties and for research on medication-related harm, adherence, and continuity. Until then, the defensible promise is modest: software may help find a sentence worth checking. The handoff still succeeds only when the medication, status, owner, response, and follow-up remain connected.