NEW YORK (Reuters Health) – A sepsis-prediction model used widely by hospitals and health systems in the United States performed poorly in predicting sepsis onset in an external validation study.
“Our evaluation shows that the Epic Sepsis Model mostly alerts on patients who are already going to get timely antibiotics even without an alert. If the purpose of a sepsis-prediction model is to bring patients to attention who would otherwise have been missed, this tool is really limited in its capability to do that,” Dr. Karandeep Singh of University of Michigan Medical School, in Ann Arbor, told Reuters Health by email.
“Our findings go against what is reported by the model developer because their evaluation relies on billing codes alone to define sepsis, which are known to be inaccurate, whereas we use a combination of criteria used by the Centers for Disease Control and Prevention and Medicare,” Dr. Singh added.
The Epic Sepsis Model, or ESM, was developed and validated by Epic Systems Corporation based on data from 405,000 patient encounters across three health systems from 2013 to 2015.
Dr. Singh and colleagues did a retrospective study of 27,697 patients who had more than 38,000 hospitalizations at the University of Michigan Ann Arbor between December 2018 and October 2019.
“The ESM identified 183 of 2,552 patients with sepsis (7%) who did not receive timely administration of antibiotics, highlighting the low sensitivity of the ESM in comparison with contemporary clinical practice,” Dr. Singh and colleagues report in JAMA Internal Medicine.
“The ESM also did not identify 1,709 patients with sepsis (67%) despite generating alerts for an ESM score of 6 or higher for 6,971 of all 38,455 hospitalized patients (18%), thus creating a large burden of alert fatigue,” they say.
The ESM had an area under the receiver operating characteristic curve of 0.63 (95% confidence interval, 0.62 to 0.64), which is “substantially worse” than the AUC of 0.76 to 0.83 reported by its developer, the researchers note.
They say it’s important to note that this external validation was done at a single academic medical center, although the cohort was large and relatively diverse.
Another limitation, they say, is the use of a composite definition to account for the two most common reasons why healthcare organizations track sepsis, namely, surveillance and quality assessment, although sepsis definitions are still debated.
Despite these limitations, the authors say the “widespread adoption of the ESM despite its poor performance raises fundamental concerns about sepsis management on a national level.”
Reuters Health reached out to Epic Systems Corporation for their perspective on the findings.
In email, a company spokesperson said, “Identification of sepsis requires both art and science, and we welcome the discussion with researchers and clinicians.”
“Clinicians often will recognize many of the patients who are septic. The purpose of the model is to identify early the harder-to-recognize patients who otherwise might have been missed. It does this by indicating the probability of a likelihood of sepsis. The model has been shown to identify patients who are coming down with sepsis up to 4 hours earlier than clinicians identify them, and that can be lifesaving. In the example given in this paper, if the Epic model was used in real time it would likely have identified 183 patients who otherwise might have been missed,” the spokesperson said.
“Each health system needs to set thresholds to balance false negatives against false positives for each type of user. When set to reduce false positives, it may miss some patients who will become septic. If set to reduce false negatives, it will catch more septic patients, however it will require extra work from the health system because it will also catch some patients who are deteriorating but not becoming septic,” the spokesperson added.
Finally, the spokesperson said it’s important to note that this study “did not take into account the analysis and required tuning that needs to occur prior to real-world deployment of the model. In their hypothetical configuration, the authors picked a low threshold value that would be appropriate for a rapid response team that wants to cast a wide net to assess more patients. A higher threshold value reducing false positives would be appropriate for attending physicians and nurses.”
The study was supported by grants from the National Institutes of Health and the National Heart, Lung, and Blood Institute. The authors have no relevant disclosures.
SOURCE: https://bit.ly/35JsAhr JAMA Internal Medicine, online June 21, 2021.
Source: Read Full Article