Tacit knowledge contributes to better hospital patient outcomes
As a new nurse in critical care, Sarah Rossetti had noticed a pattern in the notes made by senior nurses regarding the condition of the patients. “Nurses normally visit patients on a regular schedule, but if they see a change for the worse, their visits become more frequent and of longer duration.” said Rossetti who is now an Associate Professor of Biomedical Informatics and Nursing at Columbia University. “Other variations in nurses’ behavior also are indicative of patient condition.”
Interested in finding out whether increased surveillance as represented in electronic health records (EHRs) was associated with patient outcomes, Rossetti formed a team at Columbia University in 2009, which included Kenrick Cato and Dave Albers both also at the school at the time, that received funding from the American Association of Critical Care Nurses (AACN) and conducted several small analyses. At this point, Rossetti was a post-doctoral student in biomedical informatics. Under the AACN grant she and the team explored statistical signals from metadata in health records that indicated an increasing level of concern on the part of nurses.
Another grant was obtained from the National Institutes of Health’s National Institute of Nursing Research (NIH NINR) in 2017, with Rossetti and Cato as multi-principal investigators, to develop and test the COmmunicating Narrative Concerns Entered by RNs (CONCERN) Early Warning System (EWS) based on these statistical signals. They worked extensively with clinician stakeholders, including nurses and physicians, to develop output that could be used in real time to prioritize patients who were at risk, and prompt an intervention by the clinical team.
The machine learning model and associated numerical values were translated into a color-coded signal with red, yellow, and green indicators. “The green ratings were for those considered to be at low risk,” commented Rossetti, “and the red rating was for known at-risk patients. The category of most interest was the yellow group, since their condition was less defined but had the potential to worsen.”
Using an ensemble modeling approach, eventually about 1200 models were developed that encompassed a wide range of different measures, including how many days the individual had been hospitalized, whether they were in acute or critical units of the hospital, and other factors. The CONCERN EWS system picks out the model that best matches each patient. “The models’ metrics reflected what would be normal for a patient under those circumstances,” Rossetti continued. “If the nurses’ documentation patterns reflected a deviation from the expected norm, then the system would trigger an alert.”
As part of the NIH NINR grant, the team conducted a large-scale multi-site study as to how the CONCERN EWS affected patient outcomes. A group of about 60,000 patients in four hospitals across two major health systems were divided into two groups; one was a control group and the other group received the CONCERN EWS intervention, based on randomization of clinical units. The trial ran for 1 year at each study site.
Patients receiving the CONCERN EWS intervention showed a 35.6 percent reduction in mortality risk a moderately shorter hospital stay, and a 7.5 percent decrease in sepsis risk. The system detected deterioration up to 42 hours earlier than models based on physiological indicators.
Physiological indicators such as lab reports or other measures are delayed in comparison to nurses’ notes, which occur in real time as patients are observed. “In the context of nursing surveillance, many nurses’ observations are subtle,” continued Rossetti, “such as unusual pallor or a slower than expected return to normal blood pressure after turning a patient in the ICU, but the increased presence and uncommon timing of certain types of nursing documentation are an early alert that the nurse is worried and adapting surveillance as a result.”
The normal intervention at that time would be a meeting of the clinical team to determine what action should be taken. “It’s all about communication,” Rossetti summarized. “The CONCERN system does not tell the medical staff what to do or change the care procedures. It is an alert system that helps trigger early awareness, and when appropriate, escalation by the care team.”
The success of this AI system can be attributed to multiple factors. First, the underlying information in the models was based on analyses of documentation patterns from experienced nurses who were recognized experts. Therefore, it was trusted by the medical staff. In addition to domain experts, the interdisciplinary team included software engineers and data scientists who could develop complex computational models. They worked closely with stakeholders. “We focused also on user-centered design,” Rossetti added. “We wanted to provide information clinicians found useful in real time to help them prioritize their patients who were at risk.”