The demographic characteristics of patients within the annotated cohort are detailed in Table 2. Notably, the patient composition at MSHS was younger and more diverse as compared to patients at WCM.
Analysis was conducted using the gold-standard, manually annotated data. The macro-averaged precision, recall, and f-scores for classifying fine- and coarse-grained SS and SI categories at note level are provided in Table 3.
At MSHS, the RBS outperformed the LLM-based system for both fine- and coarse-grained classification. For the fine-grained categories, the RBS achieved macro-averaged f-score of 0.90 compared to 0.62 for the LLM. For coarse-grained classification, the RBS had macro-averaged f-score of 0.89 versus 0.65 for the LLM.
At WCM, the RBS outperformed the LLM for fine-grained classification with macro-averaged f-scores of 0.82 versus 0.81, respectively. The coarse-grained categories were similar, with a macro-averaged f-score of 0.85 for the RBS compared to 0.82 for FLAN. The performance of the zero-shot FLAN-T5-XL model is provided in Supplementary Table S8.
Comparison to ICD codes: There were zero visits associated with the annotated clinical notes where SI was captured by the structured ICD codes (ICD-10: ‘Z60.2’, ‘V60.3’, ‘Z60.4’, ‘Z60.9’; ICD-9: ‘V60.3’, ‘V62.4’ [see Supplementary Table S9]). Without the NLP pipelines, the presence of SI would otherwise be missed both sites.
Authors:
(1) Braja Gopal Patra, Weill Cornell Medicine, New York, NY, USA and co-first authors;
(2) Lauren A. Lepow, Icahn School of Medicine at Mount Sinai, New York, NY, USA and co-first authors;
(3) Praneet Kasi Reddy Jagadeesh Kumar. Weill Cornell Medicine, New York, NY, USA;
(4) Veer Vekaria, Weill Cornell Medicine, New York, NY, USA;
(5) Mohit Manoj Sharma, Weill Cornell Medicine, New York, NY, USA;
(6) Prakash Adekkanattu, Weill Cornell Medicine, New York, NY, USA;
(7) Brian Fennessy, Icahn School of Medicine at Mount Sinai, New York, NY, USA;
(8) Gavin Hynes, Icahn School of Medicine at Mount Sinai, New York, NY, USA;
(9) Isotta Landi, Icahn School of Medicine at Mount Sinai, New York, NY, USA;
(10) Jorge A. Sanchez-Ruiz, Mayo Clinic, Rochester, MN, USA;
(11) Euijung Ryu, Mayo Clinic, Rochester, MN, USA;
(12) Joanna M. Biernacka, Mayo Clinic, Rochester, MN, USA;
(13) Girish N. Nadkarni, Icahn School of Medicine at Mount Sinai, New York, NY, USA;
(14) Ardesheer Talati, Columbia University Vagelos College of Physicians and Surgeons, New York, NY, USA and New York State Psychiatric Institute, New York, NY, USA;
(15) Myrna Weissman, Columbia University Vagelos College of Physicians and Surgeons, New York, NY, USA and New York State Psychiatric Institute, New York, NY, USA;
(16) Mark Olfson, Columbia University Vagelos College of Physicians and Surgeons, New York, NY, USA, New York State Psychiatric Institute, New York, NY, USA, and Columbia University Irving Medical Center, New York, NY, USA;
(17) J. John Mann, Columbia University Irving Medical Center, New York, NY, USA;
(18) Alexander W. Charney, Icahn School of Medicine at Mount Sinai, New York, NY, USA;
(19) Jyotishman Pathak, Weill Cornell Medicine, New York, NY, USA.
