Guidelines for Annotating Social Support and Social Isolation in Clinical Notes | HackerNoon

Table of Links

Abstract and 1. Introduction

2 Data

2.1 Data Sources

2.2 SS and SI Categories

3 Methods

3.1 Lexicon Creation and Expansion

3.2 Annotations

3.3 System Description

4 Results

4.1 Demographics and 4.2 System Performance

5 Discussion

5.1 Limitations

6 Conclusion, Reproducibility, Funding, Acknowledgments, Author Contributions, and References

SUPPLEMENTARY

Guidelines for Annotating Social Support and Social Isolation in Clinical Notes

Other Supervised Models

References

GUIDELINES FOR ANNOTATING SOCIAL SUPPORT AND SOCIAL ISOLATION IN CLINICAL NOTES

Social isolation (SI) is a state of little or no contact or support between an individual and their community. Social support (SS) describes the physical and emotional comfort derived from one’s family, friends, colleagues, and community. Although electronic health records (EHRs) capture SI and SS within the narrative text, manual abstraction of such data is time-consuming and labor-intensive.

Research objective

To develop a hybrid approach using rule-based and machine learning (ML)-based natural language processing (NLP) for detecting patient-reported SS and SI in EHRs.

Annotation task overview

The purpose of annotating clinical notes is to highlight key terms and phrases that characterize one’s social standing and, based on their context, make a clinical determination of an individual’s present or past status with respect to SI and SS.

Annotation process

Guideline development phase: Initially, two annotators will annotate fifty (50) notes. Then, inter-annotator agreement (IAA) will be calculated, and the annotation guidelines will be revised. This process will continue until a sufficiently high agreement (IAA ≥ 0.6) is achieved. The mismatched cases will be adjudicated by iterative discussion until a consensus is reached. The first 50 clinical notes will be added to the gold standard but excluded from the final IAA computation.

Annotation Sites: Weill Cornell Medicine (WCM) and Mount Sinai Health System (MSHS).

Annotation Phase: Each document will be independently annotated by two annotators. After the annotations are completed, the conflicts will be resolved by a third annotator. The annotation guideline will be updated if needed. Finally, the IAA will be calculated on all documents excluding the first 50 documents that are used to improve the annotation and guideline.

Notes: MSHS will annotate psychiatry notes only as they don’t have access to primary care notes right now. In the annotation cohort, a single patient will have only one note. Entity attribute “negation” will be used only for “social isolation loneliness” category.

General guidelines for annotation

Annotators will annotate all mentions in a clinical note that indicate the presence (or absence) of present/past SI and SS. There may be multiple such mentions contained within a single document. The goal is to make an aggregate determination based on the various annotations within the document. The overall annotation schema is given in Table S1. The annotators will annotate all terms and phrases mentioned in Table S3 that indicate an individual’s social status based on the annotation schema. A single document may contain multiple entities from both SI and SS. Further, the annotators will annotate at the document/note level based on entities and their context available in that specific note. There are two tags for annotating at the document level and these are mentioned in Table S1.

Instructions for annotation

Annotations will be performed using a BRAT annotation tool made available at individual sites. This is a browser-based tool that users may access online in a secure environment. Each user will be provided a username and password to access the tool. Before starting the actual annotation process, each user will participate in an orientation session to familiarize themselves with the BRAT tool. Within the tool, annotators will be granted access to 300 [MS] and 225 [WCM] clinical notes provided as .txt files. After opening a .txt file, annotators will highlight spans of pertinent text and assign tags based on the entities in Table 1. In some cases, the highlighted terms may be used in an unrelated context (e.g., templated information: social isolation []) and can be tagged with ‘no context.’

We plan to annotate social isolation and social support at a fine-grained entity level. There are five (5) categories for social isolation: “loneliness,” “no social network,” “no emotional support,” “no instrumental support,” and “social isolation general.” Similarly, social support has four (4) categories: “social network,” “emotional support,” “instrumental support,” and “general.” Each term can be annotated with temporal information (present or past) and with negation (if a negation term precedes or follows the highlighted key terms [only for loneliness]). There are two (2) categories for document/note level annotation and these are “social isolation” and “social support” with “yes” and “no”. The document/note level annotations are provided based on the entity level annotations and the rules are provided in Table S2. Table S3 shows the lists of key terms and phrases that we have used for the development of NLP algorithm. We also identified some key terms and phrases that were excluded while identifying the social support or isolation from clinical encounter notes (See Table S4).

Table S3: List of typical examples of terms and phrases in the clinical encounter notes that would indicate a patient’s social support and isolation status.

Table S4: List of typical examples of exclusion terms and phrases in the clinical notes.

File naming convention

Documents are named in the following naming convention:

<Index Numer><person_id><note_id>_<note_date>.txt

E.g.: 1 101 1001 10001 100001 2000-01-01.txt

References

1. Lauren A. Lepow, Braja Gopal Patra, et al. 2021. Extracting Social Isolation Information From Psychiatric Notes in the Electronic Health Records. In AMIA 2021 Annual Symposium, San Diego, CA, USA.