The Language of Trauma: Modeling Traumatic Event Descriptions Across Domains with Explainable AI (2024)

2.1 Definition & Scope

Psychological trauma, as defined by the American Psychological Association (APA), encompasses experiences of "exposure to actual or threatened death, serious injury, or sexual violence," whether directly encountered or witnessed. This includes instances where individuals "learn that the traumatic event(s) occurred to a close family member or close friend" (American Psychiatric Association, 2013).

While psychological trauma and PTSD are frequently discussed in the context of childhood abuse and the military, trauma can manifest in a variety of situations (Vander Kolk, 2003; Yehuda, 1998). It can arise in interpersonal violence like domestic abuse and sexual assault; and accidents or natural disasters. Trauma can also result from medical issues, bereavement and loss, emotional and psychological abuse, and its manifestation can vary depending on cultural beliefs and values (Smelser etal., 2004).

2.2 Trauma Contexts & Categorization

Within the psychological literature, key events have been identified that are typical for specific trauma contexts.In armed conflict and mass atrocities, exposure to severe violence and death is prevalent. This often includes the death of close family members, forced displacement, and sexual abuse (Powell etal., 2003). For instance, Dyregrov etal. (2000) found that most child survivors of the Rwandan genocide had witnessed severe injuries and deaths, with more than half witnessing massacres.

In domestic trauma, the most common forms are physical abuse (e.g., intimate partner violence), emotional abuse, and neglect (McCloskey and Walker, 2000). Emotional abuse is particularly hard to detect due to its subtle nature, including consistent belittling, criticizing, or bullying (Dye, 2020; Idsoe etal., 2021).Sexual violence, whether in war or domestic contexts, is an especially devastating form of trauma (Kiser etal., 1991). This includes childhood sexual abuse, rape, and exploitation.

The range of traumatic events makes conceptualizations of trauma complex. Researchers have categorized trauma in line with diagnostic manuals like the Diagnostic and Statistical Manual of Mental Disorders (DSM) into types such as assaultive violence (e.g., military combat, rape, threats with weapons), other injuries or shocking events (e.g., serious car accidents and life-threatening illnesses) (Breslau etal., 2004).Identifying these events is crucial, as most subsequent issues are linked to the initial trauma due to the development of trauma-specific fears in PTSD (Terr, 2003).

2.3 NLP for Trauma Detection

Given the variety and subjective nature of traumatic experiences, detecting them in text is complex. Despite these challenges, recent research has shown that NLP methods can improve the detection of psychological disorders and aid in treatment adaptation Ahmed etal. (2022); DeChoudhury and De (2014); LeGlaz etal. (2021); Malgaroli etal. (2023); Zhang etal. (2022).

NLP and Mental Health. Major areas in this field include promoting better health and early disorder identification for intervention (Calvo etal., 2017; Swaminathan etal., 2023). For example, Levis etal. (2021) associated linguistic markers from psychotherapist notes with treatment duration. Analyzing mental health chat conversations, Hornstein etal. (2024) found that words indicating younger age and female gender were associated with a higher chance of re-contacting.

Recently, the use of Large Language Models (LLMs) has led to the development of specific models for mental health applications (Xu etal., 2024; Yang etal., 2024). While LLMs effectively detect mental health issues and provide eHealth services, their clinical use poses risks, such as the lack of expert-annotated multilingual datasets, interpretability challenges, and issues regarding data privacy and over-reliance (Guo etal., 2024).

Specifically for social media data, there has been research on using sentiment analysis and semantic structures to detect anxiety (Low etal., 2020) or depression (Tejaswini etal., 2024) on Reddit posts. In suicide prevention on social media, Sawhney etal. (2020) developed a superior model for suicidal risk screening that identifies emotional and temporal cues, outperforming competitive methods (c.f., Ji (2022) on suicidal risk detection).

Trauma Detection. In trauma research, progress is being made in analyzing patient narratives (He etal., 2017) and identifying cases of post-traumatic stress disorder (PTSD) through speech (Marmar etal., 2019). Miranda etal. (2024) developed an NLP workflow using a pre-trained transformer-based model to analyze clinical notes of PTSD patients, revealing consistent reductions in trauma criteria post-psychotherapy. Disruptions in lexical characteristics and emotional valence have been found to contribute to identifying PTSD (Quillivic etal., 2024).Using Twitter data, UlAlam and Kapadia (2020) investigated whether posts can complete clinical PTSD assessments, achieving promising accuracy in PTSD classification and intensity estimation validated with veteran Twitter users (cf. Coppersmith etal. (2014); Reece etal. (2017)).

2.4 Trauma Event Detection in this Study

Previous work has identified language markers of PTSD, such as overuse of first-person singular pronouns, increased use of words related to depression, anxiety, and death, and more negative emotions. However, these markers are not specific to trauma and can also be associated with other psychological disorders, complicating accurate identification. Additionally, the transferability of detection methods is often lacking (Coppersmith etal., 2014; Quillivic etal., 2024).

Trauma detection in NLP is distinct in that it involves identifying a specific traumatic event that precedes a PTSD diagnosis, unlike the detection of depression or anxiety, which do not require a concrete event in their definitions. This study focuses on detecting such events in online resources, avoiding symptom or diagnosis analysis. Drawing conclusions about mental health from public text data alone is impossible without additional psychological information. We aim to identify instances meeting the APA’s definition of trauma, minimizing subjectivity by closely following their criteria.

3.1 Data Sources

Our final dataset is built from four datasets, each offering unique perspectives on traumatic experiences (Table1) to identify common characteristics of trauma that extend beyond specific events, such as those related to war: The Genocide Court Transcripts (GTC; Schirmer etal., 2023a) dataset comprises text from genocide tribunals, providing insights into severe human rights violations and the profound trauma experienced by victims and witnesses. This encompasses 90 cases across the International Criminal Tribunal for Rwanda, the International Criminal Tribunal for the former Yugoslavia, and the Extraordinary Chambers in the Courts of Cambodia. The Reddit PTSD Dataset includes posts from the PTSD subreddit of the Reddit Mental Health Dataset Low etal. (2020), where individuals discuss their experiences with post-traumatic stress disorder, sharing personal stories and support. The Mental Health Counseling Conversations Dataset (Amod, 2024) features questions and answers sourced from online counseling and therapy platforms. The questions cover a wide range of mental health topics, and qualified psychologists provide the answers.

The Incel Posts Dataset (Matter etal., 2024) contains posts from Incel community forums and reflects extreme misogynistic viewpoints. This dataset serves as a control in our study: Though not explicitly trauma-related, it includes posts on depression, bullying, and violence directed towards women. The violent and aggressive language in this dataset helps quantify our models’ ability to distinguish explicit trauma from related emotional distress.

3.2 The Trauma Event Dataset TRACE

We present the final trauma event dataset TRACE (Trauma Event Recognition Across Contextual Environments).To that end, all source datasets were pre-processed to ensure comparability for the detection task, including the removal of URLs and standardization of formatting. Due to their varied origins, the samples from each dataset differ in size, with instances ranging from single-word sentences to more elaborate descriptions of events and personal thoughts across all datasets.For compatibility with the BERT-architecture, we split instances exceeding the 512-token limit into smaller segments. Our approach treats each segment as independent, with trauma classification based solely on its content. While some segments from the same text may appear in both training and test sets, we consider label leakage minimal, since the model must rely on the segment’s content for accurate prediction. 7-20% (depending on the dataset) of segments were split overall.

Our study aims to demonstrate cross-domain transferability on realistic data, making it crucial to use datasets with their expected class distribution, even if they differ in context and trauma event rates. We matched the size of all datasets to the Counseling Dataset, which had the fewest samples and the most significant class imbalance. Despite these constraints, the Counseling Dataset remains highly valuable for its unique perspective on online mental health conversations, particularly in seeking expert advice.

Annotation Process. The GTC already contains a binary trauma variable that psychologists have annotated according to the APA definition of trauma.For the PTSD and Counseling datasets, 1,200 instances each were annotated by crowdworkers. We used the Portable Text Annotation Tool (Potato; Pei etal., 2022) to set up an annotation interface for crowdworkers using Prolific as a recruitment platform for annotators. Each instance was labeled by three annotators, and all annotators received an hourly reimbursem*nt of approximately 12 US$. The crowdworkers were provided detailed instructions, the APA definition of a traumatic event, and three examples. Both the Prolific pre-screening and the instructions contained a trigger warning, ensuring that participants were free to pause or stop the study at any time (AppendixA, Figure6). Annotators were based in either the US or the UK and fulfilled English language requirements.

We conducted a pilot study comparing single-choice and span annotation setups, where participants highlighted traumatic events in the text. The final annotation task used the span setup to ensure accurate detection (AppendixA, Figure7). Annotations were quality-checked, resulting in the removal of two annotator entries who labeled an unlikely number of samples as trauma, without affecting the total sample count (e.g., 1,200).For the Incel dataset, we only labeled 300 instances since it serves as a control test set. To ensure quality, two researchers with psychology degrees annotated a subset of 200 instances from each dataset and resolved disagreements through discussion (Cohen’s $\kappa=.82$).

Annotator Agreement. To assess annotator consistency, we report Krippendorff’s $\alpha$ for agreement among crowdworkers and provide Binary F1 scores to measure agreement between the crowdworker majority vote and the expert vote, with the latter serving as the ’true’ reference (Table 1). Both agreements were best for the Counseling Dataset. All agreement scores indicate at least moderate agreement (Krippendorff, 2018). Despite variability, our primary focus is on the accuracy of labels from majority voting. The moderate F1 scores indicate that majority votes are reliable labels, supporting the robustness of our annotation process. Given the subjective nature of interpreting trauma-related constructs, some disagreement is expected, similar to lower agreement seen in tasks like hate speech detection (Li etal., 2024). This level of agreement, while not perfect, provides a solid foundation for the study.

4.1 Models and Hyperparameters

In this work, we implement five sequence classification models for natural language inputs. The suitability of the these models for trauma detection in different contexts is defined by criteria such as complexity, interpretability, hyperparameter optimization, and scalability. To help in understanding the trade-offs and strengths of each approach, we provide an overview of the models considered in Table2. The hyperparameters given are optimized with a hyperparameter optimization framework.

BoW-Naive-Bayes Model. The simplest model is obtained by fitting a Naive-Bayes model on the word counts in both classes. Let $\mathbf{t}=[t_{1},t_{2},\ldots,t_{N}]$ be an input sequence.We model the log-odds by combining two key components. First, we calculate the prior odds, which is the log of the initial ratio of the probabilities of the two categories. Second, we add the word-specific weights, which are summed over all elements in the input sequence. Each weight represents the log of the ratio of the probabilities of that element occurring in each category.

We obtain the weight of a term by counting its occurrences in documents from both classes and applying Laplace smoothing with a specified hyperparameter $\alpha$. The main advantage of this linear model is its interpretability due to the individual weights of each token that are explicitly computed.

N-Gram Logistic Regression Model. We compute $n$-grams for the datasets and fit a logistic regression model on the TF-IDF represenation of the $n$-Grams, where $n$ is $[1,2,3]$.

TF-IDF Fully-Connected Model. Furthermore, we compute TF-IDF vectors for the samples and train a fully connected neural network using this representation as an input. We use either one or two hidden layers, with the number of hidden layers and their width as a hyperparameter.

BERT-based Models. We train the popular encoder-only transformer models BERT Devlin etal. (2019) and RoBERTa Liu etal. (2019). We experiment with both pretrained and non-pretrained versions of these models. We find that the pretrained models yield superior performance, which is why we restrict our analysis to these models for the main paper. We use the learning rate, number of layers, and number of heads as hyperparameters.

Black-box API models (GPT-3.5/GPT-4). We use a prompt template to access publicly available foundation model APIs for GPT-3.5 and GPT-4 Achiam etal. (2023). We rephrase the classification tasks as a sequence completion tasks by using prompt template, which instructs the model to either output “0” or “1”, and apply basic prompt engineering, including a task definition, the trauma definition, and labeling instructions (see Appendix A.2). We use the top token log-probabilities returned by the API to compute class log-odds, which can be used to compute calibration measures and ROC curves.

4.2 Explainable AI Methods

We use explainable AI approaches to gather insights on how trauma is described and recognized across different domains. Feature-based explanations allow us to gain insights into the importance of individual input features, i.e., tokens. We chose model-agnostic approaches that treat the predictive model as a black-box function and can be applied to any model (SHAP values) and model-specific, mechanistic approaches that are only applicable to specific models but can more faithfully describe the output of certain model classes.Additionally, concept-based explanations allow us to move beyond individual feature attributions to a higher level of abstraction, and help us identify interpretable concepts that are crucial for trauma detection without requiring extensive supervision. These methods collectively enhance our ability to interpret model predictions and validate their reliability.

SHAP Explanations. Shapley values originate from game theory and have been proposed to compute the contribution of individual features to the output of a non-linear function. They are a form of feature attribution explanation that assigns each input token a numerical score. The score corresponds to the average contribution to the output obtained when this feature is added. We compute SHAP values using an efficient sampling-based algorithm with the implementation of Lundberg and Lee (2017).

SLALOM Explanations. Leemann etal. (2024) have shown that single attribution scores cannot fully describe the inner workings of modern transformer language models. The authors propose SLALOM, a model to assess the role of input tokens along two dimensions: A token value score, describes the effect each token has on its own, while the token importance describes how much weight is placed on each token when tokens are concatenated to sequences. While SLALOM can be used to approximate any model’s behavior in principle, it is particularly suited for transformer models, like the BERT and RoBERTa models used in this work.

Concept-based Explanations. Concept-based explanations have been proposed as an alternative to feature-wise explanations. They do not reason over individual input features (tokens, pixels, etc.) but instead use a higher level of abstraction Kim etal. (2018); Koh etal. (2020). However, it is difficult to discover meaningful concepts from the data without supervision Leemann etal. (2023). In case no concept annotations are present in the data, they identify clusters in a model’s latent space that best describe a model’s decision. In this work, we turn to Completeness-Aware Concept-Based Explanations Yeh etal. (2019), which are one of the few conceptual explanation techniques that are applicable to textual inputs and do not require supervision in terms of the data. The concepts are represented as a set of salient examples, i.e., sample snippets that most strongly exhibit the discovered concept.

In this study, we focus on the RoBERTa architectures for concept-based text classification, which proved reliable across all datasets. We use the logit outputs of this model to obtain SHAP and SLALOM explanations and use the latent representation before the classification head as the latent space where the concept vectors are identified. Details on explanation approaches and their hyperparameters are provided in SectionA.1.

Classification Performance

We fit all the models to the respective datasets after performing hyperparameter optimization (cf. SectionA.2) and report their performance metrics in Table3.The evaluation across GTC, PTSD, and Counseling datasets shows clear trends. Transformer-based models, especially fine-tuned BERT and RoBERTa, significantly outperform traditional models and feedforward neural networks. The Naive-Bayes-BoW and NGram Logistic Regression models show moderate performance but lag behind due to their simpler architectures. The feedforward model performs reasonably well but is outclassed by transformer models. Fine-tuned BERT and RoBERTa exhibit substantial improvements in all metrics, with RoBERTa achieving the highest F1 scores in the GTC dataset ($F1=.74$) and the PTSD dataset ($F1=.71$), highlighting its effective language comprehension capabilities. To control for dataset size effects, we ran an additional experiment using 1,000 randomly selected GTC samples in the training set to match the size of other datasets. The performance remained consistent, indicating that our findings on smaller datasets likely extend to larger ones (AppendixA, Table7).

OpenAI’s GPT-4 also performs particularly well on the PTSD and Counseling datasets and even outperforms BERT in the F1 metric on Counseling, showcasing its strong generalization abilities despite not being further fine-tuned and relying on a single prompt for these tasks. Interestingly, all models perform reasonably well, which may be attributed to the specific task of trauma event detection. However, the Counseling dataset proved more challenging due to its very imbalanced class distribution and the presence of very few trauma event samples. This is reflected GPT-4 F1 score of $.36$, which was the highest for this dataset but still indicates the difficulty of the task.RoBERTa achieves strong performance metrics overall, highlighting the impact of architectural improvements and extensive training on larger datasets, though it does not outperform BERT on the Counseling dataset.

Cross-Domain Performance

Figure2 presents the cross-domain results of RoBERTa models fine-tuned on one dataset and evaluated on other datasets, using the AUC-ROC metric (cf., AppendixA, Table5). Models trained on the GTC dataset showed the highest generalizability, performing well across all test sets. Those trained on the PTSD dataset excelled on their own test set and performed strongly on others. Models trained on the Counseling dataset achieved top performance on their own set but did less well on others. The model trained on all combined datasets showed robust and consistent performance across all test sets, maintaining high accuracy and reliability.Despite differences in trauma types across datasets, significant overlaps contribute to strong cross-testing results. For example, both the GTC and PTSD datasets include trauma related to death, acute stress reactions, and physical violence, aiding models’ cross-dataset performance. However, the GTC dataset’s unique military component may cause some performance differences. Overall, high cross-domain performance suggests that shared trauma themes enable effective generalization across different contexts.

The results show that the RoBERTa model fine-tuned on the PTSD dataset has the best generalizability across different datasets, with models trained on the full data also performing well. Given the diversity of traumatic events across datasets, this result suggests the trauma features in the PTSD dataset are broadly applicable for learning a general event type, rather than causing models to pick up on only keywords. Counseling-trained models perform well on their own dataset but do not generalize as effectively. Performance on the Incel dataset indicates all models effectively differentiate trauma-related vocabulary from control data.

SHAP Explanations

To understand how the models attribute feature importance to the trauma label, we calculated SHAP values for some samples from all datasets, focusing on comparing RoBERTa and GPT-4 due to their high performances and the interesting differences in how these language models classify trauma. While most classifications aligned (see Figure8 in Appendix A), we found that, in several instances, GPT-4 provided more non-trauma attributions for certain features compared to RoBERTa.

Figure 3 shows a counseling dataset example where RoBERTa and GPT-4 disagree. RoBERTa assigns high relevance to words like yells, abuse, and depressed, while GPT-4 does not, possibly due to the forum user’s uncertainty about defining abuse. This discrepancy may stem from GPT-4’s closer adherence to the APA definition of trauma, with less variation and personal bias than human annotators, who may classify events based on their own experiences and interpretations.

These findings, though based on exemplary instances, highlight the challenge of detecting mental abuse. RoBERTa may rely more on specific keywords related to abuse, whereas GPT-4 seems to consider contextual nuances. Human annotators might interpret such incidents as traumatic based on subjective judgment and empathy, while GPT-4, adhering strictly to the APA definition of trauma, did not classify these incidents as trauma.

Feature Characteristics with SLALOM

The SLALOM feature importance scores from all datasets focus on the highest value features for trauma classification. Features like dream and shattered, in the top right corner, contribute most to the trauma classification. For clarity, overlapping features were excluded (blue dots remain in the figure) (Figure4).

Notable feature variability includes war-related vocabulary (e.g., bombardment, bullets) likely from genocide-related data, and more generalizable words (e.g., dreams, accident, dead) applicable across domains. Amplifying words like intense, suddenly, and gloomy also appear, fitting traumatic contexts without specific events.

Groups of thematically related words are evident: dead and assassinated represent death, wounded, choking, and slapped indicate physical injury and violence, and dreams, shattered, and replay are associated with trauma’s psychological impact.

Conceptual Explanations

For each dataset, we assessed conceptual explanations to detect context-specific trauma concepts. We select the concepts that have the highest number of traumatic instances in the neighborhood closely associated with the corresponding concept (Figure5).

In the genocide dataset, concepts related to killings, death, and severe injuries were prominent, reflecting the extreme nature of the content. In contrast, the PTSD and counseling datasets, which address more everyday trauma, contained more references to domestic violence and abuse. The smaller size of the counseling dataset made it challenging to identify unique concepts without overlap.

Across all contexts, death and sexual violence were prevalent. In the genocide dataset, these were depicted through killings and executions, whereas in other datasets, they were associated with grief, loss, and suicide. Sexual violence, particularly rape, consistently appeared as a common source of PTSD, which is consistent with the psychological literature Atwoli etal. (2015).

A.1 Implementation Details: Explanation Methods

In this section, we give more details on how we computed the explanations shown in this paper.

SHAP Values. To obtain SHAP values, we use the official shap¹¹1https://github.com/shap/shap package. We use the TextExplainer class.

SLALOM. We use the SGD algorithm proposed in Leemann etal. (2024) to estimate the SLALOM model on 100k background samples of length 2. We use all the tokens that appear in the samples from the datasets used and fit one global SLALOM model.

Conceptual Explanations.We use the completeness-aware loss proposed by Yeh etal. (2019) with snippets of length of 5 token as snippets for the algorithm.We trained with concept discovery module to discover $K=10$ concepts using the Adam optimizer at an initial learning rate of $1\times 10^{-3}$, decaying to $5\times 10^{-4}$ and $1\times 10^{-4}$ in subsequent epochs. Training lasted 3 epochs with a batch size of 12. The model weights used were obtained from the best-performing model. We identified the 25 closest activations per concept. Evaluation on a separate test set involved dot products between latent representations and concept vectors, selecting the top activations.

A.2 Implementation Details: Models

We use the optuna²²2https://optuna.org/ framework for hyperparameter optimization with 50 steps for each model/dataset. We then train the models using different seeds and on five random data splits using the discovered hyperparameters. Through the optimization we obtain the parameters given in Table4.

Prompt Template. We use the following prompt template to prompt the GPT models as the system prompt.

"You are tasked with detecting trauma in text segments of transcripts of genocide tribunals. Specifically, detect instances that meet the APA’s definition of trauma. Psychological trauma, as defined by the APA, includes experiences of exposure to actual or threatened death, serious injury, or sexual violence, either directly encountered or witnessed. It also includes instances where individuals learn that the traumatic event(s) occurred to a close family member or friend. Label the text with ’1’ if there are indicators of trauma based on this definition, and ’0’ if there are no indicators of trauma. Note that trauma is rare and occurs in less than 20% of the cases. Only answer with either ’0’ or ’1’."

The samples are then passed as a user prompt.

A.3 Annotation Details

Participants were prescreened using Prolific based on self-reported English-language proficiency. We did not collect demographic data from the annotators as such data was not central to the questions our study is focused on and Prolific does not normally include this metadata.

A.4 Metrics

For completeness, we additionally report accuracy, recall, and precision for the trained models in Table6.