First, we will review the related NLP methods in our comparison study. Traditional methods include the Lexicon-based approach, where predefined dictionaries of emotional words are used to identify emotions in text. For example, Mohammad et al. [21] developed the National Research Council of Canada (NRC) Emotion Lexicon (EmoLex), a widely used resource for Lexicon-based emotion detection. The model establishes connections between words and basic emotions, including anger, joy, and sadness. This Lexicon was developed through crowdsourcing, ensuring a diverse and comprehensive set of word-emotion associations. Mohammad [20] later extended their work by adding real-valued scores of intensity to emotions to create NRC Affect Intensity Lexicon (AIL), enabling more fine-grained analysis.

The next school of methods for text classification involves converting sentences to numeric vectors using BoW or TF-IDF and applying machine learning algorithms to them. Sebastiani [33] provided a thorough analysis of these algorithms, highlighting their performance across different datasets and establishing their strengths and limitations in text classification. BoW is a technique that turns text or images into a histogram of words. BoW models convert text into a matrix of token counts, representing the frequency of each word in the text. This representation is then used as input for machine learning classifiers such as logistic regression, SVM, or Naive Bayes. Based on the study in [29], this makes the BoW computationally simple, helping it score well on performance tests. Barry [2] studied using BoW on Amazon and Yelp food reviews to classify whether they were positive or negative. With its best machine learning model, they achieved an accuracy score of over $95\% $. Desmet and Hoste [11] used BoW to detect 15 emotions. Their results varied by emotion, but six of the seven most common emotions had acceptable accuracy.

TF-IDF improves upon BoW by weighting terms based on their importance, calculated as the product of Term Frequency (TF) and Inverse Document Frequency (IDF). Ramos et al. [32] explained that the less frequently a word appears in documents, the greater the weight it should receive. This weighting helps to emphasize significant words while downplaying common ones, enhancing the model’s ability to distinguish between different classes. Rahman et al. [31] conducted sentiment classification by tweaking TF-IDF with various vectorization methods and classifiers. With the correct classifier, they achieved $100\% $ accuracy. Sundaram et al. [37] used TF-IDF for six emotions. For emotions with large training sets, they had an accuracy of about $85\% $.

The advent of the Transformer architecture in deep learning, such as BERT and GPT, has revolutionized NLP. BERT, introduced in [12], employs a bidirectional training approach to understand the context of words in a sentence. BERT’s architecture comprises multiple layers of encoders within the Transformer, enabling it to capture intricate relationships between words. BERT can be fine-tuned for specific tasks, such as emotion detection, which involves additional training on a labeled dataset to optimize the model’s performance for that particular task. Tang et al. [39] further explored the fine-tuning of BERT for multi-label sentiment analysis, showcasing its effectiveness in handling multiple co-occurring emotions under unbalanced class distributions. Ji et al. [17] developed MentalBERT, a BERT-based model fine-tuned on mental health-related text, demonstrating significant improvements in understanding and classifying emotional content compared to standard BERT models.

GPT models [30, 27], such as GPT-3.5, leverage generative pre-training on a vast corpus of text to generate human-like responses. Floridi and Chiriatti [13] explained that such models will transform the writing process and are capable of producing texts on the level of some humans. These models can be adapted for emotion detection by fine-tuning them on specific datasets or using prompt engineering to elicit desired outputs. Jain et al. [16] used two GPT models for emotion detection, achieving an accuracy score of 0.98 over the mental health datasets they tested it on. The BERT and GPT models show much promise, with the GPT models being the most cutting-edge technology available.

For this study, multiple fine-grained emotions related to mental health, such as isolation, anxiety, and depression, need to be detected from students’ responses to the open-ended question. Bouzazizi et al. [5] tackled the challenging task of multi-class emotion detection on Twitter posts, achieving $60.2\% $ accuracy for seven emotion classes. Their study emphasized the complexity of multi-class classification and proposed a model to better extract and understand emotions present in text rather than classifying them into predefined categories. The authors introduced a system that first classifies text as positive or negative and then assigns scores for corresponding emotion subclasses, improving the robustness and accuracy of emotion classification. Demszky et al. [10] created a labeled dataset of 58k comments for 27 emotions, including gratitude, confusion, and remorse. They also trained a BERT-based model, achieving 0.46 F1 score. Mustafa et al. [22] leveraged Twitter data and machine learning to classify depression severity, achieving $91\% $ accuracy by analyzing the top 100 words used by individuals and their psychological attributes. The study highlighted the importance of feature selection in enhancing classifier performance and proposed incorporating additional data, such as emojis and images, to improve future analyses. Guo et al. [14] introduced a multi-way matching deep neural network model for fine-grained emotion detection of user reviews. Their approach predicted scores for specific attributes within reviews, such as location, service, price, and environment. The model consists of two steps: attribute detection and attribute classification. In the first step, the model identifies the relevant attributes mentioned in the text. In the second step, it assigns a score ranging from $-5$ to 5 for each emotion, reflecting the user’s opinion. This fine-grained analysis offers a more detailed understanding of user emotions by focusing on specific aspects of their reviews, demonstrating that NLP methods can effectively distinguish between various emotion categories.

The dataset has been previously studied in [1] and [7], and was collected from students at a large mid-Atlantic university in the U.S. between April and June 2020. We focus on one part of the collected data containing short-answer responses from students concerning the impact of COVID-19 on their mental health. The students had to answer the question, “How is coronavirus/COVID-19 affecting your mental health?” The responses are labeled manually by our research team with various emotional indicators, which serve as the ground truth for our emotion detection models. Each response is annotated with binary labels for 10 emotions: Isolation, Depression, Anxiety, Negative Feelings, Lack of Motivation, All Stress, Issues With Home Life, No/Positive Effects, Lack of Routine, Miscellaneous. The binary labels (1 if positive, 0 if negative) indicate whether the labelers believed the respondent’s answers expressed the corresponding emotions.

The ten categories of emotions are inspired by [6] and [35]. In Appendix A, we provide an example for each emotion, and explain our definitions of Negative Feelings, All Stress, and Miscellaneous. A response could be labeled into more than one emotion category. To ensure the labeling quality, two team members collaboratively categorized emotions for each response. For any questionable answers, they would sort out with multiple members to reach a consensus on labeling. In preparation for our analysis, the data were cleaned by removing responses with no emotion detected, which typically occurred when there was no response or only random characters. We also removed responses that lacked demographic information to facilitate future analysis. This left 398 responses in the dataset. The percentages of the remaining responses labeled as 1 for each emotion, ordered from largest to smallest, are illustrated in Figure 1.

In Figure 2, we analyze correlation and hierarchical clustering for all emotions. We find that most of the emotion pairs have near-zero correlations. The clustering analysis shows that “Depression” and “Anxiety” are the closest emotions. However, their correlation is only 0.3. Other close emotions also show small correlation coefficients of $\sim 0.1$. Thus, we simplify this multiple-label classification problem into 10 binary classification problems. For every NLP method, we train ten models, each using labels for a single emotion. This simplification will provide a fair method-comparison framework.

In this section, we examine the performance of four trainable methods – Lexicon, BoW, TF-IDF, and MentalBERT – as well as the zero-shot method, GPT. For the four trainable methods, we consider training data splits of $80\% $, $50\% $, and $20\% $ to evaluate the impact of training sample size. We choose accuracy and F1 scores from the testing data, as defined in Section 3, as the performance criteria. For each percentage, we repeat the training/testing splits 100 times, and report the average performance criteria. The original results are presented in the tables in Appendix C.1.

To compare the performance of the five methods, we show the boxplots of their accuracy (upper panel) and F1-scores (lower panel) with various training percentages in Figure 3. We observe that the training percentage has a limited effect on the Lexicon’s performance. The median F1 score of the Lexicon with a $20\% $ training split is even slightly higher than those with $80\% $ and $50\% $ training percentages. The performance of BoW and TF-IDF shows modest impacts of the training percentage, and their overall trends are similar to each other. At last, the performance of MentalBERT, especially its F1 score, is significantly impacted by the training percentage. The median F1 score of MentalBERT with $80\% $ is the highest among all NLP methods; however, its median F1 score with $20\% $ is the lowest. A possible reason is that $20\% $ of the training data, comprising only 80 instances, is insufficient to fine-tune the 110M parameters in the MentalBERT model. The zero-shot GPT demonstrates modest performance, similar to BoW and TF-IDF with a $50\% $ training percentage. However, its F1 scores surpass BoW, TF-IDF, and MentalBERT with a $20\% $ training percentage. Overall, the Lexicon method is relatively resistant to decreases in the training dataset, while MentalBERT is the most sensitive. Meanwhile, the performance changes in BoW and TF-IDF with varying training set sizes are moderate.

We examine the performance of those methods for individual emotions. We select three emotions: Depression, Lack of Motivation, and Miscellaneous, which present various levels of detection performance. The emotion with good performance, Depression, can be identified by keywords such as “depression” and “depressed” from the responses. The emotion associated with poor performance, Miscellaneous, is ambiguous and has only a small positive sample size. The moderate one, Lack of Motivation, has a clear definition but requires a comprehensive understanding of the responses to detect it. The accuracies and F1 scores of the three typical emotions of NLP methods with various training percentages are shown in Figure 4. We also highlight the values of the three typical emotions in Figure 3.

For Depression, the four trainable NLP methods with $80\% $ training percentages achieve high performance with an accuracy of $0.9\sim 1.0$ and F1 scores of $0.8\sim 1.0$. Moreover, Lexicon, BoW, and MentalBERT all achieve significant improvements in both accuracy and F1 score compared to GPT, when the training percentage exceeds $50\% $, whereas TF-IDF yields a minor improvement. However, with a $20\% $ training percentage, the performance of TF-IDF and MentalBERT drops dramatically, while the performance of Lexicon and BoW remains stable.

For the emotion with moderate detection difficulty, Lack of Motivation, the four trainable methods, with an $80\% $ training percentage, can achieve higher or similar accuracy or F1 scores compared to the zero-shot GPT. However, the increase between the trainable methods and the GPT is smaller than that for the Depression case. The F1 score of the MentalBERT is superior to that of other methods, demonstrating its capability to understand the context of the responses. When the training percentage reaches $50\% $ and $20\% $, the F1 scores of most trainable methods decrease to levels below or similar to those of GPT. This suggests that we require a sufficiently large dataset to train the model for this emotion.

For the challenging emotion, Miscellaneous, the four trainable methods, along with GPT, achieve high accuracy of around $0.90\sim 0.95$ and low F1-scores of around $0\sim 0.1$. The results show that there is a very small number of positive data points for the Miscellaneous emotion, and the NLP methods will predict all sentences as negative. Due to this issue, the training percentages cannot help the performance of the four trainable results. Moreover, the zero-shot GPT also fails to recognize this ambiguous definition with a near-zero F1 score.

The next step is to evaluate the stability and distinguishing ability of the four trainable methods: Lexicon, BoW, TF-IDF, and MentalBERT. We find the mean and standard deviation of the accuracies and F1 scores from 100 repetitions of stratified 5-fold cross-validation. The means are similar to those with an $80\% $ training percentage in Section 4.1, as the models using 5-fold cross-validation were also fitted from $80\% $ of the data. The standard deviation tells us how the performance measurements change depending on which $80\% $ of the data they are trained with, showing the stability of the methods. Then, we calculate the average AUC of the 100 repetitions, which indicates the model’s ability to separate each emotion. The AUC is close to 1 when a method is capable of identifying all instances of an emotion with very few false positives. At the same time, an AUC of 0.5 means the method distinguishes an emotion no better than a random guess. The original results are presented in the tables in Appendix C.2.

The left and middle plots in Figure 5 show the boxplots of the standard deviations of the accuracy and F1 scores, and the right plot illustrates the boxplots of average AUCs among 10 emotions for the four trainable methods. There is no result for GPT, as it does not require a training process and cannot produce a probability of a positive detection. For the accuracy, it is clear that Lexicon has the smallest standard deviations, while MentalBERT has the largest. For the standard deviations of F1 scores, the trend is less obvious. However, the lower bound of Lexicon’s boxplot is lower than that of the other three trainable methods, showing that it can achieve the highest stability for some emotions. Those plots show that Lexicon excels in the stability measurements, while MentalBERT’s performance is sensitive to the part of the data with which it is trained. In the right plot, the upper bounds of the boxplots are all close to 1.0, but their lower bounds are different. Among the four methods, TF-IDF shows the highest average AUCs, and Lexicon and MentalBERT have the lowest values. Thus, when using AUC as the criterion, TF-IDF shows the highest distinguishing ability for some challenging emotions.

We show the stability and distinguishing ability measurements for the three typical emotions identified in Section 4.1. Their values are also highlighted in Figure 5 with special legends. For the emotion with good detection performance, Depression, the standard deviations of accuracy and F1 score are close to 0 for Lexicon, and slightly increase from BoW and TF-IDF to MentalBERT. The AUCs of all four methods are almost 1, showing excellent distinguishing ability. For the moderate one, Lack of Motivation, the standard deviation of accuracy again increases from Lexicon, BoW, TF-IDF, to MentalBERT. However, MentalBERT achieves the smallest standard deviation of F1 scores. One possible reason is that MentalBERT is better able to understand the concept of Lack of Motivation, thereby improving its stability. Their AUCs are around 0.75 and 0.85, and TF-IDF achieves the highest AUC.

For the emotion with poor performance, Miscellaneous, the standard deviations of Lexicon are still low despite its overall poor averages shown in Section 4.1. MentalBERT again has the largest standard deviation in its accuracy and F1 scores. The AUCs are around 0.5 and 0.6, indicating the model’s prediction is slightly better than the random guess. Among the four methods, TF-IDF has the highest AUC, while Lexicon and MentalBERT have the lowest. We can conclude that for both easy and challenging emotions, Lexicon exhibits the highest stability, while MentalBERT shows the lowest. For the moderate one, MentalBERT’s stability becomes better. For moderate and challenging emotions, TF-IDF’s distinguishing ability outperforms others when using AUC as the criterion.

In this section, we aim to compare the consistency between the detections of different NLP methods, which indicates whether they yield identical predictions for each student’s response. To handle emotions with a very small number of positive samples, we choose the Jaccard index, also known as the Jaccard similarity, to measure consistency. The Jaccard index between the two methods for a certain emotion can be calculated as: where $I(\cdot )$ is the indicator function, and ${\hat{y}_{r}^{{\text{Meth}_{1}}}}$ and ${\hat{y}_{r}^{{\text{Meth}_{2}}}}$ denote whether the two methods predict is that emotion expressed in response r. The prediction results are based on the first repetition of the 100 5-fold cross-validation processes in Section 4.2. We calculate the Jaccard indices from the 15 pair-wise comparisons between five NLP methods and the true labels, and the results of the ten emotions are listed in Appendix C.3. Figure 7 presents the average pairwise consistencies over the ten emotions. MentalBERT vs. true labels (0.5) and BoW vs. TF-IDF (0.48) have the two highest average Jaccard indices. The first pair demonstrates the capability of trainable Transformer models to predict results close to the true labels. Meanwhile, the second pair is possibly caused by the similar mechanisms of the two methods, which first convert the responses to vectors and then train a machine learning classifier. Then, we examine the leftmost column, which shows the consistency of the five NLP methods compared to the true label. MentalBERT has the highest score (0.5), followed by BoW (0.43), and Lexicon, TF-IDF, and GPT have the lowest scores (0.37).

Finally, we present the Jaccard indices for the three typical emotions. For the good one, Depression, almost all pairwise consistencies are above 0.5, while the consistencies between the true labels, Lexicon, BoW, and MentalBERT are higher than 0.75. However, the TF-IDF and GPT generate relatively inconsistent predictions, while their Jaccard index is lower than 0.5, which is consistent with the results in Figure 4 where these two methods show lower accuracies and F1 scores. For the emotion with moderate detection performance, Lack of Motivation, we find that MentalBERT achieves the highest consistency with the true labels, which is around 0.6, while the other Jaccard indices range from 0.22 to 0.57. This result highlights the capabilities of trainable LLMs. For the emotion with poor performance, Miscellaneous, every pairwise comparison is below 0.1, indicating that none of the NLP methods, whether trainable or not, can capture the ambiguous concept of Miscellaneous.

Performance and Complexity Trade-off: The NLP methods investigated in this study exhibit dramatically different levels of complexity. The Lexicon-based method requires learning scores for only dozens of keywords, while MentalBERT must fine-tune over 100 million parameters in its Transformer architecture [17]. These complexity differences translate into distinct performance patterns. The Lexicon-based method achieves stable performance across training sample sizes ranging from $20\% $ to $80\% $, demonstrating the lowest standard deviations in both accuracy and F1 scores. However, its peak performance cannot compete with more sophisticated methods. In contrast, fine-tuned MentalBERT demonstrates the highest accuracy and F1 scores with $80\% $ training data and achieves the best Jaccard Index agreement with true labels. Yet its performance drops dramatically with limited training data ($20\% $), and it exhibits the highest variability in repeated cross-validation experiments.

BoW and TF-IDF methods, both of which convert responses into numeric vectors before training traditional machine learning models, demonstrate balanced performance, complexity, and stability. Their methodological similarity is reflected in their relatively high pairwise Jaccard index. While TF-IDF achieves better AUC scores than BoW, indicating good distinguishing ability for challenging emotions, its performance declines more rapidly with reduced training data.

The zero-shot GPT method, implemented through OpenAI’s API, requires no training process and achieves performance comparable to BoW and TF-IDF models trained on $50\% $ of the data, though it underperforms compared to MentalBERT trained on $80\% $ of the data. Notably, Lossio-Venture et al. [19] found that zero-shot ChatGPT outperformed fine-tuned Transformers in sentiment analysis for COVID-19 survey data. This apparent discrepancy likely stems from task complexity differences: sentiment analysis predicts general positive/negative sentiment, a more universal task that ChatGPT’s vast training data can handle effectively. In contrast, our fine-grained emotion detection requires distinguishing among ten mental health-related emotions, many specifically defined by our research team. Without access to labeled training data, the zero-shot GPT model cannot accurately detect these domain-specific emotional categories.

Method Selection Suggestions: Based on our performance observations, we offer the following guidelines for NLP method selection. The Lexicon-based method proves particularly effective for detecting well-defined emotions with clear linguistic indicators, especially when training data is limited or stability is prioritized over peak performance. Fine-tuned MentalBERT is most suitable for detecting contextually complex emotions when sufficient training data ($\gt 50\% $ of available samples) and computational resources are available. Traditional machine learning methods, such as BoW and TF-IDF, provide effective predictions when labeled data or computational resources are insufficient for Transformer models. Finally, zero-shot GPT can generate quick assessments when no training data is available, though performance will be limited for domain-specific emotions.

Uncertainty in Mental Health Studies: When deploying NLP methods for emotion detection in mental health research, their inherent uncertainty must be carefully examined, as detection results can have serious consequences for both research conclusions and potential interventions. Uncertainty arises from multiple sources, beginning with the training dataset itself. As demonstrated by our experiments, most NLP methods show performance sensitivity to training size variations. Additionally, repeated cross-validation reveals that resampling the training dataset while maintaining the same sample size yields variable predictions, particularly for high-complexity models like fine-tuned MentalBERT. Therefore, sensitivity analysis for training sample size and cross-validation repetitions is essential for evaluating model stability.

A second source of uncertainty stems from the binarization of predicted probabilities. All four trainable methods output probabilities indicating the likelihood of emotion presence, with positive detection determined by a 0.5 threshold. However, a response with 0.99 prediction probability represents a different uncertainty level compared to one with 0.51 probability. In this study, we employed AUC to evaluate the distinguishing ability of NLP models based on predicted probabilities, and ROC curves can provide additional insights into model uncertainty characteristics.

Emotion Labeling Impact: A crucial finding is that the target emotion significantly impacts NLP method performance. Figures 3 and 5 demonstrate that accuracy and stability measurements for the same method vary substantially across the ten emotions studied. Training sample size sensitivity also depends on the specific emotions being detected. Consequently, method comparisons yield different conclusions for different emotion types, as illustrated by our analysis of Depression, Lack of Motivation, and Miscellaneous categories. Such performance disparities across emotions can lead to inconsistent findings in studies relying on NLP detection results. While traditional mental health studies, such as [1], design emotion categories based on domain knowledge, the increasing role of NLP methods in data analysis necessitates careful selection and design of emotion categories to ensure both performance and stability of automated detection algorithms. For instance, Miscellaneous, a convenient category for human labelers, leads to poor performance for all NLP methods. Such categories should be avoided when incorporating NLP methods for data analysis.

We note that all four trainable methods can improve the model’s consistency in producing predictions that are similar to those of human labelers. Thus, the zero-shot GPT-3.5 performs relatively poorly compared to other methods when we use human labels as the “golden standard”. However, as pre-trained LLMs become more powerful, their predictions could be more valuable when the quality of human labels cannot be ensured. Collaboration between LLMs and human experts can be beneficial for mental health research.

Multi-label Classification: There are machine learning research related to multi-label classification [3, 40]. They found that by adopting specialized methods, such as problem transformation and algorithm selection, we can capture the inner structure of the labels to improve accuracy. In Figure 2, there are weak associations between our ten emotions. For example, No/Positive Effects have $-0.20\sim -0.03$ correlations with other 9 emotions. Employing multi-label classification to improve the efficiency of emotion detection would be an interesting future study.