Detecting Mental Distresses Using Social Behavior Analysis in the Context of COVID-19: A Survey

4.1.1 Linguistic Features.

The linguistic features are extracted from the language use patterns such as word choice, topics of interest, and sentence structure. One of the most used linguistic feature techniques is to map the user-expressed text corpus into a psychological dictionary and measure the word frequencies in each word class. One of the most widely used psychological dictionaries is Linguistic Inquiry and Word Count (LIWC), which is a widely used text analysis technique in psychology, and can be easily adapted for natural language processing (NLP) tasks. LIWC was introduced in the early 1990s to associate linguistic dimensions of written expression with psychological states. LIWC could compute the percentage of words within 80 linguistically or psychologically meaningful categories. These categories cover various important psychological states of an individual, including personal preferences, cognition state and emotions. Over the past 30 years, LIWC has been used in several works studying the relationships between the word categories in daily language and psychological states. For instance, the correlation between the first person singular pronoun usage and depression [59], emotions and LIWC emotion-related categories [144], and LIWC positive emotion category word usage and anxiety [137]. In the context of mental distress detection using social media analysis, LIWC is by far the most used psychological dictionary-based feature extraction method [60, 78, 81, 156]. Other works used psychological dictionaries to extract linguistic features, such as Affective Norms for English Words. Another effective linguistic feature extraction method is topic modeling, the process of extracting users’ topics of interest by analyzing the text corpus generated from their posts. Topic modeling is an effective technique in computational linguistics to decrease the input of textual data feature space to a determined number of topics. Using unsupervised text mining techniques can be used to extract hidden topics from a text corpus, such as topics related to psychological distress.

Unlike psychological dictionaries such as LIWC, it is not created by a fixed set of pre-categorized words. However, topic modeling techniques automatically compute and generate the set of non-labelled words that represent the user’s topics of interest. In the context of topic modeling for psychological distress detection, the generated topics are considered the linguistic features of the user. The most used topic modeling method is Latent Dirichlet Allocation (LDA). In LDA, a topic is represented as a multinomial distribution associated with unique words in the text corpus. Furthermore, a document is represented as a multinomial distribution over all topics. LDA is used to generate topics and provides linguistic features automatically from the user’s text corpus. Many studies leverage the original unsupervised LDA algorithm for topic extraction [48, 92, 122, 153, 171]. Some works report better results using supervised or semi-supervised LDA [34, 37, 38] that guide the topic modeling process by specifying mental-related lexicons, which are likely to appear in the generated content of distressed individuals [8, 141].

Another widely used linguistic features for mental distress detection are the word frequency features that represent the original text corpus in a representational language model. The most basic models in this category are the Bag of Words (BoW) model and term frequency-inverse document frequency (TF-IDF). In BoW, the user-generated text is represented as the bag (multiset) of its words, disregarding grammar and even word order but keeping word multiplicity. TF-IDF is a term-weighting scheme that measures the importance of a word in all text generated by the user. For example, TF-IDF might be applied to know the importance of the word “depressed” in all the tweets of a given user. One disadvantage of BoW and TF-IDF is that the order of the words is discarded, and hence important information is lost about the temporal dimension of the user’s psychological state. The N-gram model [149] solves this problem, as N-gram is an adjacent sequence of entities. The entities can be syllables, phonemes, characters, words or base pairs. N-gram models are extensively used in computational linguistics and statistical NLP for various tasks. N-gram models are used as linguistic features that represent the probability of co-occurrence of each input sentence as an unigram and bigram.

Word embedding is a deep learning technique that is used for text representation for the recognition of comparatively important words. This modelling approach is based on the mapping of every word into a corresponding low-dimensional vector, where each word is represented as a positive or negative decimal number. One of the widely used word embedding techniques is Global Vector for word representation (GloVe) [124], which searches for similar words in the whole context of the user-generated text. Another widely used word embedding is word2vec [33]. Generally, linguistic features fusing from more than one class can increase the mental distress detection accuracy compared to single class linguistic features, e.g., LIWC+N-gram [161], LIWC+LDA+N-gram [145].

4.1.2 Sentiment Features.

Sentiment analysis is concerned with extracting emotions, opinion affects from user-generated texts on social media. Knowledge from this area can also be very useful to find the psychological state of the user. Especially, the sentiments that users describe toward their personal situation could be an important marker. Besides the linguistic features described above, the user-generated text on social media may contain semantic features related to mental distress that are difficult to extract using linguistic features, such as emotions and mood. In this context, many works have leveraged sentiment analysis to extract emotional features. The most naïve method of capturing user’s sentimental features is the keyword search, in which the system looks for specific pre-defined keywords in the users’ posts, e.g., “depressed,” “feeling lonely,” or “happy” [23, 168]. The drawback of keyword approach is the lack of context of using these words, as users may use these keywords in a sarcastic manner, for instance. A more selective method is to compute the frequency of using words that belong to LIWC emotion categories [51, 97], or LDA topics that represent basic emotions. Another way to measure the importance of emotion-related words is to measure the frequency of occurrence of these words using TF-IDF, which is often used in sentiment analysis tasks [3, 166]. In addition to this, the NRC Emotion Lexicon [11] and VADER Sentiment Lexicon [61] are used for user sentiment and emotion recognition.

4.1.3 Ideogram Features.

Most social networking platforms offer users the opportunity to enrich the posted text with additional ideograms such as emojis and stickers. The usage patterns of these ideograms are rich features that can reflect the users’ psychological distress. For example, continuous usage of stressed face emojis and stickers might be associated with stress. Emoji usage choices have been associated with depression [15, 89, 155], mental distress [40], and emotions [150]. One of the main advantages of the ideogram feature is the easy and straightforward extraction, unlike linguistic and sentiment features, which require various pre-processing steps.

4.2.1 Visual Features.

Visual features can be extracted from photos and videos posted on social networking websites. Due to the processing difficulties of extracting visual features from videos, most of the previous works focused on feature extraction from photos. There are two types of social media photos, namely, profile photos and posted photos. Generally speaking, profile photos convey less information about the mental state of the user, as most of the users choose socially attractive profile photos, which is known as self-presentation biases; for instance, even depressed users express positive emotions in their profile photos [52]. As shown in Figure 2, three types of features can be extracted from user-generated photos, facial expression features, object class features, and aesthetic features. Facial expression features include those of the studied users, as well as people present in the photo. Facial expression features are not only used to identify the user’s sentiments but also information about their exposure to society. For example, the tendency to post group photos is a strong indicator of the social anxiety level of the user. Another indicator is the selfie posting frequency on the psychological state [172]. However, aesthetic features represent the photo’s characteristics, such as the color and filter choices. For example, a photo might be analyzed at the pixel level averages to extract Hue, Saturation, and Value (HSV)—the three-colour properties widely used in image feature extraction. The Hue represents the photo’s coloring on the light spectrum (varying from red to purple/blue). Lower hue values reflect in redder, while higher hue values result in the photo tending more to blue. Saturation represents the vividness of the photo. Low saturation results in the photo being greyer and faded. Value is the image brightness. Lower brightness scores reflect a darker photo. Previous research associated depression with the posted images being grayscale and low aesthetic cohesion across a variety of image features. Users suffering from social anxiety also tend to post grayscale and low aesthetic cohesion photos, but less so than depressed users [52] (see Figure 3).

Fig. 3.

View Figure

4.2.2 Audio Features.

Various features can be extracted from music listening activities using online streaming services. Thanks to the available music public dataset and online streaming services API, we can analyze and extract rich features associated with mental distress. By analyzing the user’s listening history and playlists, we can extract acoustic features of these songs, such as rhythm structure, timbrel texture, pitch, and spectral features. Lyrics features are extracted by applying textual feature extraction techniques, as discussed above, on the lyrics of the listened songs. For instance, computing the LIWC categories of the lyrics can be associated with the mood of the users’ who listened to this song.

4.3.1 Community Affiliation.

Social networks offer the users the ability to join social network groups and communities (e.g., Facebook groups and Reddit communities). Analyzing the type of groups and communities a user joins might help detect their topics of interest. Mental distress-related groups and communities can specifically help identify users with mental health issues. My previous studies have shown that topics and psycholinguistic features were found to be accurate predictors of mental disorders like depression [122], bipolar [70], and suicide ideation [130].

4.3.2 Usage Patterns.

The patterns of social media usage and the variation of usage across different periods is a strong indicator of the user’s psychological state. Usage features include the distribution of social media usage within the day, the daily usage time and the usage time increase/decrease. It has been found in Smith et al. [136] that the sudden increase in Facebook posting activity is positively correlated with the depression symptoms. For instance, there is a marginal positive correlation between Instagram hourly usage time and depressive symptoms [84].

4.3.3 Online Activities.

Besides the usage time and community affiliations, the interaction activities with posted content (e.g., likes, comments, and follows) might be correlated with certain psychological markers. For example, the authors of Reference [99] found that the depressive symptom is negatively correlated with the number of Facebook friends. Reference [84] investigated the number of strangers the user follows, and its relation with depression markers, and the number of strangers followed slightly correlated with depression. Another way to reveal the psychological problem is by analysing keyword searching behaviors [63]. Analysis of Baidu search data reveals that searches related to psychological problems increased during the COVID-19 pandemic, keywords like depressed, sad, panic, fear, insomnia, obsessive-compulsive disorder, and psychological counselling [26].

5.1.1 Classification and Clustering.

In machine learning, classification is the problem of finding to which group/class/category a new observation belongs, based on existing knowledge of a training data set that includes instances whose class affiliation is already known. For example, given three sets of user groups (low, medium, high) based on their depression level, a classification algorithm is used to analyze the user-generated data on social media and assign the user to one of the three groups (classes). One of the most effective machine learning classifiers is the support-vector machine (SVM) that has been proven to perform well with short informal text and produces promising results when applied to mental health classification tasks [31]. SVM is widely used in mental disorders detection using social networks data, including depression detection [31, 108, 145], social anxiety [25, 41, 165], and suicide risk assessment [27, 85, 146]. Other widely used classifiers for mental health assessments include K-Nearest Neighbor (KNN) [62] and Naïve Bayes (NB) classifier [19].

5.1.2 Regression Methods.

Regression is a commonly used statistical procedure to iteratively refine a measure of the error in the prediction made by the model to find relationship between the variables. It can be used to predict the risk of mental distress based on the extracted social media features of the user. Specifically, the logistic regression model can calculate the probability of a categorical variable (depressed or non-depressed) from a set of predicted features. Linear regression is also used in various mental disorder detection methods. In linear regression, the relationships between the mental disorder and user data are modelled using linear predictor functions whose unknown model parameters are estimated from the user-generated social media data. For example, in Reference [119], a linear regression depression predictive model was constructed, linguistic features from the Instagram comments and post captions including emoji sentiment analysis results, and multiple sentiment scores and meta-features (e.g., likes count and average comment length) were obtained.

5.1.3 Dimensionality Reduction.

Dealing with raw data in high-dimensional spaces can be undesirable for various reasons. Raw data are often sparse because of the problem of dimensionality, and analyzing such raw data is usually computationally expensive. Dimensionality reduction is the transformation of raw data from a high-dimensional space into a low-dimensional space such that the low-dimensional representation preserves some meaningful properties of the raw data, ideally similar to its intrinsic dimension. The objective of dimensionality reduction methods is to learn the inherent latent structure in the data, however, in this case in an unsupervised way to summarize or describe the data using less information. This can be useful to visualize dimensional data or to simplify data, which can then be used for supervised learning methods. Many dimensionality reduction methods can be leveraged in classification and regression tasks. Principal Component Analysis (PCA) is the most used dimensionality reduction technique in the context of mental disorder assessment using social media data, such as the representation of social data of depressed users [139], schizophrenia users [91], and suicidal users [76].

5.2.1 Convolutional Neural Network (CNN).

It is a type of deep neural network, mostly applied to analyzing images. CNNs are a special case of multilayer perceptrons (MLP), which are typically fully connected networks, implying every neuron in one layer is connected to all neurons in the next layer, and thus over-fitting the data. This problem is usually solved by regularizing these networks by adding magnitude measurement of weights to the loss function. CNNs use a different method for regularization: they leverage the hierarchical pattern in data and form more complex patterns using smaller and simpler data patterns, making them less connected and complex yet effective. CNNs achieve good results, especially when applied to image classification, and also yield good accuracy for data with a grid-like structure. Many studies have proven that CNNs can be utilized effectively for text classification tasks. Given the user-generated social media data, CNNs can be employed using only a single layer with many filters for sentence classification [152], or using a more complex multilayer modeled as a sequence of layers such as an embedding layer, convolutional layer, dense layers, max-pooling layer, and the output [70].

5.2.2 Recurrent Neural Networks (RNN).

It is a class of artificial neural networks in which the connections between the nodes is in the form of a directed graph along a temporal sequence that allows to model temporal dynamic behavior. Inspired by feedforward neural networks, RNNs utilize their internal state to process variable-length sequences of inputs, which makes them suitable for tasks such as unsegmented and connected series of data. The most famous variants of RNNs is the Long Short-term Memory (LSTM) and Gated Recurrent Units (GRU). LSTM is augmented by recurrent gates, also known as forget gates, which stop back-propagated errors from exploding or vanishing. Instead, errors can flow backwards over various virtual layers unfolded in space. This enables LSTM to learn tasks that require memories of events that took place thousands or even millions of discrete time steps earlier. LSTM and its variants such as Bidirectional LSTM (Bi-LSTM) are the dominant deep-learning models in the literature on social media data analysis for mental distress assessment. Some works opted to use only LSTM or its variants for mental disorders assessment [2, 58, 64, 134], while others combine more than one deep learning model, such as LSTM+CNN [146, 164] or GRU+CNN [120].

5.2.3 Transfer Learning.

Transfer learning is a technique used to store the knowledge obtained from a machine learning or deep learning model that was developed for a task and reused to tackle a different but related task. One of the widely applied transfer learning techniques is the usage of pre-trained deep-learning models as the starting point on NLP and computer vision tasks, which significantly reduces the amount of data required to train these model on the target task. This is because the neural networks have already been pre-trained on a large dataset [20, 157]. In the context of mental health analysis for mental disorder detection, pre-trained classifiers that were trained on a general-purpose NLP task can be leveraged to identify the language indicative of mental disorders [5]. For instance, in Reference [123], a transfer learning classifier is used to identify mental health-related language on Twitter, such as anxiety, depression, stress, and suicidal ideation.

6.1.1 Depression Assessment.

There exist many depression assessment methods, among which the clinical interviews administrated by practising clinicians or psychologists are the most reliable method of depression assessment. However, it is extremely difficult to conduct such interviews on a large population [142]. Psychometric self-report questionnaires for depression are generally considered valid and reliable assessment methods. Some of the most commonly used depression self-report surveys are the Patient Health Questionnaire (PHQ-9) [73], CES-D [115], and Beck Depression Inventory (BDI) [13]. To the best of our knowledge, only one social media-based depression detection study has used clinical interviews to measure the detection accuracy [158]. Some studies have asked the users to answer self-report depression questionnaires and used their answers as the ground truth to measure the accuracy of their schemes [31, 117, 118]. Alternatively, most of the works used less certain ground-truth data, such as the user participating in communities about depression in Reddit [48, 82, 90] and LiveJournal [101]. Another method is to mine the shared content on social media to establish ground truth, for example, searching the user content to find specific expressions, such as “I was diagnosed with depression” [16], or searching for self-report anti-depression medication usage evidence.

6.1.2 Depression Detection using Textual Markers.

Textual features have been used in many works as depression markers. The textual content of social media posts may also help reveal some signs of depression. For instance, a post that says, “I am feeling down,” would be considered as expressing a depressive sign. In Reference [95], depressive symptoms from Facebook posts of randomly selected university students were analyzed. The findings concluded that there was a significant positive correlation between depressive signs expressed in the subject’s Facebook posts and their score on the PHQ-9 depression scale. Similarly, the study in Reference [104] concluded that adolescents who publicly expressed daily life stress in their Facebook posts had a higher BDI score than those who did not. In the same vein, social media post of adolescent girls was analyzed in Reference [42]. It was found that posts containing somatic complaints, negative effect, and call for support are correlated with depression symptoms, while no depressive peers posts were more likely to contain an offer of help and support. The authors of Reference [133] collected 67,796 Reddit posts from 365 fathers, over a 6-month period around the birth of their child. They have used a list of “at-risk” keywords that were suggested by a perinatal mental health expert. Postpartum depression was detected by monitoring the change in fathers’ use of words indicating depressive symptomatology after their childbirth. In the context of depression related to the COVID-19 pandemic, English Twitter depression dataset was created in Reference [173] containing 2,575 distinctly identified depressed users with their past tweets, and three transformer-based depression classification models were trained based on the collected dataset. Linguistic features have been proven to be good markers of depression.

Many studies have used psychological dictionaries to analyze the language usage and word choice and their association with depression. The authors of Reference [134] proposed an unsupervised depression detection algorithm based on RNN, computed a vector of LIWC features for each post text, and feed them to the LSTM autoencoder along with network-based features modeling how users connect in the forum. The results on detecting depressed users show that psycho-linguistic features derived from the users’ social media posts are good predictors of the their depression severity. In Reference [97], the 14 psychological attributes were used in LIWC to classify the post into emotions, and assign weights to each word from happy to unhappy after LIWC classification. Machine learning classifiers were trained to distinguish the users into three depression levels—High, Medium, and Low. A depression detection method was developed in Reference [154] that combines LIWC with another 39 attribute sets and 252 depression-related words with temporal and linguistic styles. The authors of Reference [145] analyzed Reddit user posts to detect depression from textual features and found that combining more textual feature types can yield very high depression detection accuracy. They combined LIWC, LDA and Bigram features, and fed them to the MLP classifier resulting in a performance of 91% accuracy and 0.93 F1 scores for depression detection. A depression predictive model was proposed in Reference [119] that makes use of linguistic features, sentiment scores, number of likes, average comment length, and emoji sentiment features.

6.1.3 Depression Detection using Multimedia Markers.

Depression signs are strongly correlated with the users’ multimedia features such as profile photo choice, shared photos and videos, and music-listening activities. Compared to other psychological distress and mental disorders, the depression has a strong correlation with multimedia features. In Reference [117], the authors used CES-D scale as ground-truth information, and collected 43,950 Instagram photos of 166 users. They applied colour analysis, metadata components, and face detection on the collected photos and investigated the presence of a correlation between the photo properties and depression scores. Specifically, face detection was used to count the number of human faces in the photo as an indicator of the participants’ social activity levels. Pixel-level averages were calculated using (HSV) values with three color properties widely used in photo analysis. The authors concluded that human ratings of photo attributes (happy, sad, etc.) are weaker predictors of depression, and no correlation was found with the computationally generated features. Similarly, various textual features such as LIWC, BoW, FastText embedding, and ELMo embedding were combined in Reference [88], along with the color of images as visual features. Specifically, the authors extracted the HSV features by taking the average of the pixels in the image, and also counted the number of faces in each image using a deep learning face detection model, resulting in a total of 12 visual features and 64 textual features. In Reference [87] is proposed a multi-modal depression prediction model that leverages text, images, and videos shared by the user to implement a joint representation. The model uses word2vec to extract textual features, VGG-16 to extract image visual features, and faster R-CNN to extract video visual features. Finally, these features are utilized to obtain a weighted average score, which is used for making the final prediction using the Softmax prediction layer.

6.1.4 Depression Detection using Behavioural Markers.

Online activities can be rich information to study the preferences and behaviors, as well as the differences among users. In Reference [136], it was observed that the sudden increases of Facebook posting activity are positively correlated with depression symptoms. The author in Reference [99] found that depressive symptoms are negatively correlated with the number of Facebook friends, and concluded that the connectedness of social networks (i.e., the number of mutual friends) is positively correlated with fewer depression signs, and depressed people tend to use the location tag function and the “like” button less often. In Reference [118], it was shown that the average word count per tweet was negatively correlated with depression; however, unlike Reference [136], no significant correlation between the depression symptoms and the frequency of tweets was observed. The relationship between the depression signs and the Instagram usage time and the number of strangers followed by the user, was studied in Reference [84]. The authors found a marginal positive correlation between the Instagram hourly usage time and depressive symptoms, and the number of strangers followed slightly moderated this relationship. In other words, the depression signs increased with the Instagram usage time if the user followed a large number of strangers, but if the user followed fewer strangers, the Instagram usage time and depression signs were unrelated. Table 3 summarizes the recent social media-based depression detection schemes.