A machine learning approach predicts future risk to suicidal ideation from social media data

Roy, A., Nikolitch, K., McGinn, R., Jinah, S., Klement, W., & Kaminsky, Z. A. (2020). A machine learning approach predicts the future risk to suicidal ideation from social media data. Npj Digital Medicine, 3(1), 1-12. https://doi.org/10.1038/s41746-020-0287-6

---

Summary

• Introduction
  The objective was to generate an algorithm termed "Suicide Artificial Intelligence Prediction Heuristic (SAIPH)" capable of predicting future risks to suicidal thoughts by analyzing publicly available Twitter data.

• Methodology
  The authors trained a series of neural networks on Twitter data queried against suicide, and associated psychological constructs (including burden, stress, loneliness, hopelessness, insomnia, depression, and anxiety.) Using tweets from suicidal ideation (SI) cases and tweets from controls, a random forest model was trained using neural network outputs to predict binary SI status.

tweet -- (10 models) --> [0 1 0 0 ... ] -- (random forest model) --> [0] or [1] SI

• Result
  The model predicted SI events with an AUC of 0.88. Using an alternative approach, the model generates a temporal prediction of risk such that peak occurrences above an individual specific threshold denote a ~7-fold increased risk for SI within the following 10 days.
  Also, the authors validated the model using regionally obtained Twitter data and observed significant associations of algorithm SI scores with county-wide suicide death rates.

• Conclusion
  Algorithmic approaches like SAIPH have the potential to identify individual future SI risk and could be easily adapted as clinical decision tools aiding suicide screening and risk monitoring.

---

Comments

Things to study more

• Previous study applying ML to identify suicidal tweets
  Du et al. have applied ML to generate a convolutional neural network capable of identifying suicidal tweets.
  Yet, the study has limitations in that its prediction of suicidal ideation requires someone to have written a suicidal tweet for the model to identify it.

  Du, J. et al. Extracting psychiatric stressors for suicide from social media using deep learning. BMC Med. Inf. Decis. Mak. 18, 43 (2018).

• What is the bootstrap aggregation?

  • Bootstrap aggregating, also called bagging (from bootstrap aggregating), is a machine learning ensemble meta-algorithm designed to improve the stability and accuracy of machine learning algorithms used in statistical classification and regression. It also reduces variance and helps to avoid overfitting.
  • Given a standard training set D of size n, bagging generates m new training sets D{i}, each of size n′, by sampling from D uniformly and with replacement. This kind of sample is known as a bootstrap sample.
  • Then, m models are fitted using the above m bootstrap samples and combined by averaging the output (for regression) or voting (for classification).

https://en.wikipedia.org/wiki/Bootstrap_aggregating#:~:text=Bootstrap%20aggregating%2C%20also%20called%20bagging,and%20helps%20to%20avoid%20overfitting.

• What is the model sensitivity and model specificity?
  Model sensitivity stands for "true positive rate" which is the probability of a positive test result conditioned on the individual truly being positive.
  Model specificity means "true negative rate" which is the probability of a negative test result, conditioned on the individual truly being negative.

• What is the inflection point of the model?
  Inflection points are points where the function changes concavity, i.e. from being "concave up" to being "concave down" or vice versa. They can be found by considering where the second derivative changes signs.

• What is the mean algorithm of the time series?
  As one of its attempts to predict SI events in temporal aspects to figure out when people are likely to be at risk, the researchers evaluated mean algorithm output data over a range of periods.
  The mean Model is that, for a time series that is independently and identically distributed (i.i.d — there is no trend and all observations have the same probability distribution and are independent from each other), the forecast at time t+1 is given by the mean of the historical data till time t.

• What is the M3Inference tool?

  • M3 is a deep learning system for demographic inference that was trained on a massive Twitter dataset. It features three major attributes:

    1) Multimodal: M3 takes both vision and text inputs. Particularly, the input may contain a profile image, a name (e.g., in the form of a natural language first and last name), a user name (e.g., the Twitter screen_name), and a short self-descriptive text (e.g., a Twitter biography).

    2) Multilingual: M3 operates in 32 major languages spoken in Europe, but note that these are not all "European" languages (e.g., Arabic is supported).

    3) Multi-attribute: Thanks to multi-task learning, the model can predict three demographic attributes (gender, age, and human-vs-organization status) at the same time.

https://github.com/euagendas/m3inference

• Existing suicide prevention models

  • Nuremburg model
  • Zero Suicide
  • OSPI

    Kohls, E. et al. Public attitudes toward depression and help-seeking: impact of the OSPI-Europe depression awareness campaign in four European regions. J. Affect. Disord. 217, 252–259 (2017).

• What is the Support Vector Machine (SVM) model?
  A support vector machine (SVM) is a type of supervised learning algorithm used in machine learning to solve classification and regression tasks; SVMs are particularly good at solving binary classification problems, which require classifying the elements of a data set into two groups.
  The key idea behind SVMs is to transform the input data into a higher-dimensional feature space. This transformation makes it easier to find a linear separation or to more effectively classify the data set. To do this, SVMs use a kernel function. Instead of explicitly calculating the coordinates of the transformed space, the kernel function enables the SVM to implicitly compute the dot products between the transformed feature vectors and avoid handling expensive, unnecessary computations for extreme cases.

https://www.techtarget.com/whatis/definition/support-vector-machine-SVM#:~:text=A%20support%20vector%20machine%20(SVM)%20is%20a%20type%20of%20supervised,data%20set%20into%20two%20groups.

• What is the sliding window approach?
  Window Sliding Technique is a computational technique that aims to reduce the use of nested loops and replace them with a single loop, thereby reducing the time complexity.

https://www.geeksforgeeks.org/window-sliding-technique/

Possible Future Research Ideas

• Idea of adding new features
  How the authors integrated the psychological theory into their model is very interesting. Leveraging this schema and yet adding new features could improve the model.
  One of the limitations of the study is that the emphasis is on the depressive and stress-based theories of suicide may represent a limited view that fails to capture additional risk trajectories leading to suicide.
  Possible future research ideas could be developed by adding some new features (e.g. linguistic feature analysis) to neural network models.

• Idea of collecting SI and control data
  By performing weekly queries for the term "I suicide thinking OR planning", the study has found users expressing suicidal ideation. Control tweets were gathered using the search term "I".
  Then, after collecting a dataset of millions of tweets, the study implemented a scanning strategy to first identify potentially suicidal tweets that would later be evaluated by a psychiatrist rater. If the tweet matches at least one of the word patterns in Supplementary Table 1, it was flagged and then be evaluated by a psychiatrist to confirm.
  Wouldn't there be a better way to extract data rather than just using the explicit search term?

  • A more robust control selection strategy may be collecting controls using the query "I think" on the same day as the collection of suicidal ideators.
  • Adding sentiment analysis considering related factors like passiveness to building datasets can help. (In the study, sentiment analysis was done using the textBlob package in Python.)

• Idea of integrating psychological theories into the model
  To offer nuanced and temporally sensitive tools to track and predict suicide ideation, the study centered on using ML constructs developed loosely around the interpersonal psychological theory for suicide, the hopelessness model, and associations of depression, anxiety, and insomnia with suicide risk. (Research by Kleimen et al., has demonstrated that efficacy for accounting for suicide ideation increases through integration of multiple theories of suicide.)
  It was realized by inferring the psychological weights of a range of constructs from the text using neural networks and using it as an input for a random forest model that predicts SI.

• Tools specialized for older individuals
  The model was found to be more efficacious at predicting outcomes in individuals of younger ages. In light of this, future studies seeking to generate social media AI tools for older individuals may benefit from a targeted representation of older users in the model training phase of the study.

• Idea of using a model specialized for imbalanced anomalies
  The study suggests that one promising approach warranting future study is the utilization of anomaly scores to model temporal rare events in imbalanced data.

Questions

---

Things to note

• Interesting way to validate the model using local statistics
  The model was validated using local statistics. The authors analyzed the association of algorithm SI scores of regionally obtained Twitter data with county-wide suicide death rates.

• Interesting way to validate the model using real-world cases
  The researchers validated that their method has relevance to actual suicidal behaviors. They analyzed Twitter profiles of persons known to have died by suicide to assess personalized risk prediction.

• Why ML techniques are avoided sometimes
  The researchers opted to avoid alternative machine learning techniques such as RNNs or LSTM models. That is because machine learning methods rely on two major assumptions including data representativeness of what is to be learned and that the various data distributions involved do not change over time.

• The potential of predictive tools
  Were the medical community to rely on diagnostic tests alone, even those with promising performance metrics would likely add an intolerable burden to the healthcare system. As such, the implementation of such tools as decision aids would give gatekeepers additional insight. Novel predictive tools may enable the identification of risk in individuals during a period prior to the progression of more severe suicidal behavior.

• How the study validated the data of certain psychological construct
  The neural networks constructed were designed to read a text and infer psychological weights across a range of constructs including stress, loneliness, burdensomeness, hopelessness, depression, anxiety, and insomnia. While neural networks were trained from the tweets of people referencing query words related to these concepts such as "burden", the researchers sought to validate their ability to reflect these constructs. To that end, they adapted psychometrically validated psychological scales by binary coding question content in relevant scales where the original responses were amenable to such adaptation, such as in true vs. false responses.

• Understanding temporal prediction methods is hard.

• The more I study, the more I feel I need to study more.