Methodological Gaps in Predicting Mental Health States from Social Media: Triangulating Diagnostic Signals

Sindhu Kiranmai Ernala, Michael L. Birnbaum, Kristin A. Candan, Asra F. Rizvi, William A. Sterling, John M. Kane, and Munmun De Choudhury. 2019. Methodological Gaps in Predicting Mental Health States from Social Media: Triangulating Diagnostic Signals. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (CHI '19). Association for Computing Machinery, New York, NY, USA, Paper 134, 1–16. https://doi.org/10.1145/3290605.3300364

---

Summary

• Introduction
  Obtaining clinically valid diagnostic information from sensitive patient populations is challenging. Thus, a growing body of research has operationalized characteristic online behaviors as "proxy diagnostic signals" for building models to predict the mental health states of individuals.
  This Paper posits a challenge in using these proxy diagnostic signals revolving around their lack of clinical grounding, theoretical contextualization, and psychometric validity.

• Methodology
  Focusing on three commonly used proxy diagnostic signals derived from social media, the authors test the validity of the predictive models built on these data.

  • Three proxy diagnostic signals were used :
  1. Affiliation Data
     -> behaviors signaling affiliation (e.g. following, hashtag usage) to mental health resources
  2. Self-report Data
  3. Clinically Appraised Self-report Data
     -> External expert appraisals on social media data

• Result
  Although the models showed strong internal validity, they suffered from poor external validity when tested on mental health patients. A deeper dive reveals issues of population and sampling bias, as well as uncertainty in construct validity inherent in these proxies.

• Conclusion
  The study discusses the methodological and clinical implications of these gaps and provides remedial guidelines for future research.

---

Comments

Things to study more

• What is data triangulation theory?
  "Data triangulation is the use of a variety of data sources, including time, space, and persons, in a study. Findings can be corroborated and any weaknesses in the data can be compensated for by the strengths of other data, thereby increasing the validity and reliability of the results."

• Interesting previous studies using social media for mental health research
  Prior research includes deciphering social support provisions to promote positive mental health outcomes, and exploring how these platforms can support intervention delivery.

  • Decipher social support provision
    • Munmun De Choudhury and Emre Kiciman. 2017. The Language of Social Support in Social Media and Its Effect on Suicidal Ideation Risk.In ICWSM. 32–41.
    • Eva Sharma and Munmun De Choudhury. 2018. Mental Health Support and its Relationship to Linguistic Accommodation in Online Communities. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. ACM, 641
  • Explore how platforms can support intervention delivery
    • Becky Inkster, David Stillwell, Michal Kosinski, and Peter Jones. 2016. A decade into Facebook: where is psychiatry in the digital age? The Lancet Psychiatry 3, 11 (2016), 1087–1090

• What kind of statistical approach did they adopt to test the homogeneity?
  The study adopted a statistical matching approach to ensure that the control users and the individuals in each of the proxy datasets are comparable by trait attributes. What kind of statistical approach did they adopt?

  • Donald B Rubin. 1986. Statistical matching using file concatenation with adjusted weights and multiple imputations. Journal of Business & Economic Statistics 4, 1 (1986), 87–94

• What is the linguistic equivalence tests?
  The study conducted linguistic equivalence tests between the Twitter data and the Facebook data to combine two data sources in building their datasets. It is said that this is a known approach in the transfer learning literature.

  • Eric H Huang, Richard Socher, Christopher D Manning, and Andrew Y Ng. 2012. Improving word representations via global context and multiple word prototypes. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics: Long Papers-Volume 1. Association for Computational Linguistics, 873–882.

• Evaluation metrics of ML

  1) Precision :
  Precision is a metric that gives you the proportion of true positives to the amount of total positives that the model predicts. It answers the question “Out of all the positive predictions we made, how many were true?”
  
  2) Recall :
  Recall focuses on how good the model is at finding all the positives. Recall is also called true positive rate and answers the question “Out of all the data points that should be predicted as true, how many did we correctly predict as true?”
  
  3) F1 score :
  F1 Score is a measure that combines recall and precision. As we have seen there is a trade-off between precision and recall, F1 can therefore be used to measure how effectively our models make that trade-off.
  
  4) Accuracy :
  Accuracy answers the question “Out of all the predictions we made, how many were true?”
  
  5) Area Under the ROC Curve (AUC – ROC) :
  An ROC curve (receiver operating characteristic curve) is a graph showing the performance of a classification model at all classification thresholds. This curve plots two parameters: True Positive Rate, and False Positive Rate.

  AUC stands for "Area under the ROC Curve." AUC provides an aggregate measure of performance across all possible classification thresholds.

  reference: 
  - https://www.labelf.ai/blog/what-is-accuracy-precision-recall-and-f1-score
  - https://developers.google.com/machine-learning/crash-course/classification/roc-and-auc#:~:text=AUC%3A%20Area%20Under%20the%20ROC,to%20(1%2C1).

• Recent approaches to overcome the issue of dataset shift
  The study suggests adopting recent approaches from the machine learning field to overcome the issue of dataset shift.
  (Data shift refers to a phenomenon that statistical data distributions are drastically different between the proxy datasets and the actual patient datasets, and it could be revealed by the population and sampling biases.)

  • Including importance weighting of training instances based on similarity to the test set
    • Hidetoshi Shimodaira. 2000. Improving predictive inference under
      covariate shift by weighting the log-likelihood function. Journal of
      statistical planning and inference 90, 2 (2000), 227–244.
  • employing online learning of prediction models to identify and recover from incorrect predictions 16 54
    • Nicolo Cesa-Bianchi and Gábor Lugosi. 2006. Prediction, learning, and games. Cambridge university press.
    • Himabindu Lakkaraju, Ece Kamar, Rich Caruana, and Eric Horvitz. 2017. Identifying Unknown Unknowns in the Open World: Representations and Policies for Guided Exploration.. In AAAI, Vol. 1. 2.

Possible Future Research Ideas

• Would the result be replicated when using different types of mental illnesses, other than schizophrenia?
  The paper used the schizophrenia patient data to test the validity of predictive models based on proxy signals by the data triangulation method. It surfaced a lack of external validity, issue of construct validity, limited theoretical underpinning, and population and sampling biases.
  Would the result be replicated when using different types of mental illnesses?

• Validation of these findings in Korean culture
  The study was based on data written in English only. I believe it would be valuable to validate these findings in other cultures too.

Questions

• Interpretation of small absolute value of feature importance
  Compared to top features across the Affiliation, Appraised Self-Report and Patient Model, feature importances denoted by beta weights (significant at the p=0.05 level) of the patient model have significantly small absolute values. What does this possibly imply?

  

• What actions could be taken to tackle the opposition to the validity of using "proxy signals"?
  The paper concluded that what proxy classifiers actually learn is the language use of individuals actively opening up about schizophrenia experiences, and seeking informational and emotional support on Twitter. It is comparable to the patient population data who did not exhibit such disclosure or support-seeking behavior on social media.
  Reading the paper, I felt like the study is almost questioning the validity of using social media data to predict mental health status. What are the possible actions that could be taken next?
  Or should the proxy signals be all abolished because of their methodological and clinical limitations?

---

Things to note

• The goal of this paper is not to be dismissive of the immense potential that lies in social media data. Rather, the key takeaway of the paper is that using patient data to build machine-learning models is imperative.