| Download | - View final version: ChatGPT for suicide risk assessment on social media: quantitative evaluation of model performance, potentials and limitations (PDF, 1.1 MiB)
|
|---|
| Author | Search for: Ghanadian, Hamideh; Search for: Nejadgholi, Isar1; Search for: Al Osman, Hussein |
|---|
| Affiliation | - National Research Council of Canada. Digital Technologies
|
|---|
| Format | Text, Article |
|---|
| Conference | 13th Workshop on Computational Approaches to Subjectivity, Sentiment, & Social Media Analysis (WASSA 2023), July 14th, 2023, Toronto, Ontario |
|---|
| Abstract | This paper presents a novel framework for quantitatively evaluating the interactive ChatGPT model in the context of suicidality assessment from social media posts, utilizing the University of Maryland Reddit suicidality dataset. We conduct a technical evaluation of ChatGPT’s performance on this task using Zero-Shot and Few-Shot experiments and compare its results with those of two fine-tuned transformer-based models. Additionally, we investigate the impact of different temperature parameters on ChatGPT’s response generation and discuss the optimal temperature based on the inconclusiveness rate of ChatGPT. Our results indicate that while ChatGPT attains considerable accuracy in this task, transformer-based models fine-tuned on human-annotated datasets exhibit superior performance. Moreover, our analysis sheds light on how adjusting the ChatGPT’s hyperparameters can improve its ability to assist mental health professionals in this critical task. |
|---|
| Publication date | 2023-07-14 |
|---|
| Publisher | Association for Computational Linguistics |
|---|
| Licence | |
|---|
| In | |
|---|
| Language | English |
|---|
| Peer reviewed | Yes |
|---|
| Export citation | Export as RIS |
|---|
| Report a correction | Report a correction (opens in a new tab) |
|---|
| Record identifier | 4d659f67-ceea-44ee-92ef-59f6290af830 |
|---|
| Record created | 2023-08-09 |
|---|
| Record modified | 2023-08-10 |
|---|
