Large Language Models as support for reflective conversations

Abstract

Large Language Models (LLMs), such as ChatGPT, are gaining popularity both in practical usage and academic research. Recent advancements in these models have enhanced their capability to produce human-like responses, thereby expanding their possible applications. This development presents new opportunities for engaging in `conversations' with artificial intelligence (AI), where it could possibly support users in reflective conversations; conversations aimed at gaining clarity and insights on a past (challenging) experience. This research investigated to what extent Large Language Models (LLMs) can support users in reflective conversations. To this end, we conducted between-subject mixed-method user studies, comparing the effects of interaction modalities - written (text) versus vocal (audio) conversations. Upon observing a significant positive effect on insight in the audio condition, we further explored the differences between reflecting on a past positive challenge and a current challenge, finding that participants preferred the latter. Our key finding indicates that, in its current state, LLMs can effectively lighten the cognitive load and steer conversations, thus supporting reflective conversations. However, we identified certain limitations in LLMs their ability to understand context, and offered cautionary remarks regarding the application of LLMs in reflective conversations, and the public perception of the 'intelligence' of AI.