The Impact of GenAI Feedback on Student Writing: A Mixed-Methods Study

Abstract

This mixed-methods study explored how master’s students engage with generative AI (GenAI) feedback during academic writing in higher education. It examined three aspects: students’ engagement with a custom-designed ChatGPT tool, changes in writing confidence after using GenAI, and how prior experience relates to perceived feedback quality. The ChatGPT tool was designed to promote dialogic feedback through reflective questioning. In total, N = 46 master’s students at a Dutch research university participated in the study through surveys, chatlogs, and semi-structured interviews. Quantitative analyses included paired-samples t-tests and regression analyses. Qualitative data were analyzed thematically. Results showed that while students’ confidence increased after using GenAI, most students engaged with the tool in a solution-seeking manner rather than participating in a Socratic dialogue. Prior experience with GenAI predicted more positive perceptions of feedback quality. However, overreliance on the tool was observed among some students. This reduced their critical engagement with the feedback. These findings highlight the importance of prompt literacy and active engagement with feedback. This study emphasizes the importance of guiding students in how to work with GenAI tools, focusing not only on writing support but also on strengthening their critical thinking and promoting self-regulated learning.