Development and Evaluation of a Diagnosis and Triage Healthcare Chatbot

Abstract

This thesis consists of three parts: a systematic review of evaluations of non-mental healthcare chatbots, the development of a triage and diagnosis chatbot, and the evaluation of the chatbot. The systematic review provides an overview of chatbots in various healthcare contexts and highlights commonly used outcome variables used to evaluate the chatbots. The chatbot was built by leveraging Azure Health Bot service and related Azure services. The evaluation was carried out by 44 participants, whose demographics were skewed towards lower age and higher education. The chatbot was evaluated with a questionnaire, consisting of questions related to outcome variables commonly used to evaluate chatbots, namely user experience, chatbot usage, health behavior, user characteristics, and system quality. The chatbot performed well on user experience, but improvements related to personality and error handling are necessary. The health behavior of the participants after using the chatbot was found to be inconclusive. The analysis of meta-data of the chatbot usage showed that the conversation was relatively quick, but with a high drop-off rate.