Benefits of Multimodal Feedback during Hand Gesture-Based Interaction in Augmented Reality

Abstract

In this study, we explored potential benefits of adding multimodal feedback to hand-gesture interactions in augmented reality applications using see-through head-mounted displays. The benefits of adding simple haptic feedback from a vibrating wearable were explored as well. We revised the decomposition of the interaction based on a literature study and proposed a new interaction framework for improved mapping of feedback triggers to interactions and for better accommodating three-dimensional interactions. We set up an experiment consisting of small selection and translation tasks in which we tested various feedback combinations. Results showed that multimodal feedback has a strong indication of being beneficial. Haptic feedback, in our experiment transmitted by a fitness band placed on the participant's wrist, showed a similar trend with statistically significant benefits to the selection of virtual objects. Our feedback-implementations for providing aid in depth perception of virtual objects during interactions did not show clear results. Three-dimensional feedback resulted in objects being dropped closer to their target, while the trajectory-length was shorter when using feedback solely focused on the depth dimension. As not all results obtained by this study showed statistical significance, further research is required.