Labeling and segmentation of treepoint cloud in immersive virtual reality

Abstract

Effective data annotation is crucial for the advancement of machine learning techniques for autonomous labeling of point cloud datasets for environmental studies and other fields. Traditional point cloud annotation workflows utilize 3D datasets on a 2D screen, which degrades the user experience and negatively impacts performance. Concepts like embodied digital twins and virtual reality may offer an alternative approach for annotating and exploring datasets in 3D. However, there is a lack of research on determining which annotation approaches offer the most optimal solution for labeling and segmenting components of individual tree point clouds. Additionally, only a few studies refer to users’ evaluations of usability and workload and compare different annotation approaches, as well as examine the influence of participants individual differences. To address this gap, game engine technologies were used to recreate a virtual annotating environment and an interactive experience to capture the users’ experience and performance on two commonly used annotation approaches and their combinations. The first annotating method, namely the Sphere Selection approach, offered a 3D pointer with adapting shapes for free-style detailed point selection. The second method, namely the Container Creation approach, offered a solution based on control points and nodes for polygon creation, offering a quick approach for massive point selection and annotation. The combination of the two methods formed the third approach, namely, the combination of the two approaches. The results indicate that the sphere selection approach is optimal in terms of time and accuracy for annotating components of individual trees. Participants reported that this approach was perceived as more effective, enjoyable, and the least physically and mentally demanding annotation technique compared to the other two. The findings of this research highlighted the potential of each approach for future environmental applications for labeling 3D point cloud data in virtual reality. The findings of this research contribute to field data annotation and highlight the potential for immersive technologies in environmental studies. Future researchers can focus on further investigation of the potential of each annotation approach for labeling and segmenting individual tree point clouds and further investigation of participant’s individual differences on performance and usability. Finally, the findings of this thesis could be a starting point for the design criteria for 3D VR tree annotation tools and applications and the participant application usability study.