Towards Real-Time Intraoperative Recognition of the Recurrent Laryngeal Nerve (RLN) in Robot-Assisted Minimally Invasive Esophagectomy (RAMIE)

Abstract

Background Recurrent laryngeal nerve (RLN) injury is a serious complication of Robot-Assisted Minimally Invasive Esophagectomy (RAMIE) that may lead to vocal cord paralysis and aspiration. Automatic anatomy recognition using deep learning (DL) could improve intraoperative orientation and nerve identification, potentially reducing RLN palsy and facilitating surgical training. However, Artificial Intelligence (AI)-based RLN recognition during complex thoracic surgery has not been extensively explored.

Objective To develop and evaluate a DL algorithm for automatic recognition of the RLN in thoracoscopic video frames from RAMIE procedures.

Methods Video recordings of McKeown RAMIE procedures were retrospectively collected at the University Medical Center Utrecht (UMCU) and manually annotated to outline RLN. These annotations served as the reference for model training and evaluation. A transformer-based MetaFormer-FPN segmentation model was trained on the annotated videoclips in collaboration with the Eindhoven University of Technology (TU/e). Four training sets of increasing size and an additional left-RLN-only subset were analysed to assess the influence of dataset size and composition on model performance. The trained model’s performance was then evaluated against the manual annotations using the Dice coefficient, Normalized Surface Dice (NSD), Average Symmetric Surface Distance (ASSD), and Hausdorff 95 distance (HD95).

Results A total of 24 annotated videoclips were used to create four training sets (14, 17, 20, and 24 videoclips) and one left-RLN-only subset. The model achieved mean Dice coefficients up to 0.27 for 24 videoclips and 0.30 for the left-RLN-only subset. Compared with the datasets 14 and 17 videoclips, the Normalized Surface Dice increased in the datasets 20 and 24 videoclips from 0.13 to 0.17 (0.21 for left RLN), recall rose from 0.24 to 0.33, and ASSD decreased from 27 to 24 pixels. The additional consistency measures yielded more stable performance across runs.

Conclusion DL enables automatic RLN recognition in RAMIE videoclips with improving performance as dataset size increases. Although accuracy remains modest, model stability and temporal consistency indicate feasibility for intraoperative real-time recognition. Expanding annotated datasets will be essential to achieve clinically applicable precision.