From Recommendations to Decisions: Assessing Explainable AI for Time Management in Educational Settings

Abstract

While explainable AI (XAI) promises improved decision-making through transparency, emerging research questions its universal benefits, particularly under temporal constraints where cognitive load theory and flow theory predict explanations may disrupt rather than support performance. This study examined whether AI recommendations improve or harm learning performance under time pressure in educational gaming, and whether Miller’s evaluative AI (EXAI) paradigm better preserves user autonomy than traditional XAI approaches. It was hypothesized that time pressure would reverse typical XAI bene- fits due to interruption costs, with EXAI showing superior subjective outcomes by main- taining user agency. A between-subjects experiment (N= 94) using PuzzlePath, an 8- minute educational puzzle game, compared traditional XAI (statistical explanations), EXAI (user-driven evaluation), Control (minimal assistance), and no-recommendation conditions. Bayesian Knowledge Tracing provided adaptive recommendations while measuring objective performance (completion rates, accuracy), subjective experience (control, time management, engagement), and behavioural patterns. Results revealed a Recommendation Paradox: participants without recommendations achieved 42.4% com- pletion versus 13.1% with recommendations (p= 0.0021, OR= 4.88), with success rates higher when ignoring (94.7%) versus following (88.5%) recommendations. EXAI signifi- cantly outperformed traditional XAI on perceived control (Mdn= 3.40 vs 2.80, p= 0.010) and time management (Mdn= 3.20 vs 2.40, p= 0.009), though both underperformed the no-recommendation baseline, with qualitative evidence indicating flow disruption in 65% of XAI participants. These findings establish time pressure as a influential boundary con- dition for XAI effectiveness and demonstrate that evaluative approaches better preserve user autonomy, highlighting the necessity of objective performance metrics in educational AI evaluation.