| dc.description.abstract | Professional Military Education (PME) faces persistent challenges in distance learning, including high attrition, inconsistent engagement, and limited individualized support. This study examined whether a voluntary, low-touch AI chatbot assistant was associated with differences in student performance, engagement, learner perceptions, and satisfaction in military distance education. The intervention consisted of an AI literacy module embedded within the existing Canvas-based orientation course and designed around the CATALYST Framework, which integrates Bloom’s 2 Sigma Problem, cognitive load theory, andragogy, and the Community of Inquiry model. Unlike intensive intelligent tutoring systems that provide continuous adaptive feedback, this research evaluated a low-touch, student-initiated approach that minimizes disruption to the PME curriculum while offering scalable, personalized support.
A quantitative research design was used to analyze data from 177 consenting learners enrolled in Airman Leadership School and Noncommissioned Officer Academy distance learning orientation courses. Student performance was assessed using pre-test and post-test scores, while a post-course survey measured learner perceptions related to usefulness, ease of use, engagement, and satisfaction. A mixed-design analysis of variance (ANOVA) examined changes in performance over time and differences based on optional chatbot use. Hierarchical multiple regression examined whether learner characteristics and chatbot use predicted post-test performance, and independent-samples t tests compared learner perceptions between chatbot users and non-users.
Only 10 participants, or 5.6% of the sample, used the optional chatbot. Student scores increased from pre-test to post-test; however, the change was not statistically significant, and optional chatbot use was not associated with significantly different performance outcomes. The regression model did not significantly predict post-test performance, and no statistically significant differences were identified between chatbot users and non-users across the four learner-perception constructs. Students nevertheless reported generally positive perceptions of the learning experience, and greater AI familiarity was positively associated with perceived usefulness, ease of use, engagement, and satisfaction.
This research advances AI-enhanced learning theory while offering actionable insights for military educators. The findings suggest that providing access to AI support alone may be insufficient to generate meaningful adoption or measurable learning effects; however, the small number of chatbot users limited the study’s statistical power and should be considered when interpreting nonsignificant differences between chatbot users and non-users. Successful implementation of AI within PME distance education may depend on learner readiness, intentional instructional integration, proactive support, and structured opportunities for engagement. | en_US |
