Measuring Performance-based Emotional Intelligence through an AI Chatbot

Abstract

The purpose of this study was to explore the plausibility of indirectly measuring performance-based emotional intelligence (EI) through an AI chatbot. A series of structured interview questions tapping into four facets of EI were developed and deployed into an AI chatbot, which include: (a) identifying/sensing others' emotions, (b) utilizing their own emotions, (c) understanding emotional contexts, and (d) managing emotions. Textual features from the interview scripts were extracted through natural language processing and were used as predictors of EI scores based on situational judgment tests (SJTs). The training sample consisted of full-time employees (n = 725) recruited from Prolific, who engaged with the AI-chatbot for about 40 minutes and then completed a series of SJT items that measure the four EI facets. The test sample (n = 118) consisted of an independent sample of full-time employees, also recruited from Prolific, who underwent the same procedure as those in the training sample but completed additional measures including a 16-item cognitive ability test, a personal intelligence measure, a life satisfaction scale, and job performance measures of counterproductive work behavior and organizational citizenship behavior. Results indicated that machine-inferred EI scores exhibited good split-half reliability, reasonable convergent and discriminant validity, acceptable nomological validity, overall reasonable generalizability, and criterion-related & incremental validity. Study limitations, practical implications, and future research directions are discussed.