From Small Talk to Smart Talk: Development of an AI Chatbot to Infer Intellect and Cognitive Ability

Abstract

This thesis sought to develop an AI chatbot that can infer participants’ intellect and cognitive ability scores. Cognitive ability has been a construct of interest for over 100 years, with links to many important outcomes of interest to organizations. The personality trait intellect has received less attention in literature, but through its connection to both personality and cognitive ability, offers a new area of research. The machine-learning models were developed using a training sample of 457 employed adults recruited from Prolific and a test sample of 224 college students. These participants completed a chatbot (for the training sample and part of the test sample) or Qualtrics interview (the second part of the test sample) and a Qualtrics survey. The interview included general interview questions as well as some relating to intellect and openness and the Qualtrics survey consisted of measures relating to intellect and cognitive ability. Participants in the test sample also reported their cumulative GPA and completed a self-reported measure of college adjustment. This thesis assessed the psychometric properties of machine-inferred intellect and cognitive ability scores. Results indicated that machine-inferred scores (a) had good split-half reliability, (b) demonstrated acceptable convergent and discriminant validity at the overall trait level, (c) showed acceptable convergent validity but poor discriminant validity at the facet level, (d) exhibited moderate criterion-related validity with some outcomes, and (e) showed incremental validity over questionnaire-derived scores in some analyses. This thesis opens the door for better measurement of these constructs, as well as better connecting them with theoretically relevant constructs.