The Effect of Prompt Construct Relevance on the Validity of Linguistic-Based Psychological Trait Scores

Abstract

A tacit assumption in both AI-based and traditional assessment research and practice is that prompts must be tailored to the constructs being measured. The present study challenges this assumption by examining whether and the extent to which predictive models (PMs) and large language models (LLMs) can infer psychological traits from chatbot transcripts generated by prompts designed to assess a different construct. Participants from a previous vocational interest chatbot study (n = 133) completed self-report measures of personality, intellect, and emotional intelligence. Their chatbot responses were then scored for these constructs using established PMs and LLMs. Results showed that vocational interest prompts produced linguistic-based scores for personality, intellect, and emotional intelligence with split-half reliability comparable to scores derived construct-relevant prompts. For personality and intellect, construct-irrelevant prompts also yielded convergent and discriminant validity comparable to, and in some cases stronger than, construct-relevant prompts. However, this pattern did not extend to emotional intelligence when using predictive models. Large language models performed comparably to predictive models for personality and intellect but were better able to infer emotional intelligence from vocational interest prompts and showed stronger overall discriminant validity. Responses generated from vocational interest prompts also produced personality scores with factorial validity comparable to those generated from construct-relevant prompts. Overall, this study provided initial evidence for the possibility that AI-based assessments may be able to use a single set of prompts to infer multiple psychological traits, potentially improving the efficiency and scalability of language-based assessments.