EMPAC: A Multimodal Dataset for Bridging Affective and Cognitive Empathy

Abstract

Empathy, a key element of social interaction, involves both cognitive and affective processes and is commonly investigated through measures such as empathic accuracy and affective physiological synchrony. While physiological synchrony offers a continuous measure of affective processes, empathic accuracy typically relies on discrete self-reports, leaving their temporal relationship largely unexplored. Advancing this line of research requires datasets that integrate time-continuous self-reports with physiological signals, yet such datasets--particularly those focusing on the empathizee--remain limited. To fill this gap, we present EMPAC (Empathy Measurement: Physiological, Affective, and Cognitive), a multimodal dataset constructed. To create empathy-eliciting stimuli, professional actors performed emotionally intense, pseudo-autobiographical narratives while their physiological signals (e.g., ECG, EDA) and continuous self-reported emotional states were recorded. We then conducted two observer experiments using these video recordings. In Experiment 1, to validate the stimuli as empathy-eliciting materials, observers continuously rated emotional intensity without being informed of the specific emotion portrayed, following the protocol of previous studies on time-series empathic accuracy. Yet this approach sometimes revealed a gap between the emotion category portrayed by the target and that perceived by the observers. In Experiment 2, we introduced a revised procedure in which the target emotion category was disclosed prior to viewing, revealing that specifying the target emotion led to a different relationship between individual empathy traits and empathic accuracy than observed in Experiment 1. EMPAC thus provides a rich, temporally aligned resource for investigating empathy dynamics in naturalistic settings and for evaluating methodological variations in empathic accuracy paradigms.