Dyad-Specific Multimodal Affective Dynamics: Integrating Temporal Cross-Modal Consistency and Probabilistic Agreement

Yernar Seksenbayev, Saule Kudubayeva, Olga Salykova

Technologies, 2026

Full text Open access: Gold Metadata (CSL-JSON)
DOI 10.3390/technologies14090539
Data confidence 70 medium
How this score is calculated

It is built from checkable properties of the record itself: whether the article has a DOI, how many independent sources confirmed its data, whether the required fields are filled in, and how recent the latest confirmation is. Disagreement between sources and a missing identifier lower it. A value of 80 or above means the record was published without further checking; 60 to 79 means it is published at reduced confidence, with a manual check advisable. A record verified by an editor by hand is never pushed below 90 by a recalculation. This scores our data about the article, not the article itself.

Abstract

Repeated dyadic interactions are characterized not only by average multimodal signal levels but also by how cross-modal consistency evolves over time. This study investigated whether the joint temporal organization of within-person face–body cross-modal consistency and probabilistic cross-modal affective agreement, quantified by the Consistency Index for Affective Synchrony (CIAS) and the Probabilistic Multimodal Consistency Index (PMCI), yielded reproducible session-level dynamic representations associated with repeated dyadic interaction. The primary analysis included 24 improvised and 17 naturalistic sessions, with 12 dyads in each condition. Time-varying signals from both partners were summarized through dynamic-state occupancy, dwell structure, transition characteristics, and temporal variability and evaluated using relational permutation analysis with shared-participant and technical controls. In naturalistic interaction, repeated sessions of the same dyad were significantly more similar (β = −2.0180, FDR q = 0.0114), with significant dyad-specificity beyond shared-participant similarity (Δβ = −2.4670, q = 0.0033). Representation ablations showed that PMCI provided the dominant state-separation structure, whereas CIAS contributed modest and condition-dependent complementary information. Most notably, frozen participant-disjoint evaluation on 73 naturalistic sessions from six dyads and 11 previously unseen participants retained significant dyad-associated structure and achieved session-weighted nearest-dyad accuracy of 0.8356 and macro-dyad accuracy of 0.8512. Additional robustness analyses showed that the principal pattern persisted across state-count specifications, temporal-window settings, partner-order reversal, and explicit technical-quality adjustment. Overall, the findings support the existence of reproducible session-level multimodal dynamic structure associated with repeated dyadic interaction and its transfer to previously unseen participants, while not implying direct interpersonal synchrony or psychologically defined interaction states.