Abstract
We introduce the Probabilistic Multimodal Consistency Index (PMCI), a prototype measure of probabilistic semantic agreement used to quantify how much two modalities share in terms of mutually compatible affective evidence. PMCI does not model feature fusion or emotion classification directly; instead, each modality is mapped to a different probability distribution over learnable latent affective-agreement anchors, and the agreement of those distributions is quantified via the Jensen–Shannon divergence. Consequently, we propose PMCI not as a substitute for a discriminative model of pair matching, but rather as an auxiliary diagnostic index of cross-modal affective agreement. Experiments were conducted on pose-based facial and body keypoint sequences obtained from the RAVDESS and MELD datasets. In the updated RAVDESS setup, we extended the cache of preprocessed keypoint files to include all 24 actors, and we sampled four distinct pose-derived 12-frame crops per source video. This resulted in a total of 11,520 pose-derived windows, with the actor label determining the train/validation/test splits for the RAVDESS dataset. MELD contained 402 filtered pose-derived windows and served as an auxiliary in-the-wild validation setting with additional noise. When applying the updated RAVDESS standard pair-matching, DirectCosine_K0 achieved ROC–AUC = 0.943 and PR–AUC = 0.917, demonstrating that it is indeed the best exact pair-matching baseline and that PMCI-based configurations are not as accurate as DirectCosine_K0 when performing common exact face–body pair discrimination. Under this protocol, PMCI_K8, PMCI_K16, and PMCI_K32 produced ROC–AUC scores of 0.851, 0.805, and 0.887, respectively. Running a one-window-per-source-video experiment with 100 repetitions yielded similar results in the following order: DirectCosine_K0 = 0.942 plus-minus 0.009, PMCI_K16 = 0.807 plus-minus 0.016, and PMCI_K32 = 0.893 plus-minus 0.012, which shows that the results obtained were not the result of only repeated temporal crops. In the DirectCosine-mined hard negative setting, PMCI produced modest diagnostic separation above chance, where Direct+PMCI_K32 yielded ROC–AUC = 0.582. In a suite of shuffle control, permutation control, model initialization control, temperature control, anchor usage control, control latency, and pose perturbation control experiments, the diagnostic stability of PMCI was tested, confirming that PMCI is sensitive to both pose-estimation accuracy and domain shift. Finally, PMCI must be interpreted as a probabilistic diagnostic index of cross-modal affective agreement that is based on pose, not as a general-purpose emotion-recognition system.