Multilingual QA-RAG: Evaluating LLMs' Contradiction Handling in English and Kazakh

2025

Closed access Metadata (CSL-JSON)
DOI 10.1109/snpd65828.2025.11254178
Data confidence 70 medium
How this score is calculated

It is built from checkable properties of the record itself: whether the article has a DOI, how many independent sources confirmed its data, whether the required fields are filled in, and how recent the latest confirmation is. Disagreement between sources and a missing identifier lower it. A value of 80 or above means the record was published without further checking; 60 to 79 means it is published at reduced confidence, with a manual check advisable. A record verified by an editor by hand is never pushed below 90 by a recalculation. This scores our data about the article, not the article itself.

Abstract

This study investigates how large language models (LLMs) respond to factual contradictions under varying prompt styles in both English and Kazakh—a low-resource language. We evaluate the performance of DeepSeek-R1-Distill-Qwen-7B, and GPT-4o-mini on a dataset of 100 question– context pairs with deliberately introduced factual distortions using Retrieval-Augmented Generation (RAG) and Zero-shot approaches across strict, standard, and weak prompt types. Results show that DeepSeek outperforms in English language, particularly in zero-shot settings, as well as it exhibits higher variability and sensitivity to prompt strength. GPT-4o-mini demonstrates more accurate performance across Kazakh datasets. Qualitative analysis reveals distinct model behaviors, such as blind acceptance of falsehoods, expression of uncertainty, and ethical refusals. The findings highlight the challenges LLMs face in contradiction detection and emphasize the need for robust solutions to improve factual reliability.