Abstract
As false news benchmarks are increasingly being developed from fact-checking sources, this introduces a slight, but under cited, threat to validity: source bias. The two classes are not necessarily similar in style or structure; a classifier could distinguish between fake and real items (both from a fact-checking outlet) based on surface features rather than truth, and obtain scores which do not reflect the ability of the classifier to actually detect the truth. We report this effect for Kazakh–Russian Telegram—a low-resource code-switched environment—and show that a naively engineered corpus yields artificially high performance (up to 0.98 F1) that plummets by about ten percentage points, after the removal of the shortcut. We then introduce a two-stage debiasing protocol: (1) structural cleaning (viral-boilerplate, interrogative openers and verdict labels removal from Tengrinews posts); and (2) a heterogeneous two-source real class constructed from Tengrinews posts and confirmed “True” (Pravda) verdicts on Factcheck.kz. This balanced, cleaned set of 804 posts (402 real/402 fake) is made available to the public. On this debiased benchmark we report honest baselines for six methods: TF-IDF + Logistic Regression, FastText, fine-tuned XLM-RoBERTa, fine-tuned RuBERT, and Llama 3.1 8B with zero- and few-shot prompting. The fine-tuned transformers show good performance (RuBERT 0.8819, XLM-RoBERTa 0.8757 macro F1), whereas classical TF-IDF is competitive (0.8508), and in-context LLM prompting lags behind (zero-shot 0.4522, few-shot 0.6601 macro F1)–quantifying the gap between in-context prompting and few hundred labeled examples. The transformer margin is very small and is calculated from a single split, so we only report the margin as indicative and discuss the evaluation conditions required to confirm the margin. We make a methodological contribution by providing a replicable debiasing recipe and showing that explicit source-bias controls are required to evaluate trustworthy fact-checks in low-resource, fact-check-derived misinformation benchmarks. Data and code are made publicly available so that they can be used to facilitate reproducible research.