Evaluating Large Language Models for Post-Publication Promotion: A Blinded Comparative Study of Social Media Posts in Public Health

Bohdana Doskaliuk, Maidan Mukhamediyarov, Marlen Yessirkepov, Olena Zimba

Journal of Korean Medical Science, 2026

Full text Open access: Gold Metadata (CSL-JSON)
DOI 10.3346/jkms.2026.41.e168
Data confidence 85 high
How this score is calculated

It is built from checkable properties of the record itself: whether the article has a DOI, how many independent sources confirmed its data, whether the required fields are filled in, and how recent the latest confirmation is. Disagreement between sources and a missing identifier lower it. A value of 80 or above means the record was published without further checking; 60 to 79 means it is published at reduced confidence, with a manual check advisable. A record verified by an editor by hand is never pushed below 90 by a recalculation. This scores our data about the article, not the article itself.

Abstract

BACKGROUND: Social media platforms such as X (formerly Twitter) are increasingly used by journals, authors, and institutions to promote newly published research. Well-designed posts can enhance visibility, accelerate knowledge translation, and increase altmetric attention. However, creating accurate and policy-compliant content is time-intensive. Large language models (LLMs) offer a potential solution, yet systematic evaluations of their performance in post-publication promotion remain limited. METHODS: We conducted a blinded, crossed, offline evaluation of four LLMs: GPT-5 (OpenAI), Gemini 2.5 Pro (Google DeepMind), Grok-3 (xAI), and Perplexity Pro (Perplexity AI), tasked with generating X-style posts (≤ 260 characters) for 36 open access articles from The Lancet Public Health, The Lancet Planetary Health, and Annual Review of Public Health. Posts were generated using a standardized system and user prompt. A single blinded rater scored outputs using a five-domain rubric (factual accuracy, clarity, policy compliance, call-to-action quality, structure/metadata; maximum score 10). Secondary measures included character count, hashtag use, and readability (Flesch-Kincaid Grade Level). General linear models with Bonferroni-adjusted post hoc tests and non-parametric analyses were applied. RESULTS: < 0.001). Perplexity Pro scored highest for policy compliance, while GPT-5 and Gemini 2.5 Pro achieved superior structural scores. Readability varied: GPT-5 8.9 (7.3-9.2) and Perplexity Pro 7.3 (6.5-8.8) generated more complex outputs, whereas Gemini 2.5 Pro 5.1 (4.8-6.5) and Grok-3 4.5 (3.6-6.3) produced more accessible posts. CONCLUSION: LLMs can reliably generate accurate and policy-compliant social media posts for research promotion, with differences in style and readability that may inform audience targeting. GPT-5, Gemini 2.5 Pro, and Perplexity Pro produced high-quality outputs, while Grok-3 underperformed across several domains. These findings highlight the potential of LLMs as scalable first-draft tools for post-publication promotion, capable of improving the reach and accessibility of scientific research. Careful model selection, tailored to audience and communication goals, together with human oversight, remains essential.