Abstract
Abstract What role should large language models (LLMs) play in social science research? The rapid rise of LLMs has generated both excitement and trepidation about their functional and ethical implications. This study evaluates their role in political science data production through the Universal Periodic Review Other Stakeholders’ Engagement (UPROSE) Dataset Project. The UPROSE project uses Universal Periodic Review reports from Non-Governmental Organizations on human rights conditions to build data on human rights and NGO engagement with international organizations. We compare LLM performance to human coders across multiple extraction tasks using established reliability metrics. LLMs perform comparably to humans for well-structured, surface-level variables but fall short in tasks requiring contextual inference or interpretive judgment. These results suggest that LLMs are best used as conditional tools that, when paired with subject-matter expertise, can expand access to high-quality data and reduce structural barriers. We conclude with best practices for hybrid workflows that enhance efficiency while maintaining interpretive quality in social science research.