Abstract
This study compares two well-known big language models, ChatGPT and Gemma, to assess the quality of autonomously generated multiple-choice questions in Kazakh. We created questions using eight different configurations that varied by model, prompt language (English vs. Kazakh), source text language, and generation technique (direct vs. translation), all based on the historical book The Nomads by Ilyas Esenberlin. A 5-point Likert scale measuring fluency, clarity, and syntax was used to evaluate 48 questions from 577 Kazakh-speaking respondents in three surveys. The findings show that while ChatGPT slightly outperformed Gemma (3.95 vs. 3.86,$p=0.15$), Kazakh-prompted questions scored much higher than Englishprompted questions (4.00 vs. 3.81,$\mathrm{p} \leq 0.01$). Translation was inferior to direct generation (3.96 vs. 3.85,$p=0.10$). These results have significant implications for the development of educational technologies in the Kazakh language as well as lowresource language NLP applications.