Abstract
The rapid advancement of Large Language Models (LLMs) has created unprecedented opportunities for transforming educational program design. However, the lack of standardized evaluation frameworks raises concerns regarding the content quality and pedagogical effectiveness. This study proposes EduDesignEval, a comprehensive evaluation framework designed to assess LLM-generated educational content across multiple dimensions of quality. The framework defines three metrics: groundedness for measuring factual accuracy and hallucination mitigation, coherence for evaluating logical flow and thematic integrity, and fluency for assessing linguistic naturalness and readability. These metrics are synthesized into a unified score that provides an adaptable, weighted assessment based on specific educational requirements. We first validated the EduDesignEval framework through metric definitions and demonstrated its utility by employing ten publicly available LLM models across two critical tasks: text generation for learning goals and course descriptions, and multi-label classification for professional standards and competency frameworks, using a dataset derived from over 8,200 educational programs across 150 Kazakhstani institutions. Our fine-tuned model has demonstrated promising zero-shot classification performance with high groundedness while maintaining balanced coherence and fluency, outperforming larger models utilizing one-shot prompting. Preliminary results show that big models excel in groundedness, while smaller models show limitations in classification due to redundancy and low coherence. EduDesignEval bridges gaps in literature by providing multi-level quality control, fostering human-Artificial Intelligence (AI) collaboration, and promoting equitable, adaptive education. Ultimately, this framework ensures reliable LLM integration, enhancing program design efficiency and pedagogical integrity for global educational systems.