CRITERION-GUIDED AI ARCHITECTURE FOR LEARNING SUPPORT AND PRELIMINARY STUDENT ASSESSMENT

С.Ж. Жакыпбеков, Artem A. Bykov, V. A. Yеrmakova, Regina Sharshova

Scientific Journal of Astana IT University, 2026

Full text Open access: Gold Metadata (CSL-JSON)
DOI 10.37943/lsgm5046
Data confidence 70 medium
How this score is calculated

It is built from checkable properties of the record itself: whether the article has a DOI, how many independent sources confirmed its data, whether the required fields are filled in, and how recent the latest confirmation is. Disagreement between sources and a missing identifier lower it. A value of 80 or above means the record was published without further checking; 60 to 79 means it is published at reduced confidence, with a manual check advisable. A record verified by an editor by hand is never pushed below 90 by a recalculation. This scores our data about the article, not the article itself.

Abstract

This paper presents a pilot evaluation of a criterion-guided human-in-the-loop architecture for an intelligent educational system that supports learning and preliminary assessment of students’ written and oral responses. Instead of assigning a final score directly, the system combines relevant context, formalized criteria, interpretable response features, supporting evidence, uncertainty estimates, and instructor verification. The architecture separates probabilistic components, including retrieval, automatic speech recognition, natural language processing, and a language model, from deterministic assessment and governance mechanisms such as rubric weights, aggregation rules, thresholds, audit logging, and referral rules. Two modes were implemented: a retrieval-augmented generation assistant for formative learning support and an AI examiner for preliminary criterion-based assessment. In the assistant pilot, 336 of 347 queries were processed without recorded technical errors, giving a completion rate of 96.8%; source citations were included in 89.0% of responses, and median response time was 980 ms. Student perceptions of the examiner were evaluated using 52 questionnaires on a five-point Likert scale. Mean ratings were highest for question clarity at 4.81, usefulness of criterion-level feedback at 4.46, and sufficiency of response time at 4.42. Students also reported notable uncertainty about the grade, with a mean rating of 3.79, and concern about speech-recognition errors, with a mean rating of 3.31. Among 45 oral responses, 12 (26.7%) lacked usable word-level timestamps and were referred for review. A worked case demonstrated traceability from response features and evidence to criterion scores, uncertainty handling, aggregation, and instructor verification. A preliminary comparison of 23 paired AI and instructor-released scores was treated as diagnostic because of a severe ceiling effect. The results support technical feasibility and procedural acceptability, but not yet the validity or reliability of automated assessment.