Abstract
Guidance on handling missing item responses in educational assessments is drawn almost entirely from large-scale testing programs; however, much of the practical work occurs in single classes and small multi-class samples. This Monte Carlo study aimed to identify which strategies can be estimated at all at that scale, and how accurate they are once estimation succeeds. A complete-data benchmark and four missing-data strategies were evaluated across sample sizes from 40 to 200. The primary design examined four missingness mechanisms at a 15% missingness rate, and an additional 25% sensitivity condition was evaluated at selected sample sizes; each condition used 100 replications. The strategies were listwise deletion, a two-parameter logistic model fitted directly to the incomplete response matrix, multiple imputation followed by calibration, and a two-factor item response tree model. Estimability, not accuracy, separated the procedures. At 40 examinees, an admissible solution was obtained in 0.42 to 0.46 of replications with complete data, 0.19 to 0.23 for direct estimation, 0.02 to 0.07 for multiple imputation, and 0.00 to 0.05 for the item response tree model. The two-factor tree model showed severe estimation instability throughout the smallest sample-size region, and the same pattern persisted when data were generated from the tree model itself, ruling out misspecification as the sole explanation. Once estimation succeeded, the procedures differed by only a few hundredths of a theta standard deviation in root mean square error, while substantial individual-level uncertainty remained. Under the investigated conditions, direct estimation was more often estimable and showed a small accuracy advantage over the implemented imputation procedure in the adequately supported paired comparisons. At classroom scale, the choice of a missing-data strategy is primarily a question of estimability, and convergence and parameter stability should be reported whenever these models are applied.