Abstract
Decisions about which items to keep, delete or replace shape the validity, precision and fairness of a scale, yet they are still often made with single statistics, such as the lowest discrimination parameter or the change in Cronbach’s alpha after deletion. This study proposes a multi-criteria framework for item decisions under Item Response Theory (IRT) and examines by simulation how single-criterion rules compare with it. Evidence is organised into six domains (information and targeting, model fit, local dependence and redundancy, differential item functioning, response categories, and content representation) and combined through a flag, investigate, compare and decide sequence. Counterfactual deletion and replacement are formalised as a purpose-weighted, constrained optimisation, and an Item Utility Profile is proposed as a decision aid rather than an index. A Monte Carlo study (500 replications) generated 60-item, six-domain two-parameter logistic pools containing locally dependent pairs and items labelled as functioning differently across groups. Five strategies for choosing 24 items were compared under two measurement purposes. Classical and slope-based rules each retained about four DIF-labelled items among the 24 chosen, and kept between 2.8 and 3.8 of six redundant pairs intact on average. An information-only rule left a content domain empty in 33% of replications when precision was needed near a screening threshold. The multi-criteria strategy kept about one DIF-labelled item, fewer than one redundant pair and every domain, at a cost of 13% to 17% of nominal weighted information. Conclusions. Item deletion is better handled as a documented measurement-design decision than as a statistical filter. The simulation is illustrative and uses known item parameters, so its findings need confirmation with estimated parameters and empirical data.