Abstract
Abstract
Simulation models are widely used to support counterfactual inference in settings where direct experimental validation is difficult or infeasible. However, for phenomenological models calibrated to observational time series, goodness-of-fit does not necessarily establish that the structural assumptions underlying such inferences are uniquely supported by the data. Here, we examine this issue using a published COVID-19 transmission model for Japan that incorporates empirical inputs including human mobility, variant prevalence, and vaccination and has been used to estimate the effects of alternative vaccination scenarios. Using the publicly archived code together with a reconstructed substitute for a restricted-access input file required by the released implementation, we first reproduced the published model behaviour. We then performed two complementary computational stress tests. First, the vaccination-related input was suppressed and the remaining parameters were re-estimated against the original epidemic trajectory. Second, as a complementary test of trajectory-level flexibility, the model was refitted to synthetic target series with qualitatively different temporal structures while retaining the empirical mobility and variant inputs. The model retained substantial fitting capacity in both tests: suppression of the vaccination-related input caused only a limited deterioration in fit to the original trajectory, while qualitatively different synthetic trajectories could also be reproduced through parameter adjustment. Variation in fitted parameters across these conditions was also consistent with compensatory parameter behaviour. These results do not imply that vaccination had no effect or that the original model is invalid. Rather, they provide empirical evidence that, in this case, goodness-of-fit to the observed epidemic trajectory alone does not uniquely support the structural interpretation required for vaccination-related counterfactual inference. The study illustrates the value of reproducible computational stress testing as a complement to, rather than a substitute for, formal identifiability or model-comparison analyses.