When Are Scoring Rules Proper? Bridging Theory and Practice in Survival Model Evaluation

Proper scoring rules encourage probabilistic predictions that match the true underlying distribution and are central to model evaluation, with increasing relevance in automated workflows such as AutoML. In survival analysis, however, their behavior under censoring is not fully understood. We study commonly used squared and logarithmic scoring rules for right-censored survival data under independent censoring, introducing a notion of marginal properness based on observable outcomes. Within this framework, we show that the SBS, evaluated at a fixed time point, along with its integrated version (ISBS) and the RCLL are strictly proper when all individuals eventually experience the event, but can become improper under finite follow-up or in the presence of cure fractions. For the SBS, we derive a closed-form expression that reveals the true mechanism: residual mass, corresponding to individuals who remain event-free at study end, systematically biases the score toward underestimating survival, with the effect increasing at later evaluation times and under heavier censoring. Through simulation experiments, we examine how these issues manifest in finite samples and under misspecification. The SBS exhibits pronounced improperness at late evaluation times and poor discrimination between models. The ISBS is more robust due to temporal integration but remains sensitive to tail regularity violations, exhibiting detectable improperness and reduced discriminatory power. The RCLL behaves consistently with strict properness and effectively separates misspecified models. Overall, our results demonstrate how theoretical improperness can translate into misleading model comparisons, underscoring the need for further methodological development in survival model evaluation under censoring and realistic data conditions.