Evidence of methodological bias in hospital standardised mortality ratios: retrospective database study of English hospitals

Abstract

OBJECTIVE: To assess the validity of case mix adjustment methods used to derive standardised mortality ratios for hospitals, by examining the consistency of relations between risk factors and mortality across hospitals. DESIGN: Retrospective analysis of routinely collected hospital data comparing observed deaths with deaths predicted by the Dr Foster Unit case mix method. SETTING: Four acute National Health Service hospitals in the West Midlands (England) with case mix adjusted standardised mortality ratios ranging from 88 to 140. PARTICIPANTS: 96 948 (April 2005 to March 2006), 126 695 (April 2006 to March 2007), and 62 639 (April to October 2007) admissions to the four hospitals. MAIN OUTCOME MEASURES: Presence of large interaction effects between case mix variable and hospital in a logistic regression model indicating non-constant risk relations, and plausible mechanisms that could give rise to these effects. RESULTS: Large significant (P<or=0.0001) interaction effects were seen with several case mix adjustment variables. For two of these variables-the Charlson (comorbidity) index and emergency admission-interaction effects could be explained credibly by differences in clinical coding and admission practices across hospitals. CONCLUSIONS: The Dr Foster Unit hospital standardised mortality ratio is derived from an internationally adopted/adapted method, which uses at least two variables (the Charlson comorbidity index and emergency admission) that are unsafe for case mix adjustment because their inclusion may actually increase the very bias that case mix adjustment is intended to reduce. Claims that variations in hospital standardised mortality ratios from Dr Foster Unit reflect differences in quality of care are less than credible.