BACKGROUND AND OBJECTIVES: Chart reviews are frequently used for research, care assessments, and quality improvement activities despite an absence of data on reliability and validity. We aim to describe a structured chart review methodology and to establish its validity and reliability.

METHODS: A generalizable structured chart review methodology was designed to evaluate causes of morbidity or mortality and to identify potential therapeutic advances. The review process consisted of a 2-tiered approach with a primary review completed by a site physician and a short secondary review completed by a central physician. A total of 327 randomly selected cases of known mortality or new morbidities were reviewed. Validity was assessed by using postreview surveys with a Likert scale. Reliability was assessed by percent agreement and interrater reliability.

RESULTS: The primary reviewers agreed or strongly agreed in 94.9% of reviews that the information to form a conclusion about pathophysiological processes and therapeutic advances could be adequately found. They agreed or strongly agreed in 93.2% of the reviews that conclusions were easy to make, and confidence in the process was 94.2%. Secondary reviewers made modifications to 36.6% of cases. Duplicate reviews ( = 41) revealed excellent percent agreement for the causes (80.5%-100%) and therapeutic advances (68.3%-100%). κ statistics were strong for the pathophysiological categories but weaker for the therapeutic categories.

CONCLUSIONS: A structured chart review by knowledgeable primary reviewers, followed by a brief secondary review, can be valid and reliable.