Does Theory Drive the Items or Do Items Drive the Theory?
At the core of the argument-based approach to test validation as it has been presented by Kane (1992, 2004, 2006) is a relatively simple premise: test validity is demonstrated by linking the score that is observed from a test instrument to the use of that score for some subsequent inference. Details, however, are not so simple: How does one craft a coherent interpretive argument and systematically collect evidence to support that argument? The application of the argument-based approach to validation presented in the University of Michigan papers is framed by what the authors are introducing as two amendments to Kane's approach: (1) A clearer distinction between assumptions and inferences in the formation and evaluation of the interpretive argument; and (2) Breaking up the interpretive argument into what the authors describe as elemental, structural and ecological pieces. Each of these pieces is then associated with specific methods for gathering the evidence needed to support a validity argument. The University of Michigan group views the lack of distinction between assumptions and inferences, and the lack of prescriptiveness in the formation and evaluation of interpretive arguments as short-comings of Kane's argument-based approach to validity. In the view of this article's authors, both of these perceived short-comings are symptoms of an apparent disconnect between test instrument design and validation.
Talbot, R.M., & Briggs, D. (2007). Does theory drive the items or do items drive the theory? Measurement Interdisciplinary Research and Perspectives, 5(2-3), 205-208.