The American Statistical Association (ASA) has adopted a statement on the use of value-added models (VAM). Basically, the ASA says that VAM can be useful, but that it has problems too. I doubt there are any VAM researchers who will disagree, but perhaps a statement from the ASA will serve as a needed warning to policy makers.
Some of what the ASA said makes sense:
VAMs are generally based on standardized test scores, and do not directly measure potential teacher contributions toward other student outcomes.
- Everyone worries–or at least everyone should worry–that VAM values what we can measure instead of measuring what we value. At the same time, I don’t know of any scientific evidence that teachers who produce good test scores do poorly on other student outcomes.
Some of what the ASA says sounds ever-so-sensible, but reflects a failure to understand statistical models:
Most VAM studies find that teachers account for about 1% to 14% of the variability in test scores, and that the majority of opportunities for quality improvement are found in the system-level conditions.
- In fact, VAM studies rarely speak to “opportunities for quality improvement.” VAM measures the effect of changing teacher ability, class size, family background, etc. Measuring the results associated with various inputs says nothing whatsoever about the cost or feasibility of changing those inputs. So nothing at all about “opportunities.” (And yes, the ASA should know better…And yes, I’m a member of the ASA.)
Regardless of the extent to which you think the ASA got the science right, what’s depressing is that having a rational, nuanced, public discussion of the science of VAM sometimes seems hopeless. Obviously taking great care in its wording, the ASA wrote:
This statement by the American Statistical Association provides guidance as to what can and cannot be reasonably expected, given current knowledge and experience, from use of VAMs. It is intended to enhance general understanding of the strengths and limitations of the results generated by VAMs and thereby encourage the informed use of these results. It is not meant to be prescriptive or advocate any particular VAM specification or promote or condemn specific uses of VAM.
which the Washington Post reported as “Statisticians slam popular teacher evaluation method.” Sigh.