Back in April I wrote about the American Statistical Association Statement on Using Value-Added Models. I wrote in part,
Some of what the ASA says sounds ever-so-sensible, but reflects a failure to understand statistical models.
Raj Chetty, John Friedman, and Jonah Rockoff (CFR) released their own assessment of the ASA statement in May. They make me think I was too nice to the ASA. CFR’s main point is simple,
the … statement provided no references to the literature. Hence, it is unclear whether the ASA’s statement fully incorporates the results of recent research that addresses many of the concerns it raises.
CFR politely point out that the ASA statement somehow missed a good deal of published research…including some published by the ASA. CFR make it clear that the American Statistical Association should be embarrassed at having released a rather unscientific statement. (“Embarrassed” is my word, not CFR’s).
CFR, writing from the economist’s perspective, sum up the situation very clearly.
While there is no doubt that VA measures are not perfectly reliable, the question is how reliable they are relative to other potential methods of evaluation. … The fact that classroom observation and VAM are both imperfect measures underscores why educators and policymakers are likely to make better decisions if they are based on multiple measures of job performance rather than any stand-alone metric.