The main tool used by economists to measure the impact of a teacher is the value-added score, or VAM. Loosely, we see how much the test score of a typical student of a particular teacher improves over the course of a year. The idea is that teachers aren’t penalized for students who start out way behind, because they’re only judged on the students’ improvement.
In a series of papers, Jesse Rothstein has documented a hole in this logic. Students differ not only in how much they know at the beginning of a year, they also differ in how teachable they are. In other words, some students are likely to learn less than others over the course of a year irrespective of what the teacher does.
In particular, Rothstein’s latest paper shows again that to some extent students get sorted amongst teachers according to “teachability.” This means that some teachers have bigger hurdles than others if they wish to achieve a high VAM score.
This sort of bias matters for research on teacher impacts. More importantly, nowadays an increasing number of teachers are being assessed in part on value-added scores.
Rothstein adds to the reasons to be nervous about the use of VAM. That’s not the same as suggesting VAM is not at all useful. Rothstein finds “evidence of moderate bias in VA scores.” He goes on to say,
my results are sufficient to re-open the question of whether high-VA elementary teachers have substantial causal effects on their students’ long-run outcomes…
the variance of the permanent component of student sorting bias is between 11% and 25% of the variance of teachers’ true effects.
…policies that use VA scores as the basis for personnel decisions may be importantly confounded by differences across teachers in the students that they teach, though the problem is not likely to be as severe as would be implied by the worst-case scenarios consistent with prior evidence. Teachers who have unusual assignments may be rewarded or punished for this under VA-based evaluations. This will limit the scope for improving teacher quality through VA-based personnel policies.