Alert: half wonk, half not wonk.
Last week I wrote about a new piece by Jesse Rothstein. Rothstein continued the argument that he’s been putting forth that value-added measures misstate teachers’ true contributions because they inadequately adjust for students’ learning ability. In particular, Rothstein questioned the results in recent work by Chetty, Friedman, and Rockoff. As a reminder, Rothstein found “evidence of moderate bias in VA scores.”
Chetty, Friedman, and Rockoff (CFR) have two new short pieces in which they argue against Rothstein’s criticisms. (Dan Goldhaber and Duncan Chaplin also show circumstances in which Rothstein’s method can cause false alarms.)
To oversimplify, Rothstein offers a “placebo test” in which he shows that measured teacher quality predicts prior student achievement. Since teacher quality can’t affect what a student learned in the past, the measures must still retain some element of a student’s past accomplishments. This is what value-added is intended to avoid and Rothstein offers this as evidence that the value-added adjustment is imperfect.
CFR respond that Rothstein’s test is picking up a mechanical artifact of the way value-added is constructed. The mechanical bias has two sources. The first source is that some teachers do switch grades and teach some of the same students for a second year. When this happens, the teacher’s ability really did have an effect on those students in the previous year. The second source of the artifact comes from the presence of noise in student test scores. CFR corrects for both issues and argues that the apparent failure of Rothstein’s placebo test disappears.
So here’s the half wonk part. Despite a serious professional disagreement, Rothstein and CFR share data and code with one another. Both sides are working hard to uncover the truth on a critically important issue in education research and education policy.
Now the not wonk half. No one thinks value-added measures are perfect. Parties interested in VAM for non-scientific reasons can be too quick to interpret “imperfect” as “not useful.” Neither CFR nor Rothstein think that VAM is perfect nor do they think that VAM is useless.