Mike Petrilli makes a point and raises a question about progress in teaching math in Education Next and Fordham’s Flypaper.

One of the great mysteries of modern-day school reform is why we’re seeing such strong progress (in math at least, especially among our lowest-performing students) at the elementary and middle school levels, but not in high school.

Here’s Petrilli’s picture from Education Next.

Petrilli then asks:

Could it be that increased graduation rates are driving down twelfth-grade performance? Recent studies have indicated that graduation rates are up significantly over the past decade; that means that we have twelfth-graders in school today who previously would have dropped out. And those students are likely to be very low-achieving. Could they be pulling down the mean? Just like we see with the SAT as more students—and more lower-income students—take the exam?

I’m not a statistician but it seems plausible to me. Number-crunchers out there: What say ye?

Let’s begin by looking at the premise that there are a lot more students staying through 12th grade. Here’s a picture I made that shows the number of 12th grade students in public school as compared to the number in 8th grade four years earlier.

You can see that Petrilli is right on the premise that we have a more kids staying into 12th grade. It’s about a 7 percentage point increase…which is a lot.

Now let’s do ye old number crunching. Turns out that if you combine my little chart above with the information Petrilli has already supplied then the answer is easy. The biggest effect of selection bias–that’s the term econometricians use for the kind of mixing of apples and oranges that Petrilli asks about–would be if all the increase in 12th graders came from students in the bottom of the score distribution. Since we have a 7 percentage point increase in enrollment we’d expect the lowest decile of scores to plummet. Look at the first graph. If anything, the bottom decile scores have improved slightly.

So there’s a clear answer to Petrilli’s pondery–*nope.* Too bad too; it would have been nice to find that the flat math test scores are just a statistical artifact.

Fair points. But applying ocular statistics to Petrilli’s graph it looks like the drop has been equal at higher percentiles. Maybe even greater. Without having thought through your math, wouldn’t that suggest a pretty minimal effect?

I don’t think we can dismiss the selection story so quickly. You have assumed that there is a perfect correlation between test scores and latent graduation propensities. But that can’t be right — my guess is that the correlation is relatively weak. That would mean that the effect of the changing selection would be spread throughout the distribution, not just at the very bottom. Combine that with a hypothesized rightward shift in the latent, unselected distribution and it seems to me that Petrilli’s figure could well be consistent with a selection story.

I’m too lazy to take the idea to data, but it seems to me there are two ways to do it. First, you could assume that the latent 12th grade distribution shifted rightward by the same amount as was observed for the 8th grade distribution, and ask whether any feasible correlation between selection and test scores could generate the observed selected data in Petrilli’s figure. Second, you could use the NELS to estimate the correlation between test scores and continuation probabilities, then apply those to the changing selection seen in your second graph to estimate the trend in the unselected distribution.

I don’t have any idea what either would show. My hunch is that selection will account for a meaningful portion, though not all, of the slowdown between 8th and 12th grades.