Look at this picture of 8th grade math scores according to the amount of daily homework.
Note that students in the first column are running roughly two years behind students in the second column. Now I have no idea what’s the “right” amount of math homework to assign in 8th grade. But I bet it ain’t zero.
Is the relation between homework and learning causal, or is it just that lousy classes also happen to be places where no one can bother to assign homework?
One of the fears (one of my fears anyhow) about standardized testing is that standardization will drive out all the neat and important stuff that isn’t easily measured. Here’s some indirect evidence that at least the arts haven’t been noticeably diminished.
Basmat Parsad and Maura Spiegelman put together the numbers on arts instruction in 2009-2010 as compared to a decade earlier. This covers a period with a large increase in standardized testing and is–mostly–before the Great Recession started wiping out all sorts of budgets. Here’s the secondary school picture. You can see that arts instruction is pretty much unchanged.
The story is about the same at the elementary level. There has been a drop-off in specialized dance and drama instruction, although there never was a whole lot of this in grammar school.
(Figures are from Arts Education in Public Elementary and Secondary Schools 1999-2000 and 2009-10.)
Mike Petrilli makes a point and raises a question about progress in teaching math in Education Next and Fordham’s Flypaper.
One of the great mysteries of modern-day school reform is why we’re seeing such strong progress (in math at least, especially among our lowest-performing students) at the elementary and middle school levels, but not in high school.
Here’s Petrilli’s picture from Education Next.
Petrilli then asks:
Could it be that increased graduation rates are driving down twelfth-grade performance? Recent studies have indicated that graduation rates are up significantly over the past decade; that means that we have twelfth-graders in school today who previously would have dropped out. And those students are likely to be very low-achieving. Could they be pulling down the mean? Just like we see with the SAT as more students—and more lower-income students—take the exam?
I’m not a statistician but it seems plausible to me. Number-crunchers out there: What say ye?
Let’s begin by looking at the premise that there are a lot more students staying through 12th grade. Here’s a picture I made that shows the number of 12th grade students in public school as compared to the number in 8th grade four years earlier.
You can see that Petrilli is right on the premise that we have a more kids staying into 12th grade. It’s about a 7 percentage point increase…which is a lot.
Now let’s do ye old number crunching. Turns out that if you combine my little chart above with the information Petrilli has already supplied then the answer is easy. The biggest effect of selection bias–that’s the term econometricians use for the kind of mixing of apples and oranges that Petrilli asks about–would be if all the increase in 12th graders came from students in the bottom of the score distribution. Since we have a 7 percentage point increase in enrollment we’d expect the lowest decile of scores to plummet. Look at the first graph. If anything, the bottom decile scores have improved slightly.
So there’s a clear answer to Petrilli’s pondery–nope. Too bad too; it would have been nice to find that the flat math test scores are just a statistical artifact.
Activists know that schools are massively underfunded. Economists know that simply pouring money into schools doesn’t much lead to better outcomes. On a good day there’s room for agreement that spending money the right way can be a really big deal. (On a bad day, the source of the disagreement is that economists actually look at the data carefully.)
Sorting out the effect of across-the-board higher spending is awfully difficult because for the most part higher spending in a district means that the parents are wealthier. Children of wealthy parents do better in school. Classic case of “correlation doesn’t imply causality.”
Leslie Papke and Jeff Wooldridge came up with a clever way to work around this entanglement. Papke and Wooldridge looked at what happened in Michigan in the late 1990s after the state funding formula was changed. While they used fancy statistical techniques, the heart of the argument is that a number of school districts received a large financial boon from the funding reform. We know that improvements in these districts are really due to the increase in spending because that was the only really big change. The authors looked specifically at the fraction of students passing Michigan’s fourth grade math test.
Findings? The statistics are clear, increased spending raised pass rates. But not by a very large amount. A ten percent increase in spending raised pass rates by about 4 percentage points. A good outcome, but not one that points to a solution to the nation’s education woes.
Michael Jones (a Notre Dame PhD student heading for the University of Cincinnati) has some new results on the effects of performance pay on teacher behavior. His answers are interesting in their own right, and also illustrate how hard it is to get definitive answers. Essentially, Jones looked at the difference in teacher behavior in districts that have some kind of performance pay versus teacher behavior in districts that don’t.
The raw answer is nothing happens to teacher hours. But when Jones controls for whether teachers might select whether to work in a district that offers performance pay, he finds that the effect of putting in performance pay is to lower teacher in-school work effort by 6 hours a week. That’s a lot.
Except when Jones does the same analysis for Florida (Florida teacher incentives depend only on individual performance while school-level performance sometimes counts in other states), teachers put in an extra 13 hours a week. So the estimated Florida positive is twice the size of the estimated negative elsewhere in the country.
If this sounds like a pretty muddled result so far, let me throw out out more factor for your consideration. Which districts adopt performance pay–maybe the ones with the biggest worries about teachers slacking? Are the nation-wide results just illustrating reverse causality?
Sometimes even good science doesn’t give us clear answers.
Most research suggests that objectively observable characteristics of teachers, things like SAT scores, don’t contribute a whole lot toward predicting who will be a good teacher. Will Dobbie has found that characteristics measured by Teach For America do say quite a bit about which teachers do well.
TFA collects quite a bit of information about its candidates. Dobbie ran test scores for New York City school kids against the characteristics of their TFA teachers. A high score on the teacher’s combined TFA quality measure predicts a large improvement in math outcomes for that teacher’s students. It was harder to identify an effect on English test performance.
Of the several characteristics TFA measures the three that are important for math are
Some school districts do a good job of identifying the best applicants for teaching jobs, but many districts don’t. It would be worth districts spending some money to buy into the TFA evaluation process and to use it for all applicants. Or maybe TFA should be hired to act as a clearing house?
The issue of how well financial incentives paid to students work to boost academic achievement is by no means settled. Perhaps some incentives work well and others don’t. Looking at a number of experiments, Uri Gneezy, Stephan Meier, and Pedro Rey-Biel summarize what we know as of now:
The results of these experiments are somewhat disappointing: given the relatively small effect sizes it is not clear that these programs represent the best return on investment.
Over the last several years Roland Fryer has run experiments in Chicago, Dallas, and New York in which students were paid for various aspects of academic achievement. The experiments are carefully designed so that there are both control groups and experimental groups, just like in medical trials. Fryer sums up the results in an article in the November Quarterly Journal of Economics:
I estimate that the impact of financial incentives on student achievement is statistically zero.
Eric Bettinger reports on a long-running study in which the Coshocton, Ohio school district pays students $15 for scoring “proficient” on each of five Ohio state tests. The reward is boosted to $20 for “advanced” or “accelerated” scores. Half of students in grades 3-6 are eligible for the payment, half aren’t. Which grade levels in which schools are eligible is chosen randomly each year.
The bottom line is that the incentives led to notable improvements (0.15 standard deviations) in math scores, but didn’t make any difference in reading, social science, or science. Bettinger points out that the program is very cheap. So this appears to have been a remarkably effective way to raise math scores.
By the way, this advance in both science and how much math Coshocton kids know is due to the generousity of a local philanthropist. Thanks go out to Robert Simpson and the Simpson Family Foundation.
Bob Roseth says: But … is this a good idea? This philosopher would say no: http://www.theatlantic.com/magazine/archive/2012/04/what-isn-8217-t-for-sale/8902/. I tend to think he’s correct, but perhaps as a society we’ve fought and lost the battle of keeping certain areas out of the market.
I’ve been curious about whether we’re going to have to hire many more teachers as current teachers retire. According to the best projections I can find, the answer is quite a few more. Probably not an overwhelming increase though.
My source is the Department of Education’s Projections of Education Statistics to 2020. Two caveats. First, the projections are purely statistical and assume no policy changes. Second, even though the projections were made in Fall 2011 the last year for which we have actual data is 2008. The near term projections might be missing any effects of the Great Recession.
Fair points. But applying ocular statistics to Petrilli’s graph it looks like the drop has been equal at higher percentiles. Maybe even greater. Without having thought through your math, wouldn’t that suggest a pretty minimal effect?
I don’t think we can dismiss the selection story so quickly. You have assumed that there is a perfect correlation between test scores and latent graduation propensities. But that can’t be right — my guess is that the correlation is relatively weak. That would mean that the effect of the changing selection would be spread throughout the distribution, not just at the very bottom. Combine that with a hypothesized rightward shift in the latent, unselected distribution and it seems to me that Petrilli’s figure could well be consistent with a selection story.
I’m too lazy to take the idea to data, but it seems to me there are two ways to do it. First, you could assume that the latent 12th grade distribution shifted rightward by the same amount as was observed for the 8th grade distribution, and ask whether any feasible correlation between selection and test scores could generate the observed selected data in Petrilli’s figure. Second, you could use the NELS to estimate the correlation between test scores and continuation probabilities, then apply those to the changing selection seen in your second graph to estimate the trend in the unselected distribution.
I don’t have any idea what either would show. My hunch is that selection will account for a meaningful portion, though not all, of the slowdown between 8th and 12th grades.