The new Brookings report “Passing Muster: Evaluating Teacher Evaluation Systems” (authors: Steven Glazerman, Dan Goldhaber, Susanna Loeb, Stephen Raudenbush, Douglas O. Staiger, and Grover J. Whitehurst) offers a practical guide for moving forward on implementation of teacher evaluation systems. Teacher evaluations will never be perfect—right now they are very imperfect—so the authors explain how state-level policy makers can judge just how error-prone one district’s system is compared to another district’s.
I want to tell you to go right out and read the whole report. I especially want to tell you to read the report if you oppose teacher evaluation because Passing Muster is incredibly respectful of the pitfalls in existing evaluation systems. But, unless you’re a serious policy, I can’t tell you to read the whole thing.
The first half of Passing Muster is elegantly written and easily digested. The second half is also clear… if you have a PhD in a statistically oriented subject. So go read the first half. Before you do that you might well read Jay Matthews Washington Post column on Passing Muster. (Matthews thinks Passing Muster is too hard to understand and impractical.)
Let me take a swing at offering an extended, metaphorical explanation.
You’re a TV producer for the Food Network, dispatched to a small city with the assignment of identifying contestants for a new show “Top Kitchen Talent.” You can’t possibly evaluate every restaurant in town, let alone the individual chefs and sous chefs who staff them. So you’d like to turn to the restaurant reviewers who cover the food beat for the town paper, the local TV stations, and even a few foodie blogger sites.
The trouble is that the reviewers routinely hand out four and five star reviews for every meal that doesn’t end in ptomaine poisoning. Not much help. Going forward though, you’re willing to offer some Food Network cash to reviewers who adopt a meaningful review system.
If you want cooperation, you have to define “meaningful.” It turns out that there is historical data recording the number of repeat visits at each restaurant. A smart MBA has been able to link the numbers to who was in the back of the house cooking. Now given kitchen turnover, historical data doesn’t help you evaluate current cook-staff. But that same MBA has copies of the food reviewers notebooks. When she compares those notes to repeat visit data a couple things turn up. First, some of the reviewers noted whether the kitchen crew wore whites or checked pants. For all the tradition behind the sartorial choice, the MBA discovers that there’s no correlation between chef dress and customers coming back for another meal. In contrast, those chefs who go to the front of the house and count reservations each night definitely get more repeat customers. Unfortunately, reservation counts were only generally only available for dinner service. Finally, everyone has noted that chefs who taste each and every dish before it goes out do really well, even though no one can identify exactly what taste the chefs are looking for.
So, as a producer, here’s what you do. Your MBA hires a statistical consultant and together they set up a web calculator for the food reviewers. The calculator gives lots of points to reviewers who promise to check whether the chefs taste the food or count the house. (No points for reviewers who focus on the kitchen dress code.) And there’s a promise that if a particular reviewer has something different that also predicts well, you’ll revise the calculator to give points for that too.
Everyone understands how to get points, even though no one understands the math that justifies the formulas used in the calculators (except Alton Brown, of course.) Local reviewers grumble, but grudgingly they do start to come around.
All this is going to be pretty rough and reviews will contain mistakes for sure. But awarding of restaurant stars is going to be a lot more meaningful than before. And down the line, with a little luck, kitchen crews are going to start paying attention to the reviews and start churning out tastier meals.