Published Wednesday, January 21, 1998

Report cards for teachers: Can computers measure the art of teaching?

Steve Berg / Star Tribune

KNOXVILLE, TENN. -- Of all the influences in a child's life -- poverty or race or family circumstance -- the quality of teaching stands out as the most important factor in academic progress.

Indeed, a series of poor teachers in the early grades may prevent a child from ever catching up to similar kids who have had better teaching.

That's the essence of a growing stream of evidence collected at the University of Tennessee -- evidence making teachers across the country nervous because it challenges the common excuse that kids from poor backgrounds are beyond help.

The bearer of this news is biostatistician William Sanders.

Professor William Sanders of the University of Tennessee at work in his office surrounded by piles of statistical information.

For the better part of two decades, Sanders has been poring over Tennessee's student achievement numbers. Now, he has emerged as an influential figure on the school reform landscape. In a sense, he's a bad teacher's worst nightmare.

A blend of brains and rural charm, Sanders, 55, is widely known for devising a mathematical model to identify good and bad teachers based on the progress their students make from year to year.

Using computers and statistical techniques to filter out the external influences in a child's life, Sanders isolates what happens in the classroom, arriving at what he calls "teacher effects."

"What we've consistently found, starting back in the early '80s, is that when compared to class size or the ethnicity of students, or whether they're on free or reduced-price lunches, all of those things pale in comparison to the effectiveness of the individual classroom teacher," Sanders said recently.

Moreover, drawing on 6 million standardized test profiles from Tennessee's 885,000 public schoolchildren, he has found that the impact of good or bad teaching accumulates.

Take one third-grader who gets poor teachers for three years and a similar one who gets excellent teachers. Sanders says that by sixth grade, their standardized test scores can differ by as many as 50 percentile points, a gap he pronounces "astounding." Getting the right teachers, he says, is "life's ultimate lottery."

In a nation desperate to improve the competence of its young people, Sanders' statistical methods are among the most exhaustive and his findings among the most controversial. Sanders, perhaps more than anyone, places teachers on the hot seat. His model is one of an array of attempts nationwide to hold teachers more responsible for the performance of students.

"Accountability isn't new," said Chris Pipho, of the Denver-based Education Commission of the States, listing such past efforts as teacher tests and merit pay. "What's new is tying teachers closely to student outcomes on standardized tests. People are seeing kids still failing these tests in the 11th grade, and they're asking, 'Who's responsible?' "

Increasingly, the answer seems to be teachers.

While it remains difficult to fire teachers in most states, Colorado and New Mexico claim to have abolished tenure, making dismissals easier. Florida has cut in half the time it takes to fire a teacher. North Carolina has lengthened the teacher probation period from three years to four.

Texas has begun using student test scores as part of teacher evaluations, although an attempt to link pay to student scores failed. A few districts -- among them Douglas County, Colo., and the Chicago suburb of Niles, Ill. -- pay bonuses to teachers based on student performance.

But nowhere is teacher accountability more heavily practiced than in Tennessee.

In 1994 the state Education Department began publishing school-by-school comparisons statewide based on gains students were making on standardized tests. In 1996 teachers of grades two through eight began receiving yearly reports showing how much their students progressed compared with expectations -- and compared with other kids in the school system and the state.

Principals also began receiving comparison reports on teachers, thus exposing strengths and weaknesses on their faculties -- again, based not on the raw level of student achievement, but rather on their progress as measured against expectations.

It's this value added that Tennessee schools emphasize when evaluating teachers -- largely because of Sanders' influence.

'Value added'  

Until a spring day in 1982, Sanders had been engrossed in the arcane world of biostatistics, mostly on agriculture-related topics. Then he read a news story that changed everything for him.

It asserted that teaching could not be measured. Sanders thought otherwise. He was convinced that the mixed-model statistical measurements he used in science could be applied to education. His views attracted then-Gov. Lamar Alexander's attention and grew into a movement that became the Tennessee Value-Added Assessment System, the state education agency he now directs from his campus office.

The heart of what Sanders attacked was the assumption that measuring teachers was impossible because of wide differences in children -- that poverty, racism or illiterate parents give some children such deficits that their teachers cannot be held accountable for their failures.

He set out to use mixed-model statistics to weigh these and other external influences -- including IQ and past test performance -- in order to predict the academic gain a child could be expected to make in a year. The next step was to add in the impact of a teacher on those expected gains.

It's the gain that's most important, Sanders contends, believing that it's less important to compare the raw scores of poor kids in the inner city with those of wealthy kids in the suburbs than it is to compare each with expected gains.

This is a controversial view. It runs counter to the notion that standards, not gains, are most important. Even if low-starting kids make gains every year, they still may never reach a high enough standard to be considered competent.

But Sanders argues that small gains will have a better chance of adding up to greater achievement in the long haul. In any case, he continues to find that, while affluent children generally score higher, low-income kids often make impressive gains.

What surprised him most, however, was not differences among schools but among teachers. "The variability among teachers is tremendously bigger than among schools," he said. "And it's only when I look at the very top teachers do I see all children making gains above expectation. These teachers are somehow dealing with the diversity within their classrooms, and everyone is making progress.

"My disappointment to date is that we haven't been learning enough from those teachers," he continued.

As for the teachers who rank low, "my suspicion is that a lot of them don't know why they're ineffective or even if they're ineffective," he said. "But they're sincere, conscientious people, most of them. Teachers are human beings, not pegs on a board. Yes, there's probably some innate talent that makes some better than others, like being good at hitting a fastball. But I can't believe the gap can't be narrowed significantly."

His advice for administrators: Get help for sub-par teachers. And make sure kids don't get inferior teachers two years in a row.

Critics take aim  

Given the pressure Sanders has placed on teachers, it's no surprise that his critics range from the skeptical to the hostile.

"Bill Sanders is a fine man, but that doesn't keep him from being wrong," said Al Mance, assistant director of the TennesseeEducation Association, the state's largest teachers union.

Mance doesn't quarrel with Sanders' value-added approach but rather with his assertion that statistical methods can accurately measure the quality of learning and teaching. "People love numbers. I love numbers. It would be great if we could rate everything 1 through 10. But we're dealing with human systems that don't fit into such a neat pattern."

TennesseeEducation Commissioner Jane Walters also has clashed with Sanders on several fronts. Last year she tried to eliminate funding for the reports that zero in on teachers, arguing that schoolwide scores were sufficient.

That attempt failed. But she succeeded in removing second-grade teachers from scrutiny. "Many second-graders don't yet read well enough to be evaluated on multiple-choice tests," she said.

Initially, teachers worried that they would be judged solely on the gains their students made on standardized tests. Indeed, the state comptroller's office was suspicious enough about Sanders' methods that it commissioned an outside audit.

Results were mixed. Two world-class statisticians endorsed Sanders' methods. A testing expert dissented, saying that while Sanders was able to identify the very best and worst teachers, judging those in the vast middle was too tricky. All three emphasized in their report that Sanders' system should be one of a number of measures "within a web of accountability."

Sanders agrees. And he has emphasized that his model is intended less as a reward or punishment than as a diagnostic tool for pinpointing ineffective teachers in hopes that principals will find ways to lessen their damage -- perhaps with mentors or team teaching.

Over the past two years, anxiety among teachers has dissipated.

"It's hard for any human being to measure another human being," said Libby Burney, who teaches eighth-graders at Whittle Springs Middle School in Knoxville. "But as a math teacher, I probably have more sympathy for statistical evaluation than most other teachers. In some ways that's more assuring than relying on an administrator who might do something drastic based on misinformation."

Burney credits Sanders for recent improvements in her school's math scores. His reports showed alarmingly low gains for kids in the already-troubled Whittle Springs area, where, she said, "a lot of kids need a warm bath and a warm meal." Consulting the numbers, the school decided that its low scores had less to do with bad teaching than with devoting insufficient time for disadvantaged kids to absorb math. After 40 minutes were added to math classes, the rate of progress doubled.

While the high math of Sanders' statistical model is beyond the grasp of ordinary citizens, many seem to have latched on to the thrust of his thinking.

The annual publishing of school-by-school progress scores in major Tennessee newspapers draws considerable attention from parents and educators.

And Sanders is seeing his influence spread beyond his home state. Dallas schools recently completed a study that matches his findings. Minneapolis uses some of his value-added techniques to assess the teaching of reading. A recent report by the National Commission on Teaching and America's Future seemed generally in tune with his thinking, concluding: "What teachers know and can do is the single most important influence on what students learn."

Criticism from urban school systems generally has subsided.

When dramatic achievement turnarounds occur, it's often in the inner city, and most often because of a forceful principal. "Usually it's a black woman who grits her teeth, decides there'll be no excuses for racism or poverty, assembles a team and a bootstrap program and, by sheer will, those schools seem to gain momentum and begin to succeed," Sanders said.

The 'shed effect'  

Test scores have edged upward slightly in all subjects except reading since Sanders' value-added approach became Tennessee law in 1992.

For that, Sanders takes only mild satisfaction. Instead, he worries about something he calls the "shed effect," so named because of the shape of a graph he uses to describe its meaning.

It shows the lowest-scoring fifth of kids making the biggest academic gains, with the highest making the smallest gains. To Sanders, this suggests that teachers are concentrating their efforts on the bottom kids while leaving those at the top unchallenged. The result, he says, is a dumbing down of the classroom experience for kids with the most promise.

"It does no good to hold some kids back so the others can catch up," he said, adding that he'd rather see everyone moving up together. "The educational community has been beat over the head for so long to do something about low-achieving kids. And that's fine. In terms of gain, they're doing quite well. But we're also seeing the severe under-education of the above-average child."

This is happening disproportionately to talented black kids in inner-city schools, he says. And it's related to another worry: a slowdown in academic progress as kids head into middle school.

While some experts blame the natural diversions of adolescence, Sanders suspects that teachers are placing so much emphasis on early learning that they fail to recognize the importance of sustaining academic progress in the middle years.

It's the topic of his next statistical foray.

© Copyright 1998 Star Tribune. All rights reserved.

 

Copyright 1998, 199, 2000, 2001  by David N. Shearon