Wednesday, March 22, 2006

Merit Pay Based on Test Scores

In Florida, Governor Jeb Bush is pushing a proposal to relate teacher's pay to student's test scores. Teachers will get a bonus according to the degree to which students that they are teaching are able to improve their standardized test scores over the course of the school year.

Imagine that you were creating a new basketball team. Unlike the other teams, you decide to pay players solely on the basis of the highest grade they obtained while they were in school and the number of years they have spent on your team.

I guarantee that this would be a recipe for a basketball team guaranteed to sit at or near the bottom of the standings for eternity. Indeed, those games that other teams play against such a team would simply be considered scrimmage games – an opportunity to practice. In the same way, American schools that follow this model, when compared to schools that make an effort to distinguish good from poor teachers and provide an incentive for teachers to improve the quality of their work, are also going to find themselves hovering at or near the bottom of their standings.

If you want to create a good team -- a winning team -- then you will need to find some way to measure the quality of different players, then make sure that you bring the best players into the positions that they are best suited to play. The sports pages are riddled with statistics, all of which are used to determine the 'merit pay' of players.

Now, the merit pay system used in basketball is not perfect. There are intangibles such as teamwork and entertainment value (the ability of a particular player to draw fans because he makes the game more entertaining) that are difficult to measure and figure into a player’s statistics. Yet, they are important to distinguishing the quality of the player. Yet, a coach would have to be mighty foolish to take this as an argument against looking at the statistics and using that information, in part, to determine how much an individual player is worth to the team.

Yet, the argument used against merit pay for teachers tend to follow the lines of, “The system is not perfect; therefore, it should not be used.” The standard argument falls along the lines of coming up with some list of shortcomings for a merit pay proposal, and using those shortcomings to argue against merit pay.

If this line of reasoning were valid, it would be as applicable to merit pay for basketball players as it is for teachers – because the system for determining merit pay for basketball players is not perfect either.

The proper way to evaluate a proposal is not to measure whether it is perfect. No proposal will ever be able to pass such a test. The proper form of evaluation is to compare a proposal to alternative proposals and see which is the least imperfect. The alternative proposal in this case is a proposal to base a teacher’s pay entirely on how many years the teacher has been working, and to condemn any attempt to even try to measure the quality of that teacher’s work.

Problem: Inaccurate Tests

There certainly are some problems with merit pay based on test scores. One of the issues that we would need to keep an eye on is the quality of the tests.

Imagine a school system that bases merit pay on tests scores. Then, the voters select a school board that insists that the local tests include the following question:

(97) "Intelligent Design" is an example of a scientific theory.

Plus, they insist that the correct answer is, ‘TRUE.’

In a system that gives merit pay for test scores, such a system would be giving merit pay to teachers who actually make their students stupider. Those students who think that ‘TRUE’ really is the right answer to this question demonstrate that they either have no idea of what constitutes a scientific theory, or no idea what ‘intelligent design’ says. Either way, they have some flaw in their education.

However, the proper conclusion to draw in these types of cases is not to abandon the concept of merit pay. It is to accept that we also have a responsibility to fight to make sure that the tests themselves are good tests.

If we use this as a reason to oppose merit pay, then we are in fact arguing, “Because the current system is producing so many people who do not understand issues such as this, we should preserve the current system.” It is a rather poor argument.

Problem: Cultural Bias

In addition to the possibility of an undereducated society putting wrong answers on the test, there is a problem with test writers putting cultural bias into their tests.

For example, boys tend to be substantially more knowledgeable about sports than girls. To the degree that questions in a standardized test make use of this background of sports knowledge, to that degree boys will have an advantage over girls. Similar problems exist for students who belong to ethnic cultures that differ from the culture of those who created the tests. Teachers whose students do not share the same cultural assumptions as those who created the tests will be punished relative to those teachers whose students share the same cultural knowledge as the test makers.

However, this is not an argument against merit pay. This is an argument for putting more effort to eliminating cultural biases in tests. Our reasons for doing this extend far beyond the benefit of allowing merit pay based on test scores to work more efficiently. Even if we do not base merit pay on these tests scores, cultural bias still is harmful to certain students. It affects the student’s comparative standing among other college graduates and affects the ability to get scholarships. It also has a psychological affect on a student’s self-esteem.

Saying that cultural bias is an argument against merit pay is like saying that the discovery of a rotten apple in an apple barrel gives reason to toss out the perfectly good apple sitting next to it. The proper response to this type of problem is to throw out the rotten apple. In this case, cultural bias in standardized testing is the rotten apple.

Problem: Relativism

Standardized testing plays havoc with the philosophy of relativism because standardized testing works on the assumption that there are right answers. Creating test questions that actually count certain answers wrong goes against the doctrine that no answers are actually right or wrong – that any answer a particular student likes for a question is ‘true for him’ and deserves just as much respect and consideration as any other answer.

However, the problem here is not to be found in the standardized tests. It is to be found in the doctrine of relativism. There is nothing at all wrong with claiming that there are right and wrong answers, and it is possible for students to get some answers wrong. Furthermore, it is the job of those who opt to be teachers to do the best job they can in helping the students to accurately identify and distinguish (objectively) true propositions from those that are (objectively) false.

Problems: Unmeasured Qualities

A system that bases merit pay on improvements on test scores do not measure all of the qualities that make a teacher a good teacher. There are other qualities that must be considered as well.

(1) Character development. If we want our children to grow up in a society in which they can live safe and secure lives, then we are going to want him to grow up in a society where their neighbors have been caused to acquire a solid moral character -- honestly, kindness, integrity, and the like. Standardized tests do not measure a teacher's ability to promote these qualities. As a result, there are aspects to being a good teacher -- a teacher very much deserving of merit pay -- that these tests do not cover.

(2) Concern for students. Teachers are in a good position to determine the state of a child's emotional and physical health as well as the quality of the student's life outside of school. He can detect signs of abuse, illness, and injury that might otherwise go unnoticed. A good teacher may be able to get help for a student who is abusing drugs or alcohol or engaging in other forms of risky behavior, or who may have become depressed or even suicidal. There is a significant difference in quality between the teacher who is concerned about the well-being of his students and one who could not care less.

These are just two examples of other qualities that we would like to look at in determining whether an individual is or is not a good teacher.

This is quite similar to the problem that being a good basketball player is determined by more than the number of points an individual player scores. Teamwork and entertainment value are also important, but are difficult to measure.

The answer to this problem is, again, not to abolish the practice of determining pay based on qualities that can be measured. It is an argument for combining this with other systems that consider these intangible qualities. There is nothing inconsistent with arguing for a system that will give a teacher a bonus of $N for bringing about an increase in her student’s test scores of I over the course of a school year – plus give that teacher merit pay (or not) depending on other criteria.

Assuming that test scores must either be the sole criteria for merit or it should not be used at all is to construct a straw man – to defeat a position that no rational person has in fact sought to defend.

Conclusion

Yes, I agree that there are a lot of issues associated with the subject of merit pay. The proposal is not perfect. Yet, the lack of perfection is not an argument against the system. If it were, then all systems must be abolished because none of them are perfect.

The problem with most critics of merit pay is that they do not offer an alternative proposal. If they did, then the two proposals can be compared to determine which has the fewest imperfections. Instead, their arguments are entirely destructive. “Your merit pay system has these problems; thus, it is a bad idea.”

It is only a “bad idea” if it has more problems than an alternative system.

The alternative system, which says, “Pay all teachers the same regardless of their individual skills and talents in the area of actually making their students smarter,” has far more problems than this merit pay system. It is a system that tells teachers that they do those who care nothing about the quality of their work are just as important and deserve just as much praise and reward as those who care a great deal.

It is a recipe for mediocrity. It is a recipe that creates a society of individuals who are less well educated than they actually need to be.

Furthermore, there is nothing to prevent us from adoping a system now, aware of its problems, and then go to work making adjustments to deal with those shortcomings. Indeed, we would need to start somewhere, and then start collecting the data that would be necessary to improve the system. Critics of such a system must not only say that 'there are problems'. They must assert 'we cannot make sufficient improvements as we see the results of our proposal.'

Besides, we are not talking about a system where a teacher's entire pay is determined by these measures. We are talking about adding a few hundred to a few thousand dollars of merit pay. Most of the existing, "pay the person regardless of the quality of his work" system will remain intact.

As I have argued earlier; when we die or suffer any catastrophe in our lives, in almost all cases we are made to suffer substantially because of our ignorance – because we did not know something that would have otherwise saved us from this danger. Poor education makes all of us worse off than we would otherwise be.

11 comments:

Joe Otten said...

As you say, Alonzo, none of the problems with merit pay that you mention are good reasons not to have it.

The problem is that it introduces an incentive to game the system, and this can lead to worse outcomes. For example a refusal to enter underperforming students for tests, or an aggressively disciplinarian attitude towards them leading to more expulsions. (One girl was excluded from Trinity Academy, Doncaster, UK for "tying her hair too loosely".)

If we reward good performance in one aspect of a teachers duties, what we are saying to teachers is that they should put more effort into that at the expense of the other aspects. If that is geniunely the outcome we seek, it may be a good policy, but either way we should have our eyes open about this.

Performance Indicators never measure precisely what they purport to measure - they are a poor substitute for professional judgement.

Alonzo Fyfe said...

"Professional judgment" is a system that allows people to rationalize their prejudices.

This is how an all-white review board decides that white teachers do a better job than black teachers.

This is how the Christian review bored decides that the Atheist teacher really is not a good teacher.

This is how the pro-Bush review board decide that the pro-Bush teachers deserve merit pay and the anti-Bush teachers do not know what they are talking about.

I am not talking about a bunch of evil bigots cackling over their hatred. I am talking about the type of prejudice that seeps into decision-making largely undetected. I am talking about the type of prejudice that causes a white woman to lock the door to her car when she sees a group of black males -- even though she preaches equality.

To the person who believes that the free-market economy is the only true and just economic system to think that a person who teaches something else is a "good teacher."

Both points that I made in my essay still stand.

The first is that performance indicators are not precise -- but they do not need to be. They only need to be better than the alternative in those circumstances in which they are used.

The second is that there is no contradiction in using two systems at the same time -- a round of merit pay based on performance indicators, and a round of merit pay based on "professional judgment." We are not talking about an either/or situation here. we can do both.

Anonymous said...

I always enjoy reading your posts, however, I was disappointed with the thinking behind this one.

There is a problem with your basketball team analogy. In professional sports, team owners (and even college coaches) get to pick the players on their team. They can also cut team members at will. Teachers do not get to pick their students and they certainly can't cut the low performers.

I have worked in both low performing and high performing schools. Regular student attendance and parental involvement are two important factors that differentiate the low from the high. These are issues that are out of a teacher's control.

Most people, teachers included, want to do a good job. The number one way to insure lower performance is to demoralize someone. Anytime you decide someone will not receive the pay of their peers because of issues beyond their control, you are sure to demoralize them. Fifty percent of teachers who enter the field drop out within the first five years. This speaks to the difficulty of the job, but also suggests that people are not going to stay with it unless they have some competence.

I could go on about why "merit pay" is a counter-productive idea. You listed a number of them, but there are others. How does one qualify for merit pay if they teach outside the "core" academic areas (such as art, driver's ed,, etc.)?

Alonzo Fyfe said...

Brad

I am pleased that you like the blog. I suppose, with nearly 200 posts completed, there certainly has to be about 20 in the bottom 10%. This may well be one of them.

However, I do not think that you are characterizing the analogy correctly. My analogy was not between picking and awarding basketball players versus students, but picking and awarding basketball players versus teachers. School boards and basketball team owners can both cut poor performers at will.

If you are looking for something to compare to the teacher's inability to pick students, it would be the basketball player's inability to pick who works for the opposing teams. They are largely out of the basketball player's control.

Teachers want to do a good job. Basketball players want to do a good job. The decision to award the best players highly might demoralize the others, but it tends instead to have the effect of encouraging them to work harder and become better. There is no reason not to expect the same with teachers.

I do not think that a high dropout rate guarantees that those who stay in are competent. In fact, I think that we have a system that drives competent teachers away due to frustration. I thought about teaching and decided against it ... one reason being that teachers are not permitted to actually teach. I look at the lawsuits filed against teachers, parents who protest about every decision, and students who, if they are made angry, know just what to say to get the teacher in serious trouble, and I saw this as not worth the risk.

As for the difficulty in measuring for merit pay in some subjects; this ends up being irrelevant. There is no law of nature that says that we must apply merit pay to all subjects or none. So, the argument that "we cannot use it in all cases" does not imply "therefore, we must use it for none."

My question is: What do we get without merit pay? I think that the basketball player analogy still applies. What type of basketball team could you create if basketball players were paid only on the basis of number of years with the team?

Anonymous said...

I will agree with the concept behind merit pay. However, There are some serious problems with it. As you say, it can be implemented before we have all the details hammered out, but since we already know what some of these problems will be, there is no reason not to discuss them now.

The first problem is that merit pay systems have a direct correlation between student grades and teacher pay. Now, lets say that one teacher has a student who, in the previous year, had received an A in Math. After a semester in the new teachers class, the students grade drops to a B. Take another student in another class who had received a D the previous year. One semester in the second teachers class and the grade rises to a C. Under the current models I've seen for merit pay, the first teacher will have received more money then the second. An obvious problem, though easily corrected by granting teachers merit pay based on a students grades in relation to their previous grades.

Another problem is that early in the semester, the assignments are more similar to the subject taught the previous year. A very good portion of the first quarter of trigonometry involves more algebra the anything else, with trig functions slowly replacing the variables in the problems. Student performance is thus disproportionately better at the beginning of the year. This can't really be changed, since trying to get students to grasp trigonometry without presenting it algebraically will just cause them to drop out of the class. Also, a single class tends to cover a very broad range of subjects, particularly at the elementary level. Students are going to have an innate ability to perform much better in certain areas then in others. Many times a student can handle any problem thrown at them when it's given in numbers or words, yet when given a diagram in geometry, they are unable to figure out what the problem is, much less solve it. The only thing that comes to mind to deal with this is basing pay around how a teacher performs in relation to others teaching the same subject, as this determines the best overall teaching method. Yet it also seems a very bad idea to make something as essential as education into a contest were a teacher will benefit by refusing to lend supplies and class materials to another teacher (A widespread habit that can be attributed to under funding for public schools).

Now this should be self-explanatory. Which of these classes do you think will have a higher average grade, remedial math or advanced placement trig?(While AP trig is much harder, these students are also much more likely to care about their grade, and it is rare to see anything as low as a C in an AP class.) This would be fairly easy to solve by basing pay on a curve favoring teachers who teach lower track classes.

Alonzo Fyfe said...

Anonymous

I agree that saying that we do not have to solve all of the problems now does not imply that we ought not to discuss the problem now.

A better model for merit pay for teachers would not depend on student's course grade at all. Instead, what is used is the student's improvement in standardized test scores between the start and end of the school year. So, we are looking at a student who receives a score of, say, "585" on a standardized test at the end of one school year, scoring a "650" at the end of the next school year. The teacher would get credit for 65 points of education.

I do agree that merit pay should compare teachers in comparable roles -- and I expect some political bickering over what counts as a "comparable role."

To eliminate the "competition" aspect, merit pay could be set to a base amount and all teachers who exceed a certain performance level will get merit pay. And/or there can be an additional school-level merit pay which depends on how well the whole school does.

My approach to these types of issues is that I like the idea of experimenting. I see nothing wrong with using different systems in different areas and seeing the results.

I love experimenting and collecting data. Whenever two people get into a dispute over something like this, rather than take sides, I always argue for, "One of you takes one school system and do it your way; the other takes a different school system and tries his method. We will meet at the end of the year and look at the results."

I simply see no need to fight over issues such as this where we should be looking for ways to determine who is right through experimentation.

LBBP said...

I am not a teacher myself, but I (play one on TV) am married to one. She has taught for 8 years and she is leaving the business. Why? Because, there is so much testing and so little positive reinforcement that she just can't stand it anymore. She loves her students, and she really wants (wanted) to make a difference, but the system is now stacked so far against most teachers, that they are just treading educational quicksand.

Further emphasis on testing takes time away from teaching. Yes, testing needs to happen. Yes, performance bonuses for teachers that are doing good work might help to create incentives. But, first and foremost, teachers need the opportunity to teach. No matter what the wingnuts say, the bottom line is that teachers need smaller class sizes, more resources, and longer (or more) school days.

In most public schools today the average k-12 teacher is responsible for child psychology, parent counseling, special education, discipline, first aide, day care, family planning, document management, and oh yeah, teaching. Society expects teachers to teach morality, religion, and patriotism, and then get around to math, reading and writing. What about science, social studies, or the arts? Being phased out to make room for more tests. Society expects them to handle all of the crap that the parents don't deal, then add in other crap like; "my child can't participate in the Thanks Giving Day project because it's against our religion", or "you have to make sure Johny goes to the nurse three times a day to get his HIV meds", or "you loose your prep period because you can't send Darius to Gym because he has a personality disorder and he gets in too many fights" or....

The list of special needs or special concerns or special situations goes on and on, yet somehow a teacher with 30 students (most with "special" needs) is supposed to meet all of those individual needs and simultaneously teach the rest of the class.

That is why our schools are failing. Because parents don't or can't hold up their end of the bargain. Because society undervalues education, and somehow expects the teachers to magically make up the difference.

The public school system in this country is a ship sinking under the weight of societal apathy, lack of funding, and insufficient resources. Basing teachers pay on how well they teach while the ship sinks around them is not going to help anything.

Clark Bartram said...

The problem of our failing educational system would be fixed by giving vouchers so that a child can go to any school he or she wants. Do away with public schools and allow competition to improve education.Throwing money at them is not the answer, competition is.
http://abcnews.go.com/2020/Stossel/story?id=1500338

Alonzo Fyfe said...

LBBP

There is nothing in what I wrote that says that increased amount of time testing, and decreased amount of time teaching, does not suffer from diminishing marginal utility.

When I was teaching, I simply did not have tests in class. I assigned papers instead, in part because I wanted to spend class time teaching instead of testing.

Yet, some tests are useful. Just as a car company will tear apart or crash-test a certain number of its cars, there are good reasons for a school to engage in some testing. Just as it is not productive for the company to crash test every car it makes, it makes no sense for a school to spend every minute testing.

I, too, have been turned off to teaching largely because the system sucks. Yet, I do not see this as being the primary source of suckiness. The major problem that I saw was that teachers are limited to only saying those things that every parent will unanimously agree is true. If a teacher says anything that even one parent disagrees with, she is in trouble. Teaching is not possible under those criteria.

I acknowledge that this is a problem with testing as well -- where parents will not allow a test question that calls into doubt their personal beliefs.

In fact, one reason that I favor testing would be to reveal the fact that getting rid of some of this garbage is actually good for students. If the teachers with the highest scores are those who put up with the fewest hassles, then this might help to create a hassle-free environment.

So, everything that you said is true, and these other issues should be addressed. Yet, they do not argue for the elimination of merit-based pay. It argues, instead, for the elimination or reduction of these other issues.

Alonzo Fyfe said...

I covered my position on voucners in School Vouchers I and School Vouchers II

My position is generally in favor -- with caveats.

Anonymous said...

The only problem I noticed is that intelligent design is a scientific theory...