THE incorrect scoring of 5,000 exams from the College Board's October SAT is a reminder that testing is not an exact science. But just how inexact is it?
The October errors occurred, in part, because some answer sheets had become damp and did not register correctly when they were processed.
Testing companies try to guard against errors where they can. The Educational Testing Service in Princeton, N.J., for example, which works on some tests with the College Board, has 700 people working to insure technical quality in test development. They scrutinize even small details like whether the size of the print on a page or the resolution on a computer screen affects performance.
"You decide what you want to measure, and then you have to make sure you are not measuring something else, like poor vision," said Ida M. Lawrence, senior vice president of research and development at the testing service.
Despite such efforts, problems crop up. Here is a sample.
Missed a line. A student racing through a multiple choice exam may skip a line on the answer form by mistake, and then put every answer that follows on the wrong line.
In a study of more than 100,000 SAT exams, computer scientists found that nearly 2 percent of the test takers made such mistakes. One of the study's authors, Steven Skiena, a professor at the State University of New York at Stony Brook, said such errors could be identified by computer so that the student received what he called an appropriate score.
"If you can detect what they were thinking, you should score them for that," he said.
The College Board, which calls the practice "mis-gridding," says students should tell their proctors when they make such mistakes, and that their scores may be corrected. Of 1.5 million SAT tests taken last year, only about 100 students reported the problem. Or if they move quickly, they can simply cancel their scores.
Bear down. A New York City student who received a score of about 1800 on the SAT last June out of a possible 2400 took the test again in October, hoping to do better.
The student, who asked that her name not be used because she did not want other people to contact her, said her score plunged 360 points.
So she paid to have her October test scored again, by hand, an option that the College Board offers. Her score shot up 510 points.
The board said she had marked her answer form so lightly that the machine processor could not read many of the answers.
Missed a page. A Maryland student who was allowed to type his responses to the essay portion of an Advanced Placement exam because he had a disability, earned a 1, the lowest possible score on a scale of 1 to 5. When his exam was scored again by hand, his score jumped to a 5.
His mother, Judi Becker, said an official at the Educational Testing Service told her the typed pages, which had been appended to the regular answer sheet, had been overlooked, making it appear that the student had answered only two of the four essay questions.
Tryout questions. Proposed new questions are inserted into real tests for a tryout. Even though the questions do not count, they can throw students off, said W. James Popham, an emeritus professor at the University of California at Los Angeles who has also worked in test development.
"Some of these previously untried items are bloody awful, occasionally having no right answers at all," he said. "The test taker doesn't know that this is a 'no-count' item and, therefore, might be traumatized by being unable to answer it." Though not a frequent problem, he said, it does occur.
There are other testing problems that have less to do with external factors like paper and pencil and more to do with what goes on inside a student's head.
Thomas M. Haladyna, a professor of educational psychology at Arizona State University, said there was a range of everyday issues like test anxiety (a quarter of the population has it, he said), fatigue, lack of motivation and language problems for students who are not native English speakers.
"Test scores can easily be corrupted," he said. "You shouldn't trust a test score unless it is validated with other information that corroborates it."
With so many problems, are there any protections? Robert A. Schaeffer, public education director of FairTest, which argues that tests are relied upon too heavily, recommends that when possible, students get copies of questions and answers, "both as a check on scoring accuracy and to learn from their mistakes in case they retake the exam."
"The unfortunate truth is that the fates of students — and their parents — are at the mercy of the testing industry," Mr. Schaeffer said. "Absent mechanisms for oversight and regulation, we have to hope they get it right."
