New Evidence of Racial Bias on SAT

A new study may revive arguments that the average test scores of black students trail those of white students not just because of economic disadvantages, but because some parts of the test result in differential scores by race for students of equal academic prowess.

The finding -- already being questioned by the College Board -- could be extremely significant as many colleges that continue to rely on the SAT may be less comfortable doing so amid allegations that it is biased against black test-takers.

"The confirmation of unfair test results throws into question the validity of the test and, consequently, all decisions based on its results. All admissions decisions based exclusively or predominantly on SAT performance -- and therefore access to higher education institutions and subsequent job placement and professional success -- appear to be biased against the African American minority group and could be exposed to legal challenge," says the study, which has just appeared in Harvard Educational Review (abstract available here).

The existence of racial patterns on SAT scores is hardly new. The average score on the reading part of the SAT was 429 for black students last year -- 99 points behind the average for white students. And while white students' scores were flat, the average score for black students fell by one. Statistics like these are debated every year when SAT data are released, and when similar breakdowns are offered on other standardized tests.

The standard explanation offered by defenders of the tests is that the large gaps reflect the inequities in American society -- since black students are less likely than white students to attend well-financed, generously-staffed elementary and secondary schools, their scores lag.

In other words, the College Board says that American society is unfair, but the SAT is fair. And while many educators question that fairness of using a test on which wealthier students do consistently better than less wealthy students, research findings that directly isolate race as a factor in the fairness of individual SAT questions have, of late, been few.

The new paper in fact is based on a study that set out to replicate one of the last major studies to do so -- a paper published in the Harvard Educational Review in 2003, strongly attacked by the College Board -- and the new paper confirms those results (but using more recent SAT exams). The new paper is by Maria Santelices, assistant professor of education at the Catholic University of Chile, and Mark Wilson, professor of education at the University of California at Berkeley. The earlier study was by Roy Freedle of the Educational Testing Service.

The focus of both studies is on questions that show "differential item functioning," known by its acronym DIF. A DIF question is one on which students "matched by proficiency" and other factors have variable scores, predictably by race, on selected questions. A DIF question has notable differences between black and white (or, in theory, other subsets of students) whose educational background and skill set suggest that they should get similar scores. The 2003 study and this year's found no DIF issues in the mathematics section.

But what Freedle found in 2003 has now been confirmed independently by the new study: that some kinds of verbal questions have a DIF for black and white students. On some of the easier verbal questions, the two studies found that a DIF favored white students. On some of the most difficult verbal questions, the DIF favored black students. Freedle's theory about why this would be the case was that easier questions are likely reflected in the cultural expressions that are used commonly in the dominant (white) society, so white students have an edge based not on education or study skills or aptitude, but because they are most likely growing up around white people. The more difficult words are more likely to be learned, not just absorbed.

While the studies found gains for both black and white students on parts of the SAT, the white advantage is larger such that the studies suggest scores for black students are being held down by the way the test is scored and that a shift to favor the more difficult questions would benefit black test-takers.

The new study is based on data for students who enrolled at the University of California system across several administrations of the SAT -- with versions used subsequent to Freedle's article. (The new research is the result of a study the authors undertook at the request of University of California officials, and they note in the paper that despite the request for information from the University of California, it took two years for the College Board to provide the data needed.) While the new study found the same DIF that Freedle did, an attempt to find a DIF for Latino students failed to show one.

But, the authors write, that doesn't minimize the significance of their findings that back the study from 2003 that the College Board has said wasn't accurate. "Although our findings limit the phenomenon observed to the verbal test and the African American subgroup, these findings are important because they show that the SAT, a high-stakes test with significant consequences for the educational opportunities available to young people in the United States, favors one ethnic group over another," write Santelices and Wilson.

"Neither the specifics of the method used to study differential item functioning nor the date of the test analyzed invalidate Freedle's claims that the SAT treats African American minorities unfairly."

Kathleen Fineout Steinberg, a spokeswoman for the College Board, said that just as the organization disagreed with the 2003 study, so it does with the new research. She questioned whether the California sample could be seen as broad enough to draw conclusions on, and said that some of the tests examined had less of a DIF than others, raising questions about the assumptions made. She called the Harvard Educational Review study an example of "presenting inconsistent findings as conclusive fact."

She said every test question used on the SAT is subjected to rigorous analysis (before use) to weed out any that would not be fair to all test takers. "We believe that our test is fair," she said. "It is rigorously researched, probably the most rigorously researched standardized test in the world."

As to the persistence of score differences, Steinberg said that this is not because of the test. "There certainly are subgroup differences in scores," she said. "We recognize that and acknowledge it. It's a reflection of educational inequity. It's something we are concerned with." She also said that the College Board welcomes research on the SAT, but viewed the Freedle study as having been "discredited," and said that nothing in the new study changed that view.

The College Board's tough stance on Freedle's research is not new -- and was recounted by Jay Matthews in an article in 2003 in The Atlantic Monthly. (Matthews broke the news about the new research, in his Washington Post blog.)

Robert Schaeffer, public education director of the National Center for Fair and Open Testing, a long-time critic of the SAT, called the new research "a bombshell," and said that the study "presents a profound challenge to institutions which still rely heavily on the SAT to determine undergraduate admissions or scholarship awards."

Schaeffer said that he agreed with the authors of the new study that use of the SAT could face legal challenges, given that this study now backs the finding that some of its questions may be harmful to the scores of black test-takers. While the College Board says colleges aren't supposed to rely too much on the SAT, and most colleges that require the SAT say that they use it only as one factor among many, Schaeffer and others have doubted those claims.

"A shrewd litigator could use this study and the process of discovery to find out a lot more about how colleges use the test and, at a minimum, embarrass them," he said.

More broadly, he said that with more colleges considering ending SAT requirements, this new study is "another strong argument" for doing so. "It's going to add to the momentum."

Marist College will be announcing this week that it is ending its SAT requirement, joining many others that have done so.