Part One: Internal documents show that the U.S. college entrance exam has been compromised in Asia far more often than acknowledged. And the newly redesigned SAT retains a key vulnerability that the test-prep industry has exploited for years.
As SAT was hit by security breaches, College Board went ahead with tests that had leaked
中文 (Chinese translation)
Xingyuan Ding is a sophomore at the University of California, Los Angeles, one of America’s most exclusive public universities. In applying to schools, the 20-year-old from China took the SAT college entrance exam four times.
He had an advantage on his final try: a booklet compiled by a Shanghai test-preparation school he attended.
His study aid was far more valuable than the practice questions that students in America use to prepare for the SAT, the standardized test used by thousands of U.S. colleges to help select applicants. Known in Chinese as a jijing, the booklet was essentially an answer key. It revealed words from the correct responses to multiple-choice questions that had appeared on past SATs - many of which would be used again on the exam Ding took.
Thanks to the booklet, Ding said he already knew the answers to about half of the critical reading section of the SAT when he took the test in Hong Kong in December 2013.
“I felt really lucky,” Ding said.
His score on that section? A perfect 800, he said.
Ding’s advance look at material from the test he took was no fluke. His cram school is part of a vibrant Asian industry that systematically exploits security shortcomings in the SAT. Chief among them is a vulnerability created by the owner of the exam: the routine practice of reusing material from tests that already have been given.
The College Board, the not-for-profit organization that owns the SAT, has acknowledged widespread problems with test security in Asia in recent years. Since October 2014, the New York-based organization has delayed issuing scores in Asia six times and canceled an exam sitting in two locations there – steps the College Board takes when it has evidence that test material has been exposed to the public.
But the breakdown in security is more pervasive than the College Board has publicly disclosed, Reuters has found. In addition to the security-related incidents the College Board has announced, the news agency identified eight occasions since late 2013 in which test material was circulating online before the SAT was administered overseas. (See table for details.)
A confidential PowerPoint presentation reveals that College Board officials had documented widespread security problems in June 2013, shortly after canceling a sitting of the SAT in South Korea. The PowerPoint, reviewed by Reuters, shows that half of the SATs in inventory at the time had been “compromised” – the College Board’s term to describe exams whose contents have leaked, in whole or in part, outside the organization. Four of the exams had been compromised by an unnamed “Chinese website.”
Even so, College Board officials confirmed that some portions of those tainted tests were later administered overseas. And the College Board took no steps to restrict testing in China, the SAT's largest international market by far, even as it tightened security in smaller countries where exams had leaked.
About 64,000 students took the SAT in East Asia during the 2013-2014 school year, including 29,000 from China. And some 125,000 mainland Chinese undergraduates now attend U.S. universities.
Because of the extent of the security breaches Reuters uncovered, admissions officers have no idea which of those foreign test-takers saw material in advance.
The news agency’s findings “add a huge new level of distrust” about the validity of international scores, said Steve Syverson, an administrator at the University of Washington Bothell and a former board member of the National Association for College Admission Counseling. “The College Board does a lot of good things, but it will clearly be a major challenge for them to restore trust in the integrity of the test.”
David Coleman, the president of the College Board, declined to comment for this article.
The SAT’s security crisis comes as the College Board, whose membership includes more than 6,000 educational institutions, has introduced a redesigned version of the exam this month in the United States. In America, the abuses appear to be less rife because the contents of standardized tests are protected by strong copyright laws.
The first overseas sitting of the new test takes place in May.
The new exam leaves in place a fundamental weakness plaguing the old one: the recycling of test material. The practice will continue with the new SAT, the College Board told Reuters. And that reuse of test material has proved to be a major security hole.
Recycling enables cram schools to gather reading passages and questions from past tests, then figure out the answers and package that material for their clients to study. The information comes from many sources. Test-prep centers have associates take the exam and memorize what they’ve seen. Some people even photograph the test booklet. The cram schools also analyze test information that American teenagers share on Internet forums. At times, cram schools have obtained actual SAT tests.
Already, American students who took the new test in March have been discussing the questions and answers online in granular detail. Asian prep centers have rushed to learn all they can about the redesigned SAT and share the intelligence with their clients.
The ability to obtain inside knowledge of what’s going to appear on upcoming exams is critical to the test-prep operators.
“Basically, the only way to survive in the industry is to have a copy of the test” in advance of a sitting, said Ben Heisler, who offers test-prep and college-consulting services in South Korea.
“It’s like doping in the Tour de France,” Heisler said. “If you don’t do it, someone else will.”
Security breaches abroad are increasingly significant for U.S. higher education because schools are allocating more seats than ever to foreign students. About a third of the 761,000 degree-seeking foreign students in America come from China, according to the Institute of International Education. Overseas students are especially attractive because most don’t qualify for financial aid and thus pay full price. Chinese students spent almost $10 billion on tuition and other goods and services in America in 2014, Department of Commerce statistics show.
Evidence that some foreign applicants are displacing Americans because of an unfair advantage on the SAT could add to a backlash against standardized testing in college admissions. Most universities still require applicants to take the SAT or its rival, the ACT, which is more popular in the United States. Yet a growing number of schools have questioned the usefulness of the exams and now make them optional. The SAT is taken by far more foreign students applying to U.S. colleges than the ACT is.
“It is hurting all students when someone cheats on any aspect of their application,” said Douglas L. Christiansen, dean of admissions at Vanderbilt University in Nashville, Tennessee, and chair of the board of trustees at the College Board. “It is displacing someone else.”
College Board officials, including Christiansen, stand behind their handling of test security and the legitimacy of the scores sent to universities. Stacy Caldwell, a College Board vice president in charge of the SAT, wrote in a letter to Reuters that the organization “would never move forward with a test administration … without the full confidence that we can maintain the integrity of the exam and deliver to our member colleges and universities valid scores.”
Caldwell said that a growing and evolving industry “thrives on the theft of test content and undermines the meaning of test scores.” In an interview, she said the test-prep companies that misuse material, not students like Ding, are the “bad actors.”
Caldwell acknowledged that College Board officials are unable to assess how many test-takers have seen actual exam material before taking the SAT – a reality that calls into question the number of current college students who gained an unfair advantage.
“I don’t think we have specific statistics on that,” Caldwell said.
“INTERNAL CONFIDENTIAL”
The College Board did, however, have information that showed a security crisis was brewing in East Asia three years ago.
The information is contained in a June 2013 PowerPoint presentation, marked “Internal Confidential.” It was written after a major security breach that year in South Korea. The College Board canceled the May 2013 sitting of the SAT there after test-prep operators allegedly obtained tests in advance.
Co-authored by Caldwell and shared with other senior College Board employees, the PowerPoint document describes broad security breaches overseas.
SAT “content theft” had been identified as a problem in South Korea, Egypt, Saudi Arabia and China, and answer keys for SAT exams were available in some of those countries, the document notes.
The document says nine of the 18 versions of tests in the entire SAT global inventory at the time “have some level of known compromise.” Five of those nine compromised tests were scheduled to be administered outside the United States in the coming school year.
The breaches extended to the hour-long tests the SAT offers in specialized subjects. Those exams are often required for students applying to elite U.S. colleges. Eight of the 10 existing Mathematics Level II subject tests were compromised – three in their entirety and five in part, including two exams that had never been given anywhere, the PowerPoint shows. Ten of the 13 Biology exams were also compromised in whole or in part, including one totally new test.
To reduce opportunities for gaming the SAT and to protect uncompromised versions of the test, the College Board contemplated cutting the number of times it offered the exam in countries with security problems, the PowerPoint shows. Doing so would limit the likelihood that test-takers had seen parts of an exam before taking the test, or that an exam would leak, the document notes.
According to the PowerPoint, College Board officials also considered another approach: Push ahead with all scheduled tests in every country, regardless of the security risks. Under this scenario, one of the “benefits” listed was giving the “appearance that security situation is under control.”
The option was “not recommended,” in part because the organization feared it could result in even more cheating and that “another large-scale incident could get attention of U.S. press and universities.”
In an interview, Caldwell said the passage was “probably beyond poorly worded.” She stressed that College Board executives ultimately rejected the option. “We consistently make these decisions for the right reasons,” Caldwell said, citing the organization’s desire to balance “the security of the exam with the needs of students and our members.”
The June 2013 PowerPoint shows that the College Board decided to reduce from six to four the number of test dates held in South Korea, Egypt and Saudi Arabia, where actual exam content had been illegally obtained. But it didn’t do the same in China, though the document notes that a Chinese website had “compromised” four SAT exams.
Caldwell said the College Board didn’t act in mainland China or Hong Kong because “we had no evidence that they had incidents of test materials being stolen.” She didn’t elaborate on the nature or extent of the breach by the Chinese website.
Included in the PowerPoint was a “market impact analysis” that showed how many fewer times the test would be taken if the College Board limited the number of test dates. The analysis noted that reducing the number of test sittings would enable the arch-rival ACT exam to gain market share in places where the College Board was struggling to stop cheating. That’s because students may have decided to take the ACT if the SAT were offered less frequently in their countries.
Based on the figures in the document, Reuters calculated that limiting the number of sittings in China would have cost the College Board roughly $1.2 million to $1.5 million in lost revenue over the next fiscal year. The College Board declined to comment on that estimate. The organization reported net income of $99 million on revenue of $841 million in the year ended June 2014.
The College Board confirmed that it continued to use material from at least some of the nine exams that had been compromised. Officials said the questions were used in countries other than those where the College Board found that the items had been circulating. They declined to say where or which portions of the compromised tests were reused.
The decision to offer fewer test dates in the three countries was no panacea, the College Board acknowledged in the PowerPoint. It wouldn’t solve what the document calls “mobility risk.” Because Asian students frequently travel abroad to take the SAT, students from South Korea might travel to Japan, for example, where they would be able to take a version of the test that had been compromised in their home country.
Test-security specialist Neal Kingston, who reviewed the PowerPoint at the request of Reuters, said he was troubled that the College Board decided to administer test material that it knew had been compromised. He also said he was perplexed that the College Board didn’t limit test sittings in China, given that it knew in 2013 that a Chinese website was responsible for four breaches.
Kingston, who spent 12 years working for the College Board’s security contractor, Educational Testing Service, said he was “significantly troubled by the magnitude” of the security failures documented in the PowerPoint.
“The only way to survive in the industry is to have a copy of the test. It’s like doping in the Tour de France. If you don’t do it, someone else will.”
“The problem is even larger than I had believed it to be,” said Kingston, the director of the Achievement and Assessment Institute at the University of Kansas.
Ray Nicosia, who heads the Office of Testing Integrity for ETS, said investigators use computer analytics to help spot cheating after the fact. But the College Board and ETS declined to discuss particular security efforts. College Board spokesman Zach Goldberg said that “sharing specific details can help the organizations and individuals attempting to gain an unfair advantage.”
HARVESTING THE TEST
Gaming the SAT can take many forms. One scam is time-zone cheating, in which test-takers in one part of the world feed questions and answers from the SAT to people sitting the exam later the same day. Another problem is the outright theft of test booklets. In 2015, SAT administrators began shipping booklets to and from some test centers in lock boxes.
A student doesn’t need complete questions and answers from an SAT to gain an edge. Getting an advance look at a single reading passage can help a test-taker master the material and get through it more quickly, leaving more time to handle other questions. One extra correct answer can boost a score by 10 or 20 points on the 800-point scale for each section, according to a scoring table for practice tests posted online by the College Board.
The largest threat to the SAT’s integrity appears to come from the test-preparation centers. The Asian cram schools have prospered by exploiting what is perhaps the College Board’s biggest security weakness: its practice of taking questions and entire sections from tests that have been previously given in the United States and reusing them in later versions of the SAT, typically those offered overseas.
Recycling makes it is possible even for an honest student to take the same test twice unintentionally. (See related story, “Taking it twice.”)
One reason for recycling exam material is rooted in the science of standardized testing. The College Board must ensure that scores are comparable on different versions of an exam. The reuse overseas of material previously administered in America helps achieve that.
Recycling also saves money. Developing a single version of the SAT can take up to 30 months and costs about $1 million, according to people familiar with the process.
One way to stop cram schools from exploiting recycled material would be to administer questions once, globally, and then never use them again. The industry calls these “one-and-done” tests.
College Board officials said offering a new SAT each time a test is given is unrealistic. “This is not a matter of just running another one off the assembly line,” Caldwell said.
Spokesman Goldberg said the cost of using exams just once would be passed on to test-takers, who potentially would have to pay more than double the current fee.
It costs up to $54.50 to take the SAT; students in East Asia also pay a $53 surcharge. That’s a fraction of the thousands of dollars many Asian students pay to attend test-prep centers.
The “one-and-done” approach also wouldn’t solve the problem of time-zone cheating. Kingston, the test security specialist, said he still believes the College Board should cut back on the use of recycled material to combat the systemic security breaches.
“One-and-done forms, from a test validity point of view, is the best practice,” he said. “They need to find the sweet spot of better reducing cheating while not exploding their costs.”
Because of its reliance on recycling, the College Board needs a large inventory of various versions of the test. The more exams the organization has, the harder it is for students to predict and prepare for the version that will be given.
Chinese and South Korean test-prep operations have circumvented the College Board’s defenses by building repositories of past SAT questions. These archives, harvested from past exams given in the United States, focus on helping clients prepare for the SAT’s reading and writing sections. East Asian students who didn’t grow up speaking English find the language sections far more challenging than the math questions.
Among the companies that offer booklets of SAT questions is Sanli, the Shanghai-based test-prep center that UCLA student Ding attended. About 8,000 students take the company’s test-prep courses each year, said Peng Wu, general manager of Sanli’s Shanghai branch.
He confirmed that Sanli has created booklets that include key words from answers to past SAT questions. Creating these study aids “can’t be helped,” Wu said. If a prep school fails to do so, students “think you aren’t teaching them properly.”
To get material for the jijings, Wu said some test-prep centers have “friends” in the United States who take exams and surreptitiously photograph the forms or memorize and later reconstruct the questions.
Test-prep centers in Asia collect Internet chatter, too. American students who have taken the SAT often go online to kibitz about the questions and even try to reconstruct entire exams within hours of taking the test. The College Board instructs test-takers not to discuss the exam, but many do anyway.
“The College Board does a lot of good things, but it will clearly be a major challenge for them to restore trust in the integrity of the test.”
After the SAT was given in the United States last December, for example, someone created a text file and linked to it on the popular website reddit.com. The file contained 21 pages of questions that test-takers recalled, along with a discussion of what were purported to be the correct answers. Some parts of that same December test were reused just a month later in parts of East Asia on Jan. 23, according to people in the region’s test-prep industry.
To try to stop the online discussions, the College Board sends “take-down notices” to site operators. Because the SAT exam booklets contain copyrighted material, the College Board maintains that distributing information from them without permission is illegal.
Goldberg, the spokesman, said the College Board sent 18 take-down notices to social media and community forum websites in the past year, and issued more after the redesigned SAT was given this month.
One website that regularly hosts discussions about what’s on the SAT is College Confidential, highly popular with American college applicants.
Daniel Obregon, vice president of marketing and user experience at Hobsons, the company that owns College Confidential, said the site usually receives a few take-down notices from the College Board after each test date. It complies, he said, “with any requests pertaining to copyrighted material.” But he said he considers online discussions of SAT questions a “gray area.”
The College Board is tight-lipped about its methods in the United States. But it recently recycled test material in America.
The SAT given this January in the United States contained several of the same reading passages - including one on celebrities and another on dark matter - as the exam from June 2014, according to online discussions among test-takers after both exams.
“Ayyy... this test (June 2014) was the same as the (Jan 2016) SAT,” one person wrote on College Confidential.
Enforcing copyright in China is more difficult.
Purported copies of SATs previously given in the United States, for instance, have been advertised for sale on Chinese websites such as Taobao, a popular shopping site run by e-commerce giant Alibaba. A recent search of the site turned up at least 11 vendors offering dozens of versions of the SAT over the past few years. Some of the ads included photos of purported test booklets. Other advertisers claimed to have the correct answers to past exams.
Asked about the ads, an Alibaba spokesman said it bars the listing of products that violate copyright. “We are currently taking steps to remove the infringing listings” and discipline the sellers, the spokesman said.
NEW TEST, OLD PROBLEMS
The debut of the completely redesigned SAT means that old jijings, the study guides filled with past test material, are obsolete.
This month, the College Board announced it was prohibiting non-students from taking the first sitting of the new exam in the United States. The measure was intended “to prevent security violations.” As a result, prep-center tutors from abroad who had registered to take the test were prevented from directly gathering intelligence on the new SAT’s content. The policy will apply to most exam dates.
Other vulnerabilities remain. The ETS security staff is small: Although it sometimes hires contractors, about two dozen ETS security staffers handle College Board matters worldwide.
ETS last year introduced lock boxes to safeguard exams, but the system doesn’t always work. After the SAT was given in Taiwan last June, staffers at the Taipei European School put the test booklets and answer sheets in lock boxes and sent them to a local ETS shipping agent. By the time the boxes reached the United States, two booklets were missing, according to emails between the school and ETS.
The boxes don’t cover the entire supply chain. Two administrators who oversee SAT exams in the United States said they weren’t given the boxes at their high schools. An ETS spokesman said the boxes have been introduced “in select locations.”
Perhaps the biggest weakness is the College Board’s plans to keep recycling versions of the new test. The risks were evident on March 5, the day the re-engineered SAT debuted in the United States.
After the exam, American test-takers went online to discuss the new test in detail. And Asian test-prep companies mobilized to take advantage.
This story was reported from Shanghai, Hong Kong, Seoul, Edinburgh, Los Angeles, Boston, New York and Lawrence, Kansas. Additional reporting by Ju-min Park, James Pomfret, Alex L. Dobuzinskis, Rebecca Jang, Benny Kung, and the Shanghai newsroom.
Next: The assault on the new SAT
FAQ about the SAT
By Renee Dudley and Steve StecklowQ: What’s on the new SAT?
A: The test is in three main parts – a math section; a reading and writing section; and an optional essay. The math section has 58 questions. The reading portion has 52 questions; the writing-and-language portion has 44 questions. The math and verbal sections are each scored on a scale of 200 to 800, so a perfect score is 1600. The core test is three hours long. The optional essay test, graded on a different scale, is 50 minutes long.
Q: Does everyone who takes the SAT on a given date receive the exact same test?
A: Not always: Multiple versions of the exam might be used on a test day.
Q: Are fresh versions of the exam used for each sitting of the SAT?
A: The College Board, the organization that owns the SAT, told Reuters that it will reuse portions of tests overseas at some point after those tests have been given in North America. This has been the prior practice as well.
Q: Has the College Board recycled tests in America, too?
A: Yes. One example: The test given in America in January contained some portions of the U.S. exam that was administered in June 2014. But interviews and an analysis of online discussions by test-takers indicate the College Board does not recycle SATs as frequently in the United States as it has abroad.
Q: How much material is reused? Entire tests? Just individual test questions?
A: Sometimes the College Board recycles one or more sections of tests. Sometimes it reuses smaller portions. College Board officials won’t say much about this for fear of helping content thieves or cheaters.
Q: Why does reusing parts of tests present security problems?
A: Exam booklets are sometimes stolen. Also, test-takers frequently go online to discuss questions immediately following the SAT. When test material leaks, and if that material is later reused, students potentially can preview actual test questions before an exam. Many Asian test-prep companies specialize in collecting material from prior tests and packaging it for their clients.
Q: How do the Asian test-cram schools learn what was on past exams?
A: They might harvest Internet chatter from discussion groups on U.S. websites like College Confidential or reddit, where students who’ve taken the exam discuss questions and answers. Cram schools also interview students who took the exam or send their own staff members to take the test.
Q: Doesn’t the College Board prohibit kids from talking about what’s on the test?
A: It instructs them not to discuss any of the material afterward. But students routinely ignore this and talk about math problems, reading passages, questions and answers in online forums. They also sometimes try to recreate the test from memory in shared documents.
Q: And how do cram schools know which material will be reused?
A: The College Board regularly releases certain exams for practice. Those don’t get used again. But other past exams could be reused. That’s the material the cram schools focus on teaching.
Q: How does this information actually help a cram school’s students?
A: Schools are able to provide those students with reading passages, test questions and answers that may appear on upcoming tests. Some create answer keys for prior SAT sections. Past SAT booklets, sometimes with the answers, also are offered for sale in Asia.
Q: I can see how getting the correct answers in advance would help – that’s obvious. But how does it help if you simply see questions in advance?
A: Students who get an advance look at questions may gain a distinct edge. A 1984 study by ETS, the SAT’s security contractor, found that test-takers who repeat the same version of a standardized exam perform much better the second time than students who take different iterations of the test. The research focused on the GRE – an admissions exam for graduate school. Neal Kingston, one of the study’s authors, says he expects the findings also hold true for the SAT. Even a single extra right answer can boost a score by 10 or 20 points on an SAT section’s 800-point scale, according to a practice-exam scoring table posted by the College Board.
Q: How does it help if all you get is an advance look at a reading passage, without seeing the questions beforehand?
A: Students who’ve seen reading passages in advance can gain a deeper understanding of the material. They also can spend more time answering the related questions, or working on other parts of the test section.
Q: Why doesn’t the College Board just stop recycling?
A: Cost is one issue. The College Board says the cost of using an exam only once would be passed onto test-takers, who could end up paying more than double the current fee of up to $54.50.
Q: How many students take the SAT internationally?
A: In the 2013-14 school year, there were 206,000 SAT test-takers overseas, according to the College Board. Of those, 64,000 were from East Asia and the Pacific region, including 29,000 from China. The College Board does not provide comparable data for test-takers in the United States.
—————
Multiple Choices
By Renee Dudley, Steve Stecklow, Alexandra Harney and Irene Jay Liu
Graphics: Matthew Weber
Video: Adam Wiesen
Photographs: Bobby Yip
Edited by Blake Morrison
