The Internet Archive’s goal, according to its website, is “universal access to all knowledge.” As part of that mission, the non-profit runs the Wayback Machine, an online tool that anyone can use to digitally preserve a snapshot of a website. It provides an important public service, in that if a company tries to quietly change its policy, or perhaps a government tries to scrub a position from its website, the Wayback Machine can provide robust proof of the switch.
But the Internet Archive has been purging its banks of content related to a company which marketed powerful malware for abusive partners to spy on their spouses.
The news highlights the broader issue of the fragility of online archives, including those preserving information in the public interest.
“Journalists and human rights defenders often rely on archiving services such as the Wayback Machine as tools to preserve evidence that might be key to demand accountability,” Claudio Guarnieri, a technologist at human rights charity Amnesty International, told Motherboard in an online chat.
The company in question is FlexiSpy, a Thailand-based firm which offers desktop and mobile malware. The spyware can intercept phone calls, remotely turn on a device’s microphone and camera, steal emails and social media messages, as well as track a target’s GPS location. Previously, pages from FlexiSpy’s website saved to the Wayback Machine showed a customer survey, with over 50 percent of respondents saying they were interested in a spy phone product because they believe their partner may be cheating. That particular graphic was mentioned in a recent New York Times piece on the consumer spyware market.
In another example, a Wayback Machine archive of FlexiSpy’s homepage showed one of the company’s catchphrases: “Many spouses cheat. They all use cell phones. Their cell phone will tell you what they won’t.”
Now, those pages are no longer on the Wayback Machine. Instead, when trying to view seemingly any page from FlexiSpy’s domain on the archiving service, the page reads “This URL has been excluded from the Wayback Machine.” (After Motherboard published a series of articles about the consumer spyware market, FlexiSpy purged its own website of content relating to illegal spying on spouses.)
Caption: A screenshot of a FlexiSpy survey previously available on the Wayback Machine. Image: Screenshot
“URLs are inherently unstable over time. Governments change, and so does their online presence. Companies go down, and so do their websites. Internet Archive URLs used to be rock-solid reliable. That does not appear to be the case any longer,” Thomas Rid, a professor of strategic studies at Johns Hopkins University, told Motherboard in an online chat.
Many websites use so-called robot exclusion in an attempt to stop crawlers from archiving their content. Typically this will come in the form of a website owner uploading a simple text file called ‘robots.txt’, which tells bots to not collect data from this particular website.
“While we collect publicly available Internet documents, sometimes authors and publishers express a desire for their documents not to be included in the Collections (by tagging a file for robot exclusion or by contacting us or the original crawler group),” a section of the Internet Archive’s terms of use reads. The terms, however, do not say that the group will honor takedown requests, but that it might.
Got a tip? You can contact this reporter securely on Signal on +44 20 8133 5190, OTR chat on jfcox@jabber.ccc.de, or email joseph.cox@vice.com.
“If the author or publisher of some part of the Archive does not want his or her work in our Collections, then we may remove that portion of the Collections without notice,” the terms add. The Internet Archive has previously said it will not respect robots.txt files from U.S. military and government domains, although it may still respond to related removal requests.
Michael Nelson from the Web Science and Digital Libraries Research Group at Old Dominion University told Motherboard in an email that “for the Internet Archive to remove something from its web archive, there must have been legal pressure involved.” Nelson also said that the Wayback Machine typically indicates when information has been removed due to robots.txt, and that FlexiSpy’s syntax in its robots.txt file is “non-standard if not wrong.”
Multiple members of the Internet Archive did not respond to requests for comment and clarification as to why it purged the FlexiSpy archives. Because of this, it is not totally clear which mechanism—legal threats or otherwise—led to the Internet Archive’s actions.
FlexiSpy, however, claimed in a recent tweet it had approached the Internet Archive, seemingly to ask to have the company’s content removed.
“We’ve already written to archive.org explaining that you are acting like an obsessed pest—and to respect our right to have content removed,” FlexiSpy tweeted last week, in reference to Motherboard’s investigative series on the consumer spyware industry. FlexiSpy did not respond to a Twitter direct message asking if the company would share a copy of this letter to the Internet Archive.
“Commercial spyware companies attempting to remove historical records is just one example of why we need resilient archiving solutions, or at the very least clarity over the expectation of integrity and durability people should set for these services,” Guarnieri said.
Not all archiving services may respect a robot exclusion or takedown request. At the time of writing, archive.is, for example, has no problems creating a snapshot of FlexiSpy’s website.
“This represents a fragility with having a *single* web archive. We need diversity in technology stacks, geographic locations (e.g., having the IA [Internet Archive] along the San Andreas Fault), organizations, and jurisdictions,” Nelson said.
“The Internet Archive is an amazing organization and resource, but it is constantly under threat, and those threats will only increase as the role of web archives expands into our public and political sphere,” he added.
This issue of disappearing or unreliable archives is part of why Motherboard created a tool called mass_archive. This basic Python script pushes a single webpage to multiple archive services at once, meaning that if, say, the Wayback Machine won’t make a copy, perhaps another digital library will.
“The notion that the internet does not forget is utter nonsense—the public internet forgets every single day, like knowledge quietly dropping off a cliff into the dark sea of time,” Rid said.
ONE EMAIL. ONE STORY. EVERY WEEK. SIGN UP FOR THE VICE NEWSLETTER.
By signing up, you agree to the Terms of Use and Privacy Policy & to receive electronic communications from Vice Media Group, which may include marketing promotions, advertisements and sponsored content.
