Web wide crawl with initial seedlist and crawler configuration from March 2011. This uses the new HQ software for distributed crawling by Kenji Nagahashi.
What?s in the data set:
Crawl start date: 09 March, 2011
Crawl end date: 23 December, 2011
Number of captures: 2,713,676,341
Number of unique URLs: 2,273,840,159
Number of hosts: 29,032,069
The seed list for this crawl was a list of Alexa?s top 1 million web sites, retrieved close to the crawl start date. We used Heritrix (3.1.1-SNAPSHOT) crawler software and respected robots.txt directives. The scope of the crawl was not limited except for a few manually excluded sites.
However this was a somewhat experimental crawl for us, as we were using newly minted software to feed URLs to the crawlers, and we know there were some operational issues with it. For example, in many cases we may not have crawled all of the embedded and linked objects in a page since the URLs for these resources were added into queues that quickly grew bigger than the intended size of the crawl (and therefore we never got to them). We also included repeated crawls of some Argentinian government sites, so looking at results by country will be somewhat skewed.
We have made many changes to how we do these wide crawls since this particular example, but we wanted to make the data available ?warts and all? for people to experiment with. We have also done some further analysis of the content.
If you would like access to this set of crawl data, please contact us at info at archive dot org and let us know who you are and what you?re hoping to do with it. We may not be able to say ?yes? to all requests, since we?re just figuring out whether this is a good idea, but everyone will be considered.
ON a Monday morning in February 1997, a taxi left the Royal Free hospital, in Hampstead , northwest London. It turned out of the car park and headed to the renowned Institute of Cancer Research, six miles southwest in Fulham.
In the back of the cab sat a California businessman, whose commercial interests lay in electroplating, but whose personal crusade was autism. On his lap was a plastic pot, in which snips of human tissue floated in protective formalin.
The snips were biopsies taken from the gut of the man’s five-year-old son, then a patient on the hospital’s Malcolm ward. The boy, Child Eleven, as he is known to protect his privacy, had been enrolled in a programme to investigate alleged risks of the three-in-one measles, mumps and rubella (MMR) vaccine.
“I’m an engineer,” said Mr Eleven. “And my doctor here [in California] suggested I should cross-check the Royal Free’s results with another lab. Just to be sure.”
Child Eleven was one of a dozen children who were enrolled in the programme at the hospital. Its research caused one of the biggest stirs in modern medical history when its results were published in The Lancet medical journal. The five-page paper suggested a potential link between MMR and what the doctors called a “syndrome” of autism and inflammatory bowel disease.
The children were not named in the tables of results. Eleven boys and one girl, aged between 2½ and 9½, were said, for the most part, to have a diagnosis of regressive autism, where children appear to develop quite normally, but then, terrifyingly, lose their language skills. The bowel disease was described as nonspecific colitis, a severe form of inflammation.
The dynamite in The Lancet was the claim that their conditions could be linked to the MMR vaccine, which had been given to all 12 children.
According to the paper, published on February 28, 1998, the parents of eight of the children said their “previously normal” child developed “behavioural symptoms” within days of receiving the jab.
“In these eight children the average interval from exposure to first behavioural symptoms was 6.3 days,” said the paper.
At face value, these findings were more than grounds for the panic that took off over MMR. If such startling results were obtained from two-thirds of a group of previously normal children turning up at one clinic at just one hospital, what might be happening, unreported, all over the world? This might be the first snapshot of a hidden catastrophe, a secret epidemic of vaccine damage.
To launch the findings, the Royal Free held a press conference, and issued a video news release. The researchers’ leader, Dr Andrew Wakefield, then 41, was emphatic in his comments to the assembled media.
“It’s a moral issue for me,” he said. “I can’t support the continued use of these three vaccines, given in combination, until this issue has been resolved.”
Eleven years later, the fallout continues around the world. The paper triggered a public health crisis. In Britain, immu-nisation rates collapsed from 92% before the Lancet paper was published, to 80% at the peak of Britain’s alarm. Measles has returned as officially “endemic”.
With less than 95% of the population vaccinated, Britain has lost its herd immunity against the disease. In 1998 there were 56 cases reported; last year there were 1,348, according to figures released last week that showed a 36% increase on 2007. Two British children have died from measles, and others put on ventilators, while many parents of autistic children torture themselves for having let a son or daughter receive the injection.
“There’s not a day go by I don’t cry because of what happened,” said the mother of a severely disabled 12-year-old girl. “I shouldn’t have took her [for the MMR], and you know everyone will say, ‘Don’t blame yourself’, but I do. I blame myself.”
Yet the science remains a problem. No researchers have been able to replicate the results produced by Wakefield’s team in the Lancet study.
Some used statistics to see if autism took off in 1988, when MMR was introduced. It did not. Others used virology to see if MMR caused bowel disease, a core suggestion in the paper. It did not. Yet more replicated the exact Wakefield tests. They showed nothing like what he said.
Wakefield himself, however, stands by his results, insisting that a link between MMR and autism merits inquiry. The 12 other doctors whose names were attached to the Lancet paper, which was written by Wakefield, were not involved in preparing the data used.
“This study created a sensation among the public that was impossible to counter, despite overwhelming evidence to the contrary,” says Professor Gary Freed, director of the child health research unit at the University of Michigan, who has watched the scare take off in America.
“Overwhelming biologic and epidemiologic evidence has demonstrated conclusively that there is no association between the MMR vaccine and autism, and yet this thing goes on.”
Aspects of the project are now before the General Medical Council (GMC), the doctors’ disciplinary body. Wakefield and two professors, John Walk-er-Smith, 72, and Simon Murch, 52, are charged with carrying out unauthorised research on the 12 children. The charges, which they strongly deny, relate to the ethics of the treatment of the 12 children, not the results of the research.
In evidence presented to the GMC, however, there has emerged potential explanations of how Wakefield was able to obtain the results he did. This evidence, combined with unprecedented access to medical records, a mass of confidential documents and cooperation from parents during an investigation by this newspaper, has shown the selective reporting and changes to findings that allowed a link between MMR and autism to be asserted.
MR ELEVEN’S taxi dash was a small ride in his desperate quest to find an answer for his son’s condition. Today, Child Eleven is much improved: at 17, he is a terrific scholar, although too nervous to drive.
The extra tests on his biopsies produced striking results. His father asked the cancer institute to look for the measles virus, which lay at the heart of Wakefield’s concerns over the vaccine. According to a theory that underpinned the project, this virus in MMR was the cause of bowel disease, which then did damage to children’s brains.
“It took a big fight to get the information,” said Mr Eleven. “They told me there was no measles virus. I had the tests repeated three times at different labs in the US, and they all came back negative.”
This struck a different note from what Wakefield suggested when describing his research to the world.
“We would not have presented this paper to The Lancet had we not undertaken extensive virological studies already,” he told the 1998 press conference.
At face value, this is an anomaly. In science, however, these are endless and can sometimes eventually be explained. This is why studies are usually repeated. But at the heart of Wakefield’s findings The Sunday Times found more discrepancies, inconsistencies and changes.
The first, in the Lancet tables, concerned the first child in the paper: Child One, from Cottesmore, Leicestershire. He was 3½ years old and the son of an air force pilot. In November 1995, his parents had been devastated after receiving a diagnosis of autism.
“Mr and Mrs [One]’s most recent concern is that the MMR vaccination given to their son may be responsible,” their GP told the hospital in a letter.
In the paper this claim would be adopted, with Wakefield and his team reporting that Child One’s parents said “behavioural symptoms” started “one week” after he received the MMR.
The boy’s medical records reveal a subtly different story, one familiar to mothers and fathers of autistic children. At the age of 9½ months, 10 weeks before his jab, his mother had become worried that he did not hear properly: the classic first symptom presented by sufferers of autism.
Child One was among the eight reported with the apparent sudden onset of the condition. So was the next child to be admitted.
This was Child Two, an eight-year-old boy from Peter-borough, Cambridgeshire, diagnosed with regressive autism, which, according to the Lancet paper, started “two weeks” after his jab.
However, this child’s medical records, backed by numerous specialist assessments, said his problems began three to five months later.
The difference between 14 days and a few months is significant, according to experts. Autism usually reveals itself in the second year of life, when the vaccine is routinely given. If there was no sudden onset after the MMR injection, as claimed for the “syndrome”, the condition could be ascribed to a conventional pattern.
More apparent anomalies lurked among the following 10 children, as they arrived at the Royal Free hospital between September 1996 and February 1997.
Only one was a girl, Child Eight, aged 3, from Whitley Bay, Tyne & Wear. She was reported in the journal as having suffered a brain injury “two weeks” after MMR.
Her medical records did not support this. Before she was admitted, she had been seen by local specialists, and her GP told the Royal Free of “significant concerns about her development some months before she had her MMR”.
Child Six, aged 5, and Child Seven, aged 3, were said to have been diagnosed with regressive autism, with an onset of symptoms “one week” and “24 hours” after the jab respectively.
But medical records show that neither boy was “previously normal”, as the Lancet article described all the children, and that both had already been hospitalised with brain problems before their MMR.
Child Six received his vaccine at the age of 14 months, but had twice previously been admitted with fits.
Child Seven was given his at the age of 20 months but, again, problems already showed.
“He developed well, had social smiling and was responsive to his mother,” a psychia-trist wrote. “But he began to have pale episodes and ? [sic] petit mal [convulsions], and had an EEG [an electroencephalogram, a common test for epilepsy] done at 15 months, which was abnormal.”
Meanwhile, neither was diagnosed with regressive autism, or even nonregressive classical autism. Three of the children had been diagnosed with Asperger’s disorder, in which language is not lost, and which is not regressive: nothing like what afflicted One and Two. This was also the diagnosis for Child Twelve in the series, a six-year-old boy from Burgess Hill, West Sussex.
And Seven would be diagnosed with an odd behavioural condition called “pathological demand avoidance syndrome”. This usually manifests as social manipulativeness, and is nothing like the “syndrome” being claimed. It is sometimes marked by a child putting his hands on his ears, while singing “lah-lah-lah, can’t hear you”.
WHEN the children first arrived at the Royal Free, in addition to autism, they were also reported with constipation, diarrhoea or other common bowel complaints. This was the reason given for them travelling between 60 and 5,000 miles to London to enter the care of Wakefield’s team.
Wakefield, now 52, a former gut surgeon, was at the time doing academic research in the Royal Free’s medical school on Crohn’s disease, an ulcerating inflammation. In 1995, he had developed a theory that this condition was caused by the measles virus, which is found live in MMR. The theory has since been discounted.
This work was the bedrock on which he based his new claims. Yet this too appears problematic. The children were supposed to have a new inflammatory bowel disease, written up in the Lancet paper as “consistent gastrointestinal findings” involving “nonspecific colitis”. Wakefield said that this inflammation of the colon caused the gut to become “leaky”, allowing food-derived poisons to pass into the blood-stream and the brain.
“The uniformity of the intestinal pathological changes and the fact that previous studies have found intestinal dysfunc-tion in children with autistic-spectrum disorders, suggests that the connection is real and reflects a unique disease process, ” the Lancet Paper explained of the “syndrome”.
Yet pathology records of samples taken from the children show apparent problems with this evidence. The hospital’s consultants who took biopsies from the children’s colons concluded that they were not uniform but varied and unexceptional.
For Child Eight, the pathology report said: “No abnormality detected”, while the Lancet paper said: “Nonspecific colitis”. This pattern was repeated for two of the other children, Nine and Ten.
The most striking change of opinion came in the case of Child Three, a six-year-old from Huyton, Merseyside. He was reported in the journal to be suffering from regressive autism and bowel disease: specifically “acute and chronic nonspecific colitis”. The boy’s hospital discharge summary, however, said there was nothing untoward in his biopsy.
A Royal Free consultant pathologist questioned a draft text of the paper. “I was somewhat concerned with the use of the word ‘colitis’,” Susan Davies, a co-author, told the ongoing GMC inquiry into the ethics of how the children were treated, in September 2007.
“I was concerned that what we had seen in these children was relatively minor.”
However, after her challenge, it was explained, Wakefield’s team met for a “research review” of the biopsies. It was not an unusual move for a group of specialists to reconsider the evidence upon which their research was relying. It was nevertheless striking that their conclusion was that 11 of the children’s bowels were in fact diseased when their colleagues had found no abnormalities in at least seven of the cases.
Further questions arise about the motivations of Wakefield. Five years ago this month, The Sunday Times reported that he worked for lawyers, and that many of the families were either litigants or were part of networks through which they would sue. Far from routine referrals, as they appeared, many of them had made contact with one another.
Child Six and Child Seven were brothers from East Sussex; Child Four, a 9½-year-old from North Shields, Tyneside, was registered with the same GP as Child Eight. In short, the 12, none of whom came from London, fetched up far-from-routinely at the hospital.
The mothers of Child Two and Child Three told me what others said in medical records: they had heard of Wakefield through the MMR vaccine campaign, Jabs. Thus, when they arrived on Malcolm ward, and produced the “finding” about MMR, it was by no means a random sample of cases.
What parents did not know was that, two years before, Wakefield had been hired by Jabs’s lawyer, Richard Barr, a high-street solicitor in King’s Lynn, Norfolk. Barr had obtained legal aid to probe MMR for any evidence that could be used against the manufacturers. He is adamant that at all times he acted professionally, and diligently represented his clients.
A string of Sunday Times reports have exposed how Wakefield earned £435,643 through his work with Barr, plus funding to support his research.
There is no suggestion the other doctors knew of Wakefield’s involvement with Barr.
What has not been reported is that the nature of the project had been visualised before any of the children were even admitted to the Royal Free.
In June 1996 – the month before Child One’s arrival at the hospital – Wakefield and Barr filed a confidential document with the government’s Legal Aid Board, appearing already to know of a “new syndrome”.
Referring to inflammatory bowel disease, and then bowel problems with autism, Wakefield and Barr wrote to the board, successfully seeking money.
“The objective,” they wrote, “is to seek evidence which will be acceptable in a court of law of the causative connection between either the mumps, measles and rubella vaccine or the measles/rubella vaccine and certain conditions which have been reported with considerable frequency by families who are seeking compensation.”
Twenty months later, the Royal Free team delivered with the paper that had found a “new syndrome”. TODAY, the 12 children are mostly teenagers. At least three are bloggers, two in support of Wakefield, while others have limited skills. The wrongful stigma of disability hangs heavy on most, and heaviest on the families with the misguided burden of guilt that the vaccine scare has visited on them.
Wakefield has left Britain to live in Austin, Texas, where he runs a clinic offering colono-scopies to American children. He tours the country, giving lectures and speeches against the vaccine, and attracting a loyal following of young mothers.
In Wakefield’s view, the Lancet paper was accurate, including reasonable reassessment of findings. Other doctors, including an experienced pathologist concurred with his judgment on the revised reports of nonspecific colitis, he has said.
Behavioural diagnoses, meanwhile, involved a confusing array of technical names, and he trusted what the parents told him. The fact that they said the problems followed MMR implied that regression was involved.
When our allegations were put to him last week, he did not respond, but his lawyers replied on his behalf. They said the GMC hearings were nearing conclusion and our revelations risked prejudicing these proceedings.
“You also know that, at this juncture in the GMC process, it would be inappropriate for Dr Wakefield to give a detailed response to you,” they said. “He has denied the allegations and gave a detailed response over many days to the GMC panel.”
Many of the parents of the original 12 children continue to support him and campaign vigorously on his behalf. But others whose children took part in the Lancet project are too burdened and traumatised for campaigning.
One mother told me that, before her son’s MMR jab, he could say “night, night mummy”, but all language slipped away “some time” after the injection. To this day, she remains convinced it was the vaccine that did it. She believes it was the rubella component.
When asked why his parents took him to the Royal Free, his father answered: “We were just vulnerable. We were looking for answers.”
Contact us | Terms and Conditions | Privacy Policy | Site Map | FAQ | Syndication | Advertising
© Times Newspapers Ltd 2010 Registered in England No. 894646 Registered office: 3 Thomas More Square, London, E98 1XY
