Web wide crawl with initial seedlist and crawler configuration from March 2011. This uses the new HQ software for distributed crawling by Kenji Nagahashi.
What?s in the data set:
Crawl start date: 09 March, 2011
Crawl end date: 23 December, 2011
Number of captures: 2,713,676,341
Number of unique URLs: 2,273,840,159
Number of hosts: 29,032,069
The seed list for this crawl was a list of Alexa?s top 1 million web sites, retrieved close to the crawl start date. We used Heritrix (3.1.1-SNAPSHOT) crawler software and respected robots.txt directives. The scope of the crawl was not limited except for a few manually excluded sites.
However this was a somewhat experimental crawl for us, as we were using newly minted software to feed URLs to the crawlers, and we know there were some operational issues with it. For example, in many cases we may not have crawled all of the embedded and linked objects in a page since the URLs for these resources were added into queues that quickly grew bigger than the intended size of the crawl (and therefore we never got to them). We also included repeated crawls of some Argentinian government sites, so looking at results by country will be somewhat skewed.
We have made many changes to how we do these wide crawls since this particular example, but we wanted to make the data available ?warts and all? for people to experiment with. We have also done some further analysis of the content.
If you would like access to this set of crawl data, please contact us at info at archive dot org and let us know who you are and what you?re hoping to do with it. We may not be able to say ?yes? to all requests, since we?re just figuring out whether this is a good idea, but everyone will be considered.
The decision by ministers and the Serious Fraud Office to halt an investigation into alleged corruption in the BAE Systems arms deal with Saudi Arabia was unlawful, an “abject surrender” and a threat to the reputation of British justice, the High Court ruled yesterday.
In one of the most strongly worded judicial attacks on government action, judges condemned how ministers “buckled” to “blatant threats” that Saudi cooperation in the fight against terror would end unless the bribery investigation was dropped.
The ruling prompted calls for a full public inquiry and the reopening of the investigation. The SFO, which is now under pressure to review its decision, said last night that it was “carefully considering the implications of the judgment and the way forward”.
The judges said: “We fear for the reputation of the administration of justice if it can be perverted by a threat.” They said that any similar unlawful threats to the rule of law in the future must be resisted by Government – or the courts would intervene.
Lord Justice Moses and Mr Justice Sullivan, sitting in London, declared: “No one, whether within this country or outside, is entitled to interfere with the course of our justice. The rule of law is nothing if it fails to constrain overweening power.”
In a further condemnation of the outgoing SFO director, Robert Wardle, the judges added that he had “ceased to exercise the power to make the independent judgment conferred on him by Parliament”.
The ruling is an indictment of both Tony Blair and Lord Goldsmith, QC, the former Attorney-General, who argued for the dropping of the inquiry –and an embarrassemnt for the Government over one of Britain’s most sensitive diplomatic relationships.
The judges have yet to make an order but are likely to quash the “unlawful” original decision to halt the investigation in December 2006. A review of that decision could come back with the same result – but the SFO would have to put forward a fresh case to show that that on purely legal grounds alone the investigation cannot proceed.
The ruling was an extraordinary victory for the antibribery pressure group Corner House Research and the Campaign Against Arms Trade (CAAT).
Susan Hawley, of The Corner House, said: “This is a great day for British justice. The judges have stood up for the right of independent prosecutors not to be subjected to political pressure and they have made sure that the Government cannot use national security arrangements just because a prosecution is not in their interests.”
Symon Hill, of CAAT, said: “It has been clear from the start that the dropping of the investigation was about neither national security nor jobs. It was due to the influence of BAE and Saudi princes over the Government.”
The SFO investigation arose out of BAE’s £43 billion al-Yamamah arms deal with Saudi Arabia in 1985.
In December 2006, Lord Goldsmith announced that the investigation was to be discontinued after a decision by the SFO director. Tony Blair had told him that the Saudis had privately threatened to cut intelligence cooperation with Britain unless the inquiry was stopped.
He made representations to the Attorney-General that to continue the investigations risked damaging Saudi confidence in Britain, to national security and to lives.
Mr Wardle met government law officers days later and reported on representations that he had received from the Saudi Ambassador. Later the same day, he told the Attorney that the investigation should be halted because of the threat to lives and national security.
The judges ruled that Mr Wardle “was required to satisfy the court that all that could reasonably be done had been done to resist the threat. He has failed to do so.”
The judges said: “The defendant [SFO director] in name, although in reality the Government, contends that the director was entitled to surrender to the threat and that while a matter of regret, such threats were ‘a part of life’. So bleak a picture of the impotence of the law invites at least dismay, if not outrage.”
The Government had “failed to recognise that the threat uttered was not simply directed at this country’s commercial, diplomatic and security interests; it was aimed at its legal system.
“Had such a threat been made by one who was subject to the criminal law of this country, he would risk being charged with an attempt to pervert the course of justice.”
What the judges said
“ There is no evidence. . . that any consideration was given as to how to persuade the Saudis to withdraw the threat, let alone any attempt made to resist the threat We fear for the reputation of the administration of justice if it can be perverted by a threat No one, whether within this country or outside, is entitled to interfere with the course of our justice It is obvious . . . that the decision to halt the investigation suited the objectives of the executive [the Government]. Stopping the investigation avoided uncomfortable consequences, both commercial and diplomatic Surrender merely encourages those with power . . . to repeat such threats, in the knowledge that the courts will not interfere with the decision of a prosecutor to surrender It is not difficult to imagine what they [the Saudis] would think if we attempted to interfere with their criminal justice system”
Contact us | Terms and Conditions | Privacy Policy | Site Map | FAQ | Syndication | Advertising
© Times Newspapers Ltd 2010 Registered in England No. 894646 Registered office: 3 Thomas More Square, London, E98 1XY