COLLECTED BY

Organization: Internet Archive

The Internet Archive discovers and captures web pages through many different web crawls. At any given time several distinct crawls are running, some for months, and some every day or longer. View the web archive through the Wayback Machine.

Collection: Wide Crawl started March 2011

Web wide crawl with initial seedlist and crawler configuration from March 2011. This uses the new HQ software for distributed crawling by Kenji Nagahashi.

What’s in the data set:

Crawl start date: 09 March, 2011
Crawl end date: 23 December, 2011
Number of captures: 2,713,676,341
Number of unique URLs: 2,273,840,159
Number of hosts: 29,032,069

The seed list for this crawl was a list of Alexa’s top 1 million web sites, retrieved close to the crawl start date. We used Heritrix (3.1.1-SNAPSHOT) crawler software and respected robots.txt directives. The scope of the crawl was not limited except for a few manually excluded sites.

However this was a somewhat experimental crawl for us, as we were using newly minted software to feed URLs to the crawlers, and we know there were some operational issues with it. For example, in many cases we may not have crawled all of the embedded and linked objects in a page since the URLs for these resources were added into queues that quickly grew bigger than the intended size of the crawl (and therefore we never got to them). We also included repeated crawls of some Argentinian government sites, so looking at results by country will be somewhat skewed.

We have made many changes to how we do these wide crawls since this particular example, but we wanted to make the data available “warts and all” for people to experiment with. We have also done some further analysis of the content.

If you would like access to this set of crawl data, please contact us at info at archive dot org and let us know who you are and what you’re hoping to do with it. We may not be able to say “yes” to all requests, since we’re just figuring out whether this is a good idea, but everyone will be considered.

TIMESTAMPS

Dictionary definition of “haji”

haji

n. an Iraqi; any Muslim, Arab, or native of the Middle East. Also hadji; attrib. Subjects: English, Iraq, United States, Military, Slang

Editorial Note: Haji is an honorific given to Muslims who have made a pilgrimage to Mecca, but the word often takes on a pejorative sense when used by American military personnel. The plural is also sometimes haji, without a terminating “s.” Etymological Note: < Arabic haji ‘pilgrim.’

2 Comments | Cites | Permalink | Tell a Friend

Citations: 2003 [Saddam_ized] @ Tampa Bay, Fla. Usenet: soc.culture.iraq (Mar. 30) “Re: Amercans, you idiot”: Learn to type correctly Hadji—we will conduct the war on OUR terms not yours. You are obviously one of the terrorists. 2003 American Enterprise (June 1) “Dust and stars: An Iraq war journal” (in Tallil Air Base, South-Central Iraq) vol. 14, no. 4, p. 18-55: Patrols wearing NODs will watch our perimeter all night; they chamber cartridges as they move out. If nature calls, don’t go out far or you could be mistaken for a haji. 2003 Andrew England @ Baghdad, Iraq (AP) (Sept. 9) “Iraqis say it will take more than words and international troops to solve their problems”: Some have grown to despise Iraqis, whom they call “Hajis,” scowling rather than waving as they pass locals along highways and dirt roads. 2004 Colin Campbell Usenet: us.military.army (May 25) “Re: crocodile tears (was Re: gotta find a scapegoat to keep the ragheads happy)”: To me, the ordinary Iraqi people count as “people” the thugs trying to set themselves up as warlords are “Hadji” or “enemy.” I am not here to be politically correct. 2004 Douglas Jehl, Andrea Elliott @ Afghanistan New York Times (May 29) “Cuba Base Sent Its Interrogators To Iraqi Prison” p. 1: We were pretty much told that they were nobodies, that they were just enemy combatants. I think that giving them the distinction of soldier would have changed our attitudes toward them. A lot of it was based on racism, really. We called them hajis, and that psychology was really important. 2004 John Koopman @ Ramadi, Iraq San Francisco Chronicle (California) (July 9) “Trolling for bombs”: There was little left in the bomb crater. The explosive had been placed inside a tiny roadside market—a “haji stand” in GI parlance—which was closed when the bomb went off. 2004 Jimmy Schaffer @ Taji, Iraq Uncool Adventures (Derby, Kan.) (July 11) “Everyday’s a holiday”: Hadji—A term used for any local national, primarily those that are dumber than a box of rocks. 2004 Gordon Lubold @ Camp Hurricane Point, Iraq Marine Times (July 17) “Marines in Iraq: Reporter’s notebook; But it’s a dry heat”: It’s SO hot that if I wash my clothes in the Haji washing machine (that’s shorthand for anything bought locally, and by the way, “Haji,” Lloyd tells me, is not a pejorative term since it means “wise man.” Marines use the term as a prefix for anything. “Haji-movies,” “hajified,” you name it.) 2004 Steve Smith @ FOB Bernstein, Iraq ArmySteve (Nov. 12) “Fobbits and other Iraqi critters”: Many haji shops have signs out front of them that read “hajee shop” so it seems at least these proprietors don’t mind the term. 2004 WorldNetDaily (U.S.) (Nov. 27) “Combat in Iraq: A ground-level view” (in Iraq): I can recognize a Sadiqqi (Arabic for friend) from a Haji (Arabic word for someone who has made the pilgrimage to Mecca, but our word for a bad guy). 2004 Brita Brundage Fairfield County Weekly (Conn.) (Dec. 2) “Commando Correspondents”: Outside the base, soldiers freely shopped at a marketplace set up by Iraqis (derisively called “Haji Marts” by soldiers) that sold everything from alcohol to bootleg DVDs to bayonets. 2005 Paul Rockwell Online Journal (Apr. 1) “Army reservist witnesses war crimes”: “Hajji” is the new slur, the new ethnic slur for Arabs and Muslims. It is used extensively in the military. The Arabic word refers to one who has gone on a pilgrimage to Mecca. But it is used in the military with the same kind of connotation as “gook,” “Charlie,” or the n-word. Official Army documents now use it in reference to Iraqis or Arabs. It’s real common. There was really a thick aura of racism. 2005 Grace Leonhart Hope Star (Ark.) (July 6) “Driver sees all sides of humanity in Iraq”: I saw Haji (another name for Iraqis) successfully load two full-grown camels in a Ranger pickup without sideboards.…It was amazing. They made the first one lay down and folded his legs up and when he was in, they put the other one in.

Posted 10 Jul 04 | Permalink | Tell a Friend

Reader comments:

haji is not a pejorative, but a multi-faceted description of the iraqi world and war i was in for 18 months. i am very aware of racial insensitivity - arabs used to be called sand niggers or dune coons, even diaper heads or camel jockeys. these words are now archaic and impolite in a politically incorrect profession- warrior/infantry. indeed i was the EO [race relations] nco for our battalion [seen it all and heard it all]. haji can be the bad guys, the insurgents “haji’s gonna light you up” or children waving at you “some haji kids up ahead” Also haji shops or haji coke or haji dvd. i equate it to calling europeans, in regard to their country - jack, pierre, fritz, ivan. maybe even japanese dudes as fuji. “my name isn’t fuji, its hoshino” “ok, have a beer hoshino” i had friends in iraq- aziz, habib, horsham [kurd]etc. they never took offense to haji, and we discussed the world, from family to american imperialism [which many in this world consider a natural result of being strong] [not me] i never saw a black or hispanic or white or polynesian or asian US brother take offense to haji- i was with the 27th inf out of hawaii, talk about your rainbow. take care and i thought you’d like to hear it from the horse’s mouth.

ps. there is a lot worse going on over there than innocuous nicknames. do not make a PC mountain out of a molehill, or call a haji “ali-baba”... .. or me a “pork-eating flatlander” i live in the mountains and eat elk

by rich 23 Jun 07, 0158 GMT

I thought the proper term was “hadji” and it refers not to Iraqis or even Muslims but was historically used to reference Hindu people of India.
Not until the Iraq war did people use it to refer to people of Iraq (as seen in HBO’s Generation Kill series).

by MarthaK 25 Aug 08, 0413 GMT