Starting in 1996, Alexa Internet has been donating their crawl data to the Internet Archive. Flowing in every day, these data are added to the Wayback Machine after an embargo period.
Crawler-based search engines automatically visit Web pages to compile their listings. This means that, unlike directories, you are likely to have several, if not many, pages listed with them. This also means that by taking care in how you build your pages, you might rank well in crawler-produced results.
Optimizing pages for crawlers is covered in more detail in the Optimizing For Crawlers section of Search Engine Watch. If you have time, it is recommended that you read the pages in this section, especially the Search Engine Placement Tips page. However, by simply following the submission tips below, you can at least get your pages listed with crawlers, where they might naturally rank well for certain terms.
Submitting To Google
One of the most important crawler-based search engines is Google, because many people search there, plus it "powers" the main results of several other services (see the Search Engine Results Chart for an explanation of what "main" results are).
The absolute best way to get listed with Google is to get links to your Web site from other sites. Indeed, this is the best way to get listed for free with all the major crawlers listed on this page. Crawlers follow links, so if you have good links pointing at your Web site, the crawlers are more likely to find and include your pages in their databases.
Here's the good news: if you submitted your site to the major directories and got listed with one of them, then Google and other crawlers will almost certainly pick up the URL that was listed. This means you may not need to do additional work to get listed with crawlers.
Nevertheless, you may find it helpful to do some link building beyond the directories. Basic tips on building good links are covered on the Search Engine Placement Tips page. The Link Analysis & Link Building page available to Search Engine Watch members provides in-depth advice on building relevant links to your Web site. Consider reading what's covered on one or both of these pages.
The Submitting & Encouraging Crawlers page available to Search Engine Watch members also has advice on how your site architecture can be improved to naturally allow more of your pages to be added by crawlers.
Aside from link building, Google provides an Add URL page that lets you submit a URL directly to its crawler.However, there's no guarantee that Google will actually include a URL submitted to it this way. Despite this, it makes sense to submit your home page and perhaps one or two other URLs from "inside" your Web site via the Add URL page.
In addition to the ADD URL page, Google offers other options to submit a large list of URLs (a sitemap file) and verify ownership of your site within Google Webmaster Central for faster indexing, free of charge.
Once you have completed one of the options above, you really don't need to submit more than this. The only reason for submitting some of your inside pages is to give Google an alternate route into your site in case there is a problem reaching your home page. From whatever page it visits, Google will look for links to other pages within your site, perhaps including those in its index. This is true for other crawlers, as well.
If you have a brand new Web site, it will probably take about a month before Google lists your web pages naturally. Because of this, you might consider making use of its paid placement program Google AdWords, which is covered in the next part of this guide.
Finally, Search Engine Watch members have access to a detailed Guide to Google page that guides you even more through the process of submitting to the crawler and ranking well within its results. To learn more about becoming a member to access this information, visit the membership information page.
Submitting To Yahoo
Yahoo is an important crawler-based search engine because many people use the Yahoo search function, which provides the main results of several other services (see the Search Engine Results Chart for more about this).
As with Google, building links is the best way to get listed for free in Yahoo. Yahoo also offers a free URL submission form that you'll find listed on this page. Submit according to the same instructions as given for Google, above.
What if you aren't picked up for free? Yahoo has paid inclusion programs that guarantee to add the pages you submit quickly. The downside to these programs is that you'll be charged every time someone clicks on your listing. If you run out of money, potentially, your listing may be dropped. However, there's still a chance that even if you run out of money, you might continue to be listed for free.
Confused? You're to be forgiven, if so. The programs are so complex that we do not recommend beginners bother with them. Instead, there's a good chance that many pages in your site will just naturally get listed for free.
By the way, Yahoo's crawler incorporates technology from three different crawlers that it purchased in 2002 and 2003: Inktomi, AltaVista and FAST's AllTheWeb. However, any references you hear about those crawlers are superceded by the single Yahoo crawler.
Submitting To Ask
Ask.com search results have been powered by the Teoma ExpertRank algorithm since 2001 (see the Search Engine Results Chart).
Ask has no free Add URL page and no longer offers a paid site submission service. However, the Ask Jeeves/Teoma agent automatically crawls the Web, so if you have links pointing at your site from other sites, it will get included naturally.
Basic tips on building good links are covered on the Search Engine Placement Tips page, while the Link Analysis & Link Building page available to Search Engine Watch members provides in-depth advice on building relevant links to your Web site. Consider reading what's covered on one or both of these pages, to help yourself with Ask.
Submitting To Microsoft Live Search
Microsoft Live Search (formerly MSN Search) is an important crawler-based search engine used by many people. Formerly powered by Yahoo's crawler-based results, Live Search currently uses its own search technology, MSNBot, to automatically crawl the web looking for links within Web sites for indexing and ranking.
Editors' Note: The date shown on this article reflects updates provided by Claudia Bruemmer, Internet Marketing Writer and former ClickZ Managing Editor. This article was originally published by Danny Sullivan in July of 2004.
