Spiders, Crawlers and the Tangled Web

Considering the sheer volume of data, the major search engines are doing a commendable job of categorizing the more than two billion Web pages out there. These search engines use complex automated spidering agents to constantly update their databases. But even the large search engines cannot index it all; it is estimated that only 20% of the Web's content under the top-level domains (.com's) is indexed by Google.

But is all this information too much? What about the user who types in "drivers" hoping to find golf clubs and instead finds 9 million results, with no golf-related links in sight? An automated spider does not have the intelligence to know a user's search intentions. The frustration with the cumbersome, generic search results is mounting each day, as the chasm of the Web grows deeper.

Let's briefly explore the world of search engine categorization in an attempt to untangle the burgeoning Web.

Ever-Changing Formulas

Search engines are commonly built with automated intelligence, also known as spiders and crawlers, which traverse the Web or site servers for content and then complex algorithms to weight and catalog Web pages.

Both Internet search engines and site search engine employ algorithmic searches that use mathematical formulas to judge the relevance of Web pages for any given set of search terms.

Generally, the cataloging and ranking process is akin to these steps: when a user types in a keyword or phrase, the algorithm first hunts for the search term in the HTML title tag of a page. When it finds a page that has the search topic in the title, the crawler assumes that page to be more relevant than others.

Initial page content is also scanned for the search keywords, looking for the topic in a heading or the first few paragraphs of text. The engine assumes that any relevant page will have the search topic in the beginning of its main body text.

Crawlers assess how often the keywords appear in a page. The more frequent the keyword usage, the more relevant the page the crawler deems the page. Some engines also use metatags, but each engine has a different protocol for use and weighting.

Google adds another dimension in the ranking process with its Page Rank™ system of using the link structure to an individual Web page as an indicator of the page's value. Other engines consider subject communities to determine relevancy

Results are prioritized by what pages the algorithm determines contain the best title, content and redundant use of keywords. The algorithms' intent is to untangle the web of data, not interpret your thought patterns. Type in "drivers" and get the best matches according to mathematics.

Search engines are constantly changing their algorithms to counter search engine optimizers. Optimization is an ever-changing art, and because of the multiple factors involved, ranking or even inclusion can never be assured for long.

A Better Way

User frustration illustrates that automated intelligence ranks results well - but quite often, not well enough. This especially applies to site search where a Forester study shows, 80% of all online buyers terminate potential purchases when they cannot find what they want.

As user frustration shows, no automated intelligence can rank results as well as human intelligence. This especially applies to site search where a Forester study shows, 80% of all online buyers terminate potential purchases when they cannot find what they want.

Crawlers, spiders and algorithms are necessary to build a database of relevant search results. But the key to relevancy is the analysis of user behavior and subsequent ranking modifications. Untangling the Web of data is sure to make search your site's most user- and profit-friendly feature.