How search engines work

Search engines are designed to quickly and efficiently search and retrieve stored information. Once the search engine has retrieved the information, they then rank the results in order of importance, using their own ranking criteria.

Most search engines consist of three major components. They are:

• The Spider
• The Index
• The Ranking System

The Spider
Search engines send out electronic robots (bots for short) sometimes called spiders, to relentlessly search the Internet, gathering information. The spider is the lifeblood of the search engine.

When a search engine’s spider finds a new website, it crawls all over the site and records the information on that website’s pages and returns this information to the search engine. This process is referred to as spidering or crawling a website.

You can also ask the search engine to spider (list) your site by submitting your domain name to them. If you have a website and you have access to your logs (statistics of who visits your website) you can actually track the constant movement of spiders through your site. They will show up in your logs with names like googlebot (a Google spider) or Scooter (Altavista).

The Index
Everything the spider finds, goes into the second part of the search engine, the index or catalogue. This is simply a very large database, accessible through the internet. Everything the spider finds is stored in the search engine’s index.

The Ranking System or Algorithm
The final and trickiest part of the search engine, is the system they use to rank their listings. This is done by a complex series of formulae know as an "algorithm". This is the formula the search engine uses to rank webpages in order of relevance (as they judge it) to a specific search.

Each search engine has its own algorithm for indexing and ranking webpages, which explains why you can do the exact same search on two different search engines and get completely different results.

Exactly how the algorithm work, is always a closely guarded secret of the search engine concerned however, most of them tend to follow a similar pattern, however, some are more complex and refined than others.

TIP: To find out more about how the search engines work and how you can use this knowledge to get a better ranking for your website, order a copy of the manual, Search Engines Secrets.