Order the manual

How search engines work

Search engines vs. directories

Getting to the top

Importance of keywords

Keyword analyzer

The top search engines

Is your website spider friendly?

Search Engine Secrets

Home page

What is a search engine spider?

The search engines send out electronic robots called spiders or crawlers, to relentlessly search the internet looking for websites and gathering information. Spiders are really the lifeblood of the search engines and we talk a lot about spiders and how they work in the Search Engine Secrets manual.

When a search engine's spider finds a new website, it crawls all over the site recording words and other information on that website's pages and returns this information to the search engine's index or database.

This process is referred to as spidering or crawling a website. You can also ask the search engine to spider (list) your site by submitting your domain name to them. If you have a website and you have access to your logs (statistics of who visits your website) you can actually track the constant movement of spiders through your site. They will show up in your logs with names like googlebot (a Google spider) or Slurp (a Yahoo! spider).

WARNING! BEWARE!

DO NOT SPAM THE SEARCH ENGINES!

One of the main reasons people don't get their website listed with some search engines, is they try to cheat the system by spamming and end up getting barred. Unfortunately, some people get found guilty of spamming even though they have unintentionally committed their breach in all innocence.

The term 'spam' is normally associated with junk email (unsolicited, unwanted emails) but it is used by the search engines in much the same way. i.e. You send them a whole heap of information they don't want!

If a search engine suspects you of spamming it i.e. you are using questionable techniques designed to fool it into thinking your web page is something it is not, purely to get you ranked further up the list — you may be either refused a listing or booted off the list if you are already on it.

Therefore, search engine spamming should be avoided at all costs. And with the search engines it's very much Marshall Law. I.e. There is no system of trial by jury, no right to explain yourself - you are just booted off! There is an appeal process if this happens but don't hold your breath waiting for the Governor to come riding through on his white charger with a pardon!

Suffice to say the easiest way to avoid these sorts of problems, is not to commit the offence in the first place. Don't spam! Learn what the engines don't like and don't do it!

In the Search Engine Secrets manual you will find a list of the things NOT to do when preparing your pages for the search engines, lest you be accused (rightly or wrongly) of spamming.

Order your copy of Search Engine Secrets TODAY!