Wednesday, January 24, 2007

Bots, Crawlers and Spiders

Bots, Crawlers and Spiders, Oh my!
How search engines work...
By: David K. Every from
Most people creating their own websites want their site to be at the top of the search engines response list when someone searches for a particular topic. While that isn't likely, there are things you can do to move up the priority list, and it is good to understand how these things work.

There is something referred to as a bot (robot), crawler or spider. Their purpose is to automatically surf the web. They go to every site, and try to follow every link they can find; then they crawl all over anything they can find there as well; hence the bug references. When you register your site with a search engine, that is all you are doing - telling their spider to start crawling your site; they will find you eventually, whether you register or not.

Search engines employ these (none-too-smart) automatons to look for anything new, and to create an "index" of what they find on each site. They keep a list of topics and key words, and they count them up. If each page on your site has the word "computer" in it, then they can "guesstimate" that your site has something to do with computers. Then when someone searches for the word "computer", they know that your site has something to do with that topic, and you should be in the list of 15,000+ sites that also refer to computers.

The problem is that there are so many sites and pages that have to do with every topic, that they also need to figure out who should be first on the list. They need to figure out popularity and give each site (or page) some relative weight, with the most massive sites showing up higher on the list.

If you want a lot of weight quickly, the search engines will let you rent it (a form of advertising); but most of us don't have the budgets to create that artificial weight, and must do it other ways.

Search engines can do little to figure out true "popularity" and how often people visit; they can't really snoop other people's sites and see who is visiting or how often, so they resort to less direct methods.

On the extreme high end, there is some ability to poll users and figure out where they are going; but you're not likely to show up in those, and we're not talking about creating a site for a company that can measurably effect the nations GDP; just a normal site.

One of the ways that search engines can guess at popularity is to just count links; they look at how many other sites are pointing to a page on your site, then they can rate how "popular" your site is. The more people that point to you, and the bigger they are, then the more valuable your information must be; and the more weight you get. So if you want to show up better on search engines, then you need to make web-friends and link to each other. Advertising banners on other sites (that have weight), don't hurt, since that adds some weight (links and readers); but the banner links stop when you stop advertising.

Another way is just based on how big your site is; if there are many articles on the site (a lot of "content"), that all have articles about the same subject, then you're probably getting visitors on that topic, and others would probably have interest as well.

In fact, content is often the secret to being a successful site. If you create or put up good information (content) on subjects that people have interest in, then you'll rank higher. You can go to other sites that have interest in that topic, and ask them to link to your articles. Others will link to them on their own. Either way, you'll be gaining readers, and "weight" as far as the search engines are concerned; which means others will find your site easier, and they'll be more likely to link to it; which adds even more to your weight. Like a snowball rolling down a hill, the more content you have, the more momentum you pick up, and then the more people notice you. Content and a little self-promotion go a long way in the popularity of a web site.

0 responses:

Post a Comment

Thanking you for your comment(s). Hope you will visit this blog again!

Subscribe to geeklog feed Bookmark and Share

Design by Free blogger template