Meet the “robots.txt” File

When a search engine spider first arrives at a website, the spider looks for the “robots.txt” file. This is a text file that resides on the web server at the hosted domain. The “robots.txt” file has specific instructions for the search engine spider such as which parts of the site should or should not be indexed into their database.

Following the directions of the “robots.txt” file is standard procedure for all of the major search engines. By the search engines working together, it is now easier for a web master or SEO to control what they want people to find on their site.

Getting the Search Engines to Your Site

The most likely way that the search engines find your site is by links from other sites pointing to your website. Another way is to submit your site to the search engines is through a submit form hosted by the search engine. Submitting your site to the search engines is pretty much old school as the top search engines do not offer a submit form at all. If you do this, you are wasting your time and just asking for a flood of spam into your email box.

I never submit my sites to the search engines at all, and my sites always get indexed pretty quickly (normally within 7 days).

The $300 Yahoo Scam

Websites like Yahoo do offer to add your site to their web directory for about $300. There is no real SEO value to doing this, although you may get a little traffic from it. Yahoo will give your site top priority for indexing if you pay them the $300, but this is not necessary at all. At his stage of the SEO game, it is a bit of a scam.

In some of our later posts, we will discuss the best techniques for getting your site indexed more quickly and a whole lot cheaper than $300.