devxlogo

November 19, 1998

Stop Web Crawlers From Indexing Your Page

Using a file named ROBOTS.TXT in your root directory allows you to control the indexing behavior of most Web crawlers with respect to your site (search engines like AltaVista use