Login | Register   
LinkedIn
Google+
Twitter
RSS Feed
Download our iPhone app
TODAY'S HEADLINES  |   ARTICLE ARCHIVE  |   FORUMS  |   TIP BANK
Browse DevX
Sign up for e-mail newsletters from DevX


Tip of the Day
Language: Web
Expertise: Intermediate
Nov 18, 1998

Stop Web Crawlers From Indexing Your Page

Using a file named ROBOTS.TXT in your root directory allows you to control the indexing behavior of most Web crawlers with respect to your site (search engines like AltaVista use Web crawlers to discover pages on the Web). However, by using a META tag, you can disable indexing of an HTML page. As with all META tags, make sure to place it inside the header of the HTML file. The various options to control Web crawler indexing include:
 
<META NAME="robots" CONTENT="none"
This line prevents the indexing of the page and of links on the page.
 
<META NAME="robots" CONTENT="noindex"
This line prevents the indexing of the page.
 
<META NAME="robots" CONTENT="nofollow"
This line prevents the indexing of links off the page.
Erik Goetze
 
Comment and Contribute

 

 

 

 

 


(Maximum characters: 1200). You have 1200 characters left.

 

 

Sitemap
Thanks for your registration, follow us on our social networks to keep up-to-date