devxlogo

Stop Web Crawlers From Indexing Your Page

Stop Web Crawlers From Indexing Your Page

Using a file named ROBOTS.TXT in your root directory allows you to control the indexing behavior of most Web crawlers with respect to your site (search engines like AltaVista use Web crawlers to discover pages on the Web). However, by using a META tag, you can disable indexing of an HTML page. As with all META tags, make sure to place it inside the header of the HTML file. The various options to control Web crawler indexing include:

 

This line prevents the indexing of the page and of links on the page.

 

This line prevents the indexing of the page.

 

This line prevents the indexing of links off the page.

devxblackblue

About Our Editorial Process

At DevX, we’re dedicated to tech entrepreneurship. Our team closely follows industry shifts, new products, AI breakthroughs, technology trends, and funding announcements. Articles undergo thorough editing to ensure accuracy and clarity, reflecting DevX’s style and supporting entrepreneurs in the tech sphere.

See our full editorial policy.

About Our Journalist