A situation may arise in which you would need to hide certain sections of your page from crawlers or determine what actions to take based on the referrer of the Request. You can use the HttpCapabilitiesBase.Crawler
property to determine this.
In the sample below, the Request.Browser object represents HttpCapabilitiesBase object.
if (Request.Browser.Crawler){ HideCertainSections(); }