The System.Web.HttpBroswerCapabilities
?has an important property called “Crawler” that enables us to handle requests from crawlers. Please check out the example below:
Var isCrawler = ((System.Web.Configuration.HttpCapabilitiesBase)Request.Browser).Crawler;