You might not want sure internet pages of your internet site crawled because they won't be valuable to users if located in a internet search engine's search engine results. Observe that if your website makes use of subdomains and you wish to acquire specified webpages not crawled on a selected subdomain, you will have to create a individual robots.