krishnanayaks Posted November 30, 2015 Share Posted November 30, 2015 I want to know about robot.txt what benefit of robot.txt? Link to comment Share on other sites More sharing options...
Ingolme Posted November 30, 2015 Share Posted November 30, 2015 Some parts of your website are not meant to be seen by search engines. You would use robots.txt to tell search engines not to crawl those pages. Link to comment Share on other sites More sharing options...
dsonesuk Posted November 30, 2015 Share Posted November 30, 2015 Can set to total or partial restriction, set to target specific files such as in sitemap.html or sitemap.xml which would have a hierarchical link structure listing all your website pages. Note: not all bots will follow these rules. Link to comment Share on other sites More sharing options...
krishnanayaks Posted December 2, 2015 Author Share Posted December 2, 2015 Thanks for giving such valuable details about robot.txt..you mean to say that with robot.txt we can fix those page which we want to do not crawling by search engine.. Link to comment Share on other sites More sharing options...
dsonesuk Posted December 2, 2015 Share Posted December 2, 2015 Yes! but only those bots that will follow the rules (google bot for example), others will ignore. Link to comment Share on other sites More sharing options...
quangluong Posted December 21, 2015 Share Posted December 21, 2015 Sometimes, even you have specified in the robot.txt to tell Google Bot not to crawl to a specific folder but I don't know why it just crawl it. The result is that I can see that folder info in my Google Webmaster Tools. Just don't know if my folder is published or not. Link to comment Share on other sites More sharing options...
Recommended Posts
Create an account or sign in to comment
You need to be a member in order to leave a comment
Create an account
Sign up for a new account in our community. It's easy!
Register a new accountSign in
Already have an account? Sign in here.
Sign In Now