Jump to content

Why use robot.txt?


krishnanayaks

Recommended Posts

  • 3 weeks later...

Sometimes, even you have specified in the robot.txt to tell Google Bot not to crawl to a specific folder but I don't know why it just crawl it.

The result is that I can see that folder info in my Google Webmaster Tools. Just don't know if my folder is published or not.

Link to comment
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
×
×
  • Create New...