Jump to content

Robots.txt


GermanPrince
 Share

Recommended Posts

ROBOTS.txt files are used to stop search engines from refrencing certain files or directories. Say you don't want search engines to index "/scripts" or "/data" or "/images". You would write the following code:

User-Agent: *Disallow: /scripts/Disallow: /data/Disallow: /images/

What if you don't want anyone BUT Google to index your images. You would use the following code:

User-Agent: *Disallow: /images/User-Agent: Googlebot-ImageAllow: /images/

"#" is used for comments. Take a peak at the following code for a better idea.

User-Agent: Googlebot-Image #Disallows Google Images from indexing certain imagesDisallow: /images/

Each search engine has its own bot name. It would be great if you guys listed some for people to use. That is how you use robots.txt files.This is subject to change.

Edited by GermanPrince
Link to comment
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
 Share

×
×
  • Create New...