ben3001 Posted November 22, 2006 Share Posted November 22, 2006 HiIve looked everywhere for info re: robots.txt.could you explain please in the simplist form what it is, what it includes and how to construct one.Much ApprecitedB Link to comment Share on other sites More sharing options...
jesh Posted November 22, 2006 Share Posted November 22, 2006 Maybe this will help: http://www.robotstxt.org/wc/exclusion-admin.htmlAlso, you might think about using the meta tags as well:http://www.robotstxt.org/wc/meta-user.htmlExample: <meta name="robots" content="none" /> <!- tells bots to not index the site nor follow links --> Link to comment Share on other sites More sharing options...
ben3001 Posted November 22, 2006 Author Share Posted November 22, 2006 cheers for that site, its explained in a nut shell Link to comment Share on other sites More sharing options...
Recommended Posts
Create an account or sign in to comment
You need to be a member in order to leave a comment
Create an account
Sign up for a new account in our community. It's easy!
Register a new accountSign in
Already have an account? Sign in here.
Sign In Now