astralaaron Posted May 16, 2007 Share Posted May 16, 2007 okay say you have a CMS blog page where you are typeing in the input fields and it stores the information in a database.. and you have the page where people read it that loads the blogs from the database.. do webcrawlers crawl the information that gets pulled from the database?not sure if that made sence, i just woke up maybe I should think of a better way to ask Link to comment Share on other sites More sharing options...
justsomeguy Posted May 16, 2007 Share Posted May 16, 2007 Do web crawlers index dynamic pages..If there are links to the pages, I believe that the engines will pick it up. They will definately pick it up if each page has a unique URL (not just the same .php page with different querystring variables). That's why you see URLs like this one, which is search-engine-friendly: http://www.upi.com/International_Intelligence/Analysis/2007/05/16/analysis_taking_stock_of_the_gaza_fight/1611/ But, that being said, Google News still ranks links like these very highly: http://www.bloomberg.com/apps/news?pid=20601103&sid=azcOyFUIB.Qk&refer=us The point is that if each page can be navigated to through the site, the crawlers should be able to find those pages. It won't connect to the database and look in the database (that would be very bad), so the content does need to appear on a page that your site links to somewhere, but as long as there is a link on your site to the page it should get indexed. One thing to make sure the engines will index all your pages is to put the robot and cache control meta tags to instruct robots to follow all links and not cache any pages. Link to comment Share on other sites More sharing options...
astralaaron Posted May 16, 2007 Author Share Posted May 16, 2007 thanks for the information, I did not think that the crawler actually goes into the database.. I know when a user goes to a page with php, the information is preprocessed and then the page is made.. I was just wondering if when an engine crawls the page if the same thing happens but I guess it does!thanks again. Link to comment Share on other sites More sharing options...
justsomeguy Posted May 16, 2007 Share Posted May 16, 2007 Yeah, the crawler makes a request to the web server the same way a web browser does, and the web server will process the request and send the response the same way, it doesn't treat a search engine crawler any differently. Link to comment Share on other sites More sharing options...
astralaaron Posted May 16, 2007 Author Share Posted May 16, 2007 Yeah, the crawler makes a request to the web server the same way a web browser does, and the web server will process the request and send the response the same way, it doesn't treat a search engine crawler any differently.thanks Link to comment Share on other sites More sharing options...
Recommended Posts
Archived
This topic is now archived and is closed to further replies.