Jump to content

basic web-crawler project - where to start?


Recommended Posts

Hi, been a while.

Started a new project today. A WEBSITE Crawler made with pure PHP (if possible of course).

I know there are other websites out there like this - but never the less i always wanted to create my very own.

The idea about the project is to create a website that will gather information from various different website links i have pasted into the code/database etc, and it would then take the prices from those websites and place them in a sort of "history" of prices in the database. You probably know what im talking about - but just making this for the sake of seeing how its actually done.

However, i am a little bit clueless as to where to start? I mean... would i create some sort of CRON job to run a file that would then execute the code for me each hour or something? And if so, how would a basic code like this look like? Any ideas what to look into would be really appreciated! :D

Many thanks in advance! :D

Link to comment
Share on other sites

  • 1 year later...

So, how's your project doing? It sounds like a very great thing that could be helpful for most of us. There are similar services on the internet. But they are not that good. Some of them are pure trash! I know it's been a year since you posted this, and no one helped you here, but I hope that you made it a real thing. I was browsing through google right now, and I could only find one proper web-crawler. It's proxycrawl.com. I've also used this one back when I created my own website. If you are still working on your project, you could use their website and their services as a template.

Edited by KripyPicli
Link to comment
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

  • Create New...