mysteriousmonkey29

server-side storage general questions?

8 posts in this topic

Hello, I am trying to create a web comic aggregation website in PHP/HTML/JavaScript. Currently, my general design plan for the website is an RSS reader that periodically checks various web comics for new comics, and then adds these comics to an ever-growing database, which is then accessed via the client side of the website and displayed in a user-friendly and interactive format. I also may eventually attempt to use web scraping to backfill the database with all the previous comics from before the RSS updater was launched.

 

Currently, I have created an RSS reader that gets the image source of all the comics from a given RSS feed. So mostly I would want to be storing text strings (the URLs) along with some kind of organizational keys (probably based on publication date). Currently, it is not particularly important to me that this information be kept secure at all, although I may eventually add an account creation option in which case I would need to securely store usernames and passwords. I would also like to back up all the comic links I gather by actually downloading the images somewhere (in case the links ever get changed or taken down). Finally, this database can be the same for every user of the website, although it needs to be able to get very long (probably between thousands and hundreds of thousands of individual comics), and of course I would like it to be reasonably quickly accessible so my site isn't incredibly slow.

 

After some research online, it seems like I could use simple file storage, or some kind of database storage API, such as MySQL or Microsoft SQL Server. I was wondering what the advantages and disadvantages of these various options might be (and if I missed any major options), along with how server-side web storage generally works? For example, when I actually launch my website, I know that I would have to rent a server from somewhere or set one up myself. Would I also have to rent a database storage means along with this? How would that generally work?

Share this post


Link to post
Share on other sites

Databases are included with most hosting plans, they're typically on the same server but some hosts have dedicated web and database servers.

 

If your site is going to be publicly accessible then you need to make sure you have permission to publish all of that content.

Share this post


Link to post
Share on other sites

Okay cool. If I decided on a particular type of database API to use (say MySQL), would that limit my options for purchasing hosting plans? Also, would most hosting plans include databases capable of handling tens or hundreds of thousands of lines of text and maybe also images?

 

As for the copyright issue, thanks. If I can get the site to work locally, then I plan on going to the next step and contacting the content owners to pay for content. But for now I'm just seeing if I can get it to work

Share this post


Link to post
Share on other sites

If I decided on a particular type of database API to use (say MySQL), would that limit my options for purchasing hosting plans?

Of course. Not every host supports every kind of database. A Linux web server probably won't support Microsoft SQL Server at all, for example, because the host would need dedicated database servers which is more to manage.

 

Also, would most hosting plans include databases capable of handling tens or hundreds of thousands of lines of text and maybe also images?

Databases don't have hard limits for how much data they can hold, the major things that affect performance are the hardware that the server is running on, and how efficiently you've designed the database. It's pretty easy to have one design that causes a query to take a minute, but a few small changes can bring it down to a fraction of a second. Design is a big part of an efficient database.

 

I wouldn't store binary data like images in a database though, filesystems are better for storing the actual data and you would store only the filename in the database.

Share this post


Link to post
Share on other sites

Actually I think I just figure out the answer to my question. My guess is that it's just included in the hosting plan you get.

 

I think I generally get how database and file system storage works in the context of actually producing a website now--thanks for the help.
Any recommendations as to different types of file system storage and/or database storage?

Share this post


Link to post
Share on other sites
More specifically, I have read a lot about MySQL, It seems basically perfect except that I have read that it has major performance issues if you attempt to scale it up a lot. Do you guys agree with this assessment, and if so, how large does a website need to get in order to run into these performance issues?


I was also considering some other options like Microsoft and Oracle SQL.

Share this post


Link to post
Share on other sites

MySQL can scale just fine. If your site outgrows a single server then you can create a MySQL cluster and replicate the database across several servers. Facebook uses MySQL for a lot of its general data storage, for example, so I think that you're fine worrying about MySQL for now. Focus on how to design a database instead of trying to find an entry-level database. MySQL is over 20 years old, it's fair to say that they know a thing or two about scaling up by now.

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!


Register a new account

Sign in

Already have an account? Sign in here.


Sign In Now