Jump to content
  • Announcements

    • boen_robot

      Guidelines and Netiquette   03/28/2017

      Posting Problems:   Having problems posting your topic? Read through this: To join, you agree to our terms and conditions and fill out and submit a registration form. An activation email will be sent to your email adress, so you'll need to verify your account. After that the account has to be validated by one of the moderators. This will mean that it can take up to a day to be activated. A couple of things to remember to ensure approval: Don't use an email address in one of those $2 four character .com domains eg. xyds.com. These will be deleted and the domain added to the banned list. Don't use an email address that is within a domain with a bad reputation for spam. A Google search is run on every email address and email domain. Don't sign up with an email address that doesn't exist, doesn't work or requires the sender to answer a quiz before their email can get to you. Put your country and or state and city in the signup form. Blank forms will go to the botton of the "to do" list. And make sure that your email address and your country match, saying you're from Alabama and using a .ru email address is not going to get you activated. After a membership is activated the first few posts will be monitored. Posting spam or unapproved topics described in the agreement results in an immediate ban. The email provider and the IP addresses associated with the account will be banned and all posts will be deleted. These strict measures have been deemed necessary to hinder spam. Sorry for any inconvenience this causes, but it's not liable to change. If, after reading this, you still can't post and don't understand why, contact one of the Moderators listed here.   Topic Guidelines   Including the following information can expedite an accurate response from board members: Must be a Specific Problem or Question related to web design and development Include Code in Question (wrap with   for small blocks of code and for longer blocks   ) Include Code Author Include Extra Notes/Modifications/Attempts Include web link to page/file when possible Content Guidelines   You may not post, upload, link to, or email any Content that contains, promotes, gives instruction about, or provides prohibited Content. Prohibited Content includes any Content that breaks any local, state, county, national or international law. Prohibited Content also includes: No direct or indirect advertising or websites, forums, products, services No hijacking of posts (do not post your question in someone elses) Content that infringes upon any rights [ex. MP3s and ROMs] (including, but not limited to, copyrights and trademarks) Abusive, threatening, defamatory, racist, or obscene Viruses or any other harmful computer software False Information or libel Spam, chain letters, or Pyramid schemes Gambling or Illicit drugs Terrorism Hacking or cheating for internet/online games Warez, Roms, CD-Keys, Cracks, Passwords, or Serial Numbers Pornography, nudity, or sexual material of any kind Excessive profanity Invasive of privacy or impersonation of any person/entity Hacking materials or information Posting Tips   There are more BBcodes than there are buttons for on the reply menu. To get the full list, click "BB Code Help" underneath the clickable smilie face menu. Use   for small snippets of code Use   for lengthy snippets of code Use   if your snippet is HTML (optional) Use   if your snippet is SQL (optional) Rules of Conduct   Be nice. There's no need for calling someone stupid if they ask an 'easy' question. Keep your avatars and signatures absolutely child friendly. We have a younger audience on this forum. Keep your language appropriate for the same reason above. Do not PM moderators for help on the forum. Post on the topic, or create a new one.   Spam:   Recently, as you have all without doubt noticed, we have had lots of spam and advertisement on the forum. Therefore, we'd like to alert you as to what to do when you have found any of the aforementioned annoying messages: it. Immediately. Give a clear reason, please, if the advertising is not evident. DO NOT POST! Report, let the post stay as is, and we will get to it, meanwhile if you continue to post as normal in the other threads, it won't be on the top so long. Refrain from PMing the member. This won't help at all, as they are most likely spambots anyway. Thank you.       Images in signatures:   After thinking of users on dial-up, we have decided to enforce the following rules regarding signatures. Please pay heed to them. Respecting these rules is respecting the members on this forum with dial-up. Signature rules: No animated images AT ALL. No matter the amount of animation. Maximum image widthxheight: 300x150 Maximum image (file) size: 15kb Use calm colors. Do not use highly contrasting images in your signature, as this can get really annoying when seeing several posts from one member in the same thread. The same prohibited content goes for images as for posts. Lastly, use common sense. No lengthy signatures please. Save us some scrolling. Thank you.       Links in signatures:   Please understand that w3schools.com only exists because of voluntary work and is barely supported by the advertising littered throughout the tutorials and the forum. So, please, stop advertising other sites. DO not post links that drive traffic away from the w3schools domain - especially to a site that offers similar if not identical information. Please help support the site by keeping individuals on it. Thank you. Here are some guidelines as to what you can put in your signature: w3schools links --> allowed w3.org links --> allowed browser links --> allowed html editor links --> allowed personal sites --> allowed tutorial sites competing with w3schools --> NOT allowed sites completely irrelevant to webprogramming and this forum --> NOT allowed   Thanks for understanding, and for taking the time to read this. ~W3Schools Modstaff~
mysteriousmonkey29

server-side storage general questions?

Recommended Posts

Hello, I am trying to create a web comic aggregation website in PHP/HTML/JavaScript. Currently, my general design plan for the website is an RSS reader that periodically checks various web comics for new comics, and then adds these comics to an ever-growing database, which is then accessed via the client side of the website and displayed in a user-friendly and interactive format. I also may eventually attempt to use web scraping to backfill the database with all the previous comics from before the RSS updater was launched.

 

Currently, I have created an RSS reader that gets the image source of all the comics from a given RSS feed. So mostly I would want to be storing text strings (the URLs) along with some kind of organizational keys (probably based on publication date). Currently, it is not particularly important to me that this information be kept secure at all, although I may eventually add an account creation option in which case I would need to securely store usernames and passwords. I would also like to back up all the comic links I gather by actually downloading the images somewhere (in case the links ever get changed or taken down). Finally, this database can be the same for every user of the website, although it needs to be able to get very long (probably between thousands and hundreds of thousands of individual comics), and of course I would like it to be reasonably quickly accessible so my site isn't incredibly slow.

 

After some research online, it seems like I could use simple file storage, or some kind of database storage API, such as MySQL or Microsoft SQL Server. I was wondering what the advantages and disadvantages of these various options might be (and if I missed any major options), along with how server-side web storage generally works? For example, when I actually launch my website, I know that I would have to rent a server from somewhere or set one up myself. Would I also have to rent a database storage means along with this? How would that generally work?

Share this post


Link to post
Share on other sites
justsomeguy    954

Databases are included with most hosting plans, they're typically on the same server but some hosts have dedicated web and database servers.

 

If your site is going to be publicly accessible then you need to make sure you have permission to publish all of that content.

Share this post


Link to post
Share on other sites

Okay cool. If I decided on a particular type of database API to use (say MySQL), would that limit my options for purchasing hosting plans? Also, would most hosting plans include databases capable of handling tens or hundreds of thousands of lines of text and maybe also images?

 

As for the copyright issue, thanks. If I can get the site to work locally, then I plan on going to the next step and contacting the content owners to pay for content. But for now I'm just seeing if I can get it to work

Share this post


Link to post
Share on other sites
justsomeguy    954

If I decided on a particular type of database API to use (say MySQL), would that limit my options for purchasing hosting plans?

Of course. Not every host supports every kind of database. A Linux web server probably won't support Microsoft SQL Server at all, for example, because the host would need dedicated database servers which is more to manage.

 

Also, would most hosting plans include databases capable of handling tens or hundreds of thousands of lines of text and maybe also images?

Databases don't have hard limits for how much data they can hold, the major things that affect performance are the hardware that the server is running on, and how efficiently you've designed the database. It's pretty easy to have one design that causes a query to take a minute, but a few small changes can bring it down to a fraction of a second. Design is a big part of an efficient database.

 

I wouldn't store binary data like images in a database though, filesystems are better for storing the actual data and you would store only the filename in the database.

Share this post


Link to post
Share on other sites

Actually I think I just figure out the answer to my question. My guess is that it's just included in the hosting plan you get.

 

I think I generally get how database and file system storage works in the context of actually producing a website now--thanks for the help.
Any recommendations as to different types of file system storage and/or database storage?

Share this post


Link to post
Share on other sites
More specifically, I have read a lot about MySQL, It seems basically perfect except that I have read that it has major performance issues if you attempt to scale it up a lot. Do you guys agree with this assessment, and if so, how large does a website need to get in order to run into these performance issues?


I was also considering some other options like Microsoft and Oracle SQL.

Share this post


Link to post
Share on other sites
justsomeguy    954

MySQL can scale just fine. If your site outgrows a single server then you can create a MySQL cluster and replicate the database across several servers. Facebook uses MySQL for a lot of its general data storage, for example, so I think that you're fine worrying about MySQL for now. Focus on how to design a database instead of trying to find an entry-level database. MySQL is over 20 years old, it's fair to say that they know a thing or two about scaling up by now.

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

×