Retrieve API Information Contained In XML

[Usarjjaco]

Hey Guys,So this will be sort of broad based, but I'm hoping that someone here can help out.What I'm aiming to do: I'm desiging a site for a popular MMORPG, EVE Online. They have an API system which stores in game information consistantly into XML on their servers. This information can be accessed to provide dynamic sites the relate to in game happenings.The issue: There isn't a ton of documentation on this, I'm wondering "how" I would "import" the data once per hour, and have it stored to a SQL database. I'm familiar with PHP, CSS, HTML, and I've read the tutorials for javascript, and XML. ... However I'm still kind of at a loss for how I call their pages on an hourly basis and convert the raw data to usable info that can go into a database.1) How do I pull the data, and do it on an hourly basis (Has to be hourly or longer due to the company's request)2) How do I take the data and convert it to a readable and storeable format once I've retrieved it.Thanks in advance guys, I didn't even know what "category" to really put this in because it involves several programming languages. Any help would be appreciated!P.S. Here's the API Documentation provided by the companyhttp://wiki.eve-id.net/APIv2_Page_Index

[thescientist]

You would probably want to look into running a cron job on the server side. It can be tasked to get the XML info and add it to a table in a database, by either overwriting the current info or just adding it as a new record (but this may lead to a lot of useless data if its being done every hour, unless you want to track previous data usage for creating 'history' statistics, or something.) Basically I would figure out what data you need to display, then create a table within a database to store that info (as fields for a record). Then you might want to go about creating a script (PHP/ASP) that can parse an XML document so that you can extrapolate the data you need, and then insert (add a new record) or update (overwrite) the info into that table in the database. You would than have this script run every hour by using the cron job. Now that you have the backend updating your database with new information every hour, you can have the front-end retrieve the data from the table already parsed for you so that you can display it within your webpages. That's just one option, and I'm others will offer different suggestions and whatnot, but at least this will get you thinking about the bigger picture.

[Usarjjaco]

You would probably want to look into running a cron job on the server side. It can be tasked to get the XML info and add it to a table in a database, by either overwriting the current info or just adding it as a new record (but this may lead to a lot of useless data if its being done every hour, unless you want to track previous data usage for creating 'history' statistics, or something.) Basically I would figure out what data you need to display, then create a table within a database to store that info (as fields for a record). Then you might want to go about creating a script (PHP/ASP) that can parse an XML document so that you can extrapolate the data you need, and then insert (add a new record) or update (overwrite) the info into that table in the database. You would than have this script run every hour by using the cron job. Now that you have the backend updating your database with new information every hour, you can have the front-end retrieve the data from the table already parsed for you so that you can display it within your webpages. That's just one option, and I'm others will offer different suggestions and whatnot, but at least this will get you thinking about the bigger picture.

Thanks, I'm going to start experimenting with it, this is my first delve into importing data from an external server and parsing the information. Any others who wish to post something too are welcome, I take all advice I can get . I'll post later with my progress... or lack thereof

[thescientist]

As far as getting data cross-domain, if you are using PHP, you should check to see if something like the cURL library is installed.

[Usarjjaco]

As far as getting data cross-domain, if you are using PHP, you should check to see if something like the cURL library is installed.

Ok I'll look into that. Now also a lot of items are only accessible via API keys. Which each user in the game has... how would I get it to authenticate that I'm using a certain user's API? Store that into a DB as well and have it pull from the DB based on a session?

[thescientist]

well, if you are looking for specific users info that requires access to their account, then I'm sure the docs have that covered. (I would imagine). If you don't want to store their information after they have left the site, then it should probably be enough to get their username and password and then pass that to the API. Assuming that it also returns the info in XML, you could create a script that will parse the data into an object in PHP that can be stored in $_SESSION and passed back to the client side encoded as JSON and then Javascript can get the data it needs that way. If you are looking for some sort of login/registration functionality, then that would probably require a little more work.

[Usarjjaco]

well, if you are looking for specific users info that requires access to their account, then I'm sure the docs have that covered. (I would imagine). If you don't want to store their information after they have left the site, then it should probably be enough to get their username and password and then pass that to the API. Assuming that it also returns the info in XML, you could create a script that will parse the data into an object in PHP that can be stored in $_SESSION and passed back to the client side encoded as JSON and then Javascript can get the data it needs that way. If you are looking for some sort of login/registration functionality, then that would probably require a little more work.

Ok, I'm just going to try a test run here...I've read documentation for one type of data, and the input I need to send to their server isInput Arguement:Name: idsType: StringDescription: comma seperated list of characterid's to query.My question is how would I code something like this to send out that data to the xml page so it knows what to send back? Thanks for all the help already!P.S. What's the quickest way to check if I have cURL ? Sorry for all the questions .. trying to learn

[thescientist]

Ok, I'm just going to try a test run here...I've read documentation for one type of data, and the input I need to send to their server isInput Arguement:Name: idsType: StringDescription: comma seperated list of characterid's to query.My question is how would I code something like this to send out that data to the xml page so it knows what to send back? Thanks for all the help already!P.S. What's the quickest way to check if I have cURL ? Sorry for all the questions .. trying to learn

I'm guessing this is just through a URL? Do they have examples? It's probably just going to look something like this:http://www.theirdomain.com/api?ids=23543423,2342342,343242 or whatever.To find out if you have cURL installed just upload a php file to your webserver that contains this:

<?phpecho phpinfo();?>

and see if you can find a section for cURL when you view the page. Another alternative can be found here:http://us3.php.net/manual/en/book.http.php

[Usarjjaco]

I'm guessing this is just through a URL? Do they have examples? It's probably just going to look something like this:http://www.theirdomain.com/api?ids=23543423,2342342,343242 or whatever.To find out if you have cURL installed just upload a php file to your webserver that contains this:

<?phpecho phpinfo();?>

and see if you can find a section for cURL when you view the page. Another alternative can be found here:http://us3.php.net/manual/en/book.http.php

Yeah you are right, I tested it out on a URL and it worked fine. ... How would I go about it if there were say... 3 inputs required? Such asidsuserIDapiKey... do I just do something like this?http://api.eve-online.com/corp/AccountBala...apiKey=afwefrg1or?Also I found out I do have cURL.

[thescientist]

Yeah you are right, I tested it out on a URL and it worked fine. ... How would I go about it if there were say... 3 inputs required? Such asidsuserIDapiKey... do I just do something like this?http://api.eve-online.com/corp/AccountBala...apiKey=afwefrg1or?Also I found out I do have cURL.

sort of. you need & to separate params in a query string. Also don't forget to URI encode the values of name/value pairs (what something like this characterID=602236694 is called) as well. It should be a native function within PHP. (there's definitely one for Javascript)AccountBalance.xml.aspx?userID=1904604&characterID=602236694&apiKey=afwefrg1

[boen_robot]

It's very important to carefully read every word of their documentation... in particular, notice that

•On all requests, application/web site sends the authentication parameters as POST arguments

It sounds like you'll really need cURL. With it, you can send POST arguments in HTTP requests. Without it, you can only use GET ones (which are part of the URL, as described above).

[Usarjjaco]

Ok, so something like this for PHP

<?phpecho '<a href="http://example.com/department_list_script/',	rawurlencode('sales and marketing/Miami'), '">';?>

... Seems to work.... the pieces are slowly coming together. I'll head off and storm the castle a bit and let you guys know how it goes. Thanks for all the help thus far.Also, is there a link on a quick tutorial on how to us cURL for the requests? I know cURL is installed. (Heading to the w3 tutorial section right now to see if I can scrounge something up)

[thescientist]

that will just echo a URL...you'll need to actually use it conjunction with cURL so can make the request and get a response back.I'm not sure if w3schools has it, but I think you should be able to google info on it.

[Usarjjaco]

Ok, I looked up most of it, the only question I'm having is trying to find the proper info on inserting the data into an SQL database... I feel like such a peon right now :)Here is the XML data I get from a sample page I took. This is what it shows up as, so what would I do to "parse" this and then "place" the info into the SQL database (for example's sake lets call the database "test" and the table "account") using php if possible... thanks a ton guys (I'm looking all this up but wouldn't mind seeing a written example using the stuff I'm working with)Here's the XML output

  <?xml version="1.0" encoding="UTF-8" ?> - <eveapi version="2">  <currentTime>2010-11-03 18:32:17</currentTime> - <result>- <rowset name="accounts" key="accountID" columns="accountID,accountKey,balance">  <row accountID="32729307" accountKey="1000" balance="151345.83" />   <row accountID="32729311" accountKey="1001" balance="200071893.69" />   <row accountID="32729312" accountKey="1002" balance="47939699.29" />   <row accountID="32729313" accountKey="1003" balance="43523797.00" />   <row accountID="32729314" accountKey="1004" balance="199000000.00" />   <row accountID="32729315" accountKey="1005" balance="10000000.00" />   <row accountID="32729316" accountKey="1006" balance="16550000.00" />   </rowset>  </result>  <cachedUntil>2010-11-03 18:47:22</cachedUntil>   </eveapi>

[thescientist]

ok, no offense, but now you're getting into stuff that you should at least be trying to find on your own first. Heck, the w3schools PHP page has links on the left for both parsing XML and how to interact with SQL and a database.http://www.w3schools.com/php/default.aspSee what you can get from there. I would try and save that text as XML somewhere and pull it into PHP (for testing) and then try and assign all the information you need to variables in PHP. From there you should have no problems inserting that info into a database. Then, once you get your parser working, you would want to implement cURL so you can be retrieving that XML data dynamically. Note: you may want to parse XML in two ways. One parser for a users account info, and one to parse game data. A conditional should easily implemented based on what type of call you are making. i.e. if you are getting account info, use the account parser on the returned XML, else if its game data, use the game data parser. I had to make a parse for an XML based API and at first I made a generic parser that would handle any kind of document save a couple of exceptions, but in the event that the API changed, it might be harder to maintain the code. If you make a parser specifically for one set of data, you only have to change the parser for whichever one changes, without having to wonder if it will break when parsing other kinds of data. Just something to consider. boen_robots has a good handle on XML, so he may be able to shine more light on the issue.Other note: you may want to make a separate scripts for getting and parsing the XML data actually, because the one that's getting the game data is going to have to be run every hour. just a thought.

[Usarjjaco]

Sorry about that, just in the haste of the moment :)Anyways, after some looking and messing around here's what I came up with

$xml_feed_url = 'http://api.eve-online.com/eve/CharacterInfo.xml.aspx?characterID=602236694';$ch = curl_init();curl_setopt($ch, CURLOPT_URL, $xml_feed_url);curl_setopt($ch, CURLOPT_HEADER, false);curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);$xml = curl_exec($ch);curl_close($ch);$xmlr = new SimpleXMLElement($xml);$charid = $xmlr->result->characterID;echo $charid;echo "<br>";echo $xmlr->result->corporation;?>

That is displaying the information I wanted, and I was able to make the certain elements into variables which I will be able to store in the DB. The only question I have is this line

$xml_feed_url = 'http://api.eve-online.com/eve/CharacterInfo.xml.aspx?characterID=602236694';

At the end the character ID number, I would like to make that a variable , but I can't for the life of me remember how to put a variable into that string properly. I would like something like

$xml_feed_url = 'http://api.eve-online.com/eve/CharacterInfo.xml.aspx?characterID=' $number;

Where $number is the ID number that has been designated as that variable prior to this line executing. Thanks in advance!

[boen_robot]

You concatenate (join together) strings with the "." operator, like:

$xml_feed_url = 'http://api.eve-online.com/eve/CharacterInfo.xml.aspx?characterID=' . $number;

PHP is a loosely typed language, so it will automatically convert the number into a string before concatenating it.