Getting Data From Web

I need to get data from multiple web pages into one array so I can plot it on a graph.

More in depth: I am trying to create a distribution graph of the review scores on metacritic. What I figure I need to do is to: Load up the all time high scores page, add the review scores to table, next page, repete.

Unfortunately I have absolutely no web programing knowledge whatsoever. Literally never wrote a single line of html or php or anything like it. As such I have not got even the slightest clue how to solve this problem. I understand MatLab might not be the most ideal language for doing this, and I have been told by someone who knows only slightly more than I do that Python would probably do a better job at actually collecting the data.

Any help would be extremely useful, even if it is just a link to a forum post or similar.

Thanks

Answer 1

[Image Analyst]

People use MATLAB's urlread() function to retrieve web pages all the time. Check it out. I have no comments on whether Python would be easier or not.

[Adam Shaw]

Okay, I have managed to kind of work with this. I turned the page script into a string with urlread(), then did a search for metascore_w (the class containing the metascore for each item). Problem is on each page it also has the top rated for that week, so I can't just do a nice simple search and grab.

What I need to do is chop the string between the point where it says "<div class="product_condensed">" and the closing statement for the function, "</div>".

Unfortunately "</div>" is the closing function for pretty much everything else on the page, so chopping the page down will require some serious loops and counters and string searching.

That is something I really do not want to have to do because atm what I am using to search is strfind(), and I am not massively comfortable with how to use it properly.

I dunno, I'll see what I can come up with, any suggestions are welcome

Answer 2

[Stephen Doe]

If you have MATLAB R2014b you can use the webread() function to get data from a web service.

If metacritic provides a RESTful API, they might have a service that returns data as JSON objects. If so, webread() can convert JSON to MATLAB structures. See webread for more details. This would be much simpler than the string manipulation you mentioned in your question.

You will have to search the metacritic web site for a RESTful API. They might require you to sign up for an account before providing access.