GAMES: GameSpot: Best of 2008 | GameFAQs | SportsGamer MUSIC: Last.fm | MP3.com MOVIES: Metacritic | Movietome TV: TV.com
Previous  |  Next Showing 1 - 8 of 8
The wide world of reviews

(And the users who ignore them.)

A while back I wrote a blog entry on the correlation (or lack thereof) between review scores of games on the three consoles of this generation and the sales of their games. Although this was all well and good, there was one thing that this did not really provide to the reader: namely, relevance to the ultimate purpose of gaming, that being entertainment. Judging by the fact that user scores did not correlate very strongly with sales either, we can reasonably conclude that it is probably not the case that greater sales of a game necessarily imply a greater degree to which those who purchased it enjoyed it. In other words, the mere fact that many people bought a game - often against the advice of reviewers everywhere - doesn't necessarily mean that they had fun playing it.

Indeed, probably one of my greatest mistakes when writing that entry was the unjustifiably rosy conclusion that I inserted, in which I said that people were buying games that they want to play when they ignore reviews. I was called on that statement in the comments section, and those who did so were absolutely right: the evidence I presented did not at all support that statement at all, and I was wrong in making it. It might have been more accurate to say that people are buying games that they think they want to play. There was, after all, no strong correlation between user reviews and sales, either.

But nonetheless, the results of that study do raise a rather pertinent question: If people ignore the reviews of a game that they buy, is that likely to be a good idea? Are they likely to be vindicated, or will they instead probably curse the fact that they didn't listen to the reviewers? To what extent ought people trust a review, if they are unsure whether or not they will enjoy a game? How often do players in general and critics agree, and where do they agree? These are all questions that I intend to investigate within the bounds of this article.

To begin with, I went mining for data once again at Metacritic and at GameSpot. This time around, I concerned myself with two points of data per game: the Metacritic average and the GameSpot user average. The former I used to gauge the general feel of the critics for a game, and the latter I used to gauge the general feel of the masses for a game. To ensure that the overall score could not be influenced too much by one single person, I filtered out all of the games that did not have at least ten reviews both by critics and by users.

People will surely tell me that this is an inexact way of doing things, especially the GameSpot user average (it being a self-selected sample), and on this point they would be correct. Nonetheless, given that the question posed above concerns itself with those who bought a game, not all those who were uninterested in it from the start, I feel that this is a good enough metric to use with which to examine the question.

Once I had my data, I next separated games into a number of categories. A well-received game was one where both critics and users gave it an average score of at least 8.0 (e.g., Metal Gear Solid 4 got a 9.3 from critics and a 9.4 from users). An average game was one where both critics and users gave it an average score of at least 7.0, but less than 8.0 (e.g., Mortal Kombat vs. DC Universe got a 7.5 from critics and a 7.8 from users). And a poorly received game was one where both critics and users gave it an average score of less than 7.0 (e.g., Wii Play got a 5.8 from critics and a 6.4 from users).*

Astute readers will notice, however, that all three of these designations are ones where the critics and users agreed, on average, on the overall quality of the game. I also made two additional designations: a user-favored game was one where critics' overall assessment of it was one level less than the users' (e.g., critics thought it average while users thought it well-received), and where the users' average score was at least 0.5 points higher than the critics' average score (e.g., Spectrobes got a 6.3 from critics and a 7.5 from users). A user favorite game was one where critics thought it poorly received, but users thought it well-received (e.g., Tales of the World: Radiant Mythology got a 6.6 from critics and an 8.3 from users).

One could also invent the designations of "critic-favored game" and "critic favorite game", but these proved to be utterly nonexistent: out of the hundreds of reviews I sifted through, in no case did critics ever review a game significantly higher than users did. So I will not include these designations for consideration.

Finally, I denote that the users agree with the critics when the average critic score is within 0.5 points of the average user score. The percent agreement for a given console is the percentage of time this is the case for the games on that console.

Note that all of these designations are completely arbitrary - one could alter them to get slightly different results. I assert that they are reasonable, however: the range of 7.0-7.9 is indeed a roughly average score for a video game, and GameSpot presents an increment of 0.5 points as the smallest tangible differentiator between two games. Both of these facts I have effectively used verbatim in this article.

Now that we have both the data and the terminology in place, we can analyze the makeup of the libraries of each of the three current-gen consoles and the two current-gen handhelds:

PS3

Well-received: 42%
Average: 23%
Poorly received: 15%
User-favored: 18%
User favorite: 2%
% agreement: 67%

Xbox 360

Well-received: 32%
Average: 21%
Poorly received: 22%
User-favored: 23%
User favorite: 1%
% agreement: 56%

Wii

Well-received: 23%
Average: 18%
Poorly received: 23%
User-favored: 31%
User favorite: 3%
% agreement: 41%

DS

Well-received: 28%
Average: 18%
Poorly received: 16%
User-favored: 30%
User favorite: 8%
% agreement: 40%

PSP

Well-received: 24%
Average: 9%
Poorly received: 9%
User-favored: 51%
User favorite: 7%
% agreement: 24%

(Note that these may not add to 100% due to rounding.)

At face value, the Wii's detractors would seem to be vindicated at least somewhat: in terms of percentages, the PS3 does, in fact, have almost twice as many well-received games as the Wii, and the Wii does indeed have more poorly received games than the PS3 (with the Xbox 360 in the middle in both cases). However, if you take a second look, the picture becomes less clear: the Wii, in turn, also has almost twice as many user-favored games than the PS3. In fact, both sides in System Wars appear to be right to a certain extent: while it appears to be the case that the Wii does indeed have worse games than the PS3 and Xbox 360 on average, it also appears to be the case that critics reviewing Wii games do, in fact, not seem to quite "get it" to the same degree that critics reviewing games for the other consoles do. In a blog entry a while back where I discussed reviewer bias, it was suggested in the comments section that perhaps all reviewers were biased in the same direction. I said at the time that I did not know how one would gain evidence for this assertion, but perhaps the above might actually qualify, although I say this only as a passing musing that I imagine may well not stand up to serious scrutiny.

Interestingly enough, however, the biggest rebels of the consoles' and handhelds' userbase were not the Wii gamers, but in fact the PSP gamers. Fully 58% of PSP games were rated significantly higher by the users than by critics, and on only 24% of PSP games - that's less than a quarter! - did users agree with the critics' precise assessment of the games. This came as quite the surprise to me, but there it is. I won't claim to understand why this is the case - users are free and invited to venture their own theories if they have them.

But back to the question at the beginning. Based on this data, what can we say with respect to the question of whether or not people are liable to regret it if they don't listen to the critics when buying a game? Well, we can more or less say "it depends".

On consoles, it would certainly seem to be the case that the critics are more in line with the users on the PS3 than the users on the Wii. In 80% of cases PS3 users agreed with the critics' rough evaluation of the game (well-received, average, poorly received), and users agreed with the critics' precise score 67% of the time. On the Wii, on the other hand, in only 64% of cases did Wii users agree with the critics' rough evaluation of the game, and users agreed with the critics' precise score only 41% of the time. On the PS3, in only 20% of cases did users score a game significantly higher than the critics, whereas on the Wii that figure rises to 34%. In other words, on the Wii, over 1 out of every 3 games released is received significantly better by the users than by critics, whereas on the PS3 it's only 1 out of every 5 games.

On the other hand, however, on both consoles it was far from a majority of cases where users broke from the critics in terms of liking a game significantly more.

On the handheld side, the picture is much bleaker for critics - the DS looks much like the Wii, except for having more games well-received by both parties and fewer games poorly received by both parties, and the PSP is just off in its own little world, with users in agreement with critics' rough evaluation of a game only 42% of the time, and rating games significantly higher 58% of the time - well over a majority of cases.

So what should we conclude? Well, the conclusion should come as no surprise to one who is a savvy buyer of video games: reviews are generally not useless, but at the same time they are most certainly not the be-all and end-all authority of quality. People will very likely find it useful to read the review just to learn the bare facts of what is in the game at least as much as to listen to the reviewer's subjective opinion of what is in the game. Those whose main console is the PS3 are the most likely to find that their subjective assessment of a game is in line with the critics', while those whose main console is the Wii are the least likely. Those whose main handheld is the PSP should probably just completely ignore review scores, as in this case they actually do appear to be almost useless in terms of predicting whether or not the reader will agree with the critics' assessment of the game being reviewed.

I know that this almost certainly will not stop those who consider an 8.5 on a single site to be a flop while considering a 9.0 on that site to be pure AAA gold. But there it is, and I hope at least someone can make use of it. And, of course, if you've already got a scheme that works perfectly for you, don't let me tell you otherwise.

*Note that this is not quite precisely true, but I didn't want to take up even more space to enunciate the precise definition since it is not entirely straightforward. To be completely precise, a game is considered well-received, average, or poorly received if both critics and users rate it with that score, or if the critics rate it with that score and the users' rating is less than 0.5 points away. For example, a game that the critics rate as 8.1 and the users rate as 7.9 will still be considered well-received. This is to avoid the oddity of having percentages not add up to 100% since, for a game to be user-favored, there must be at least a 0.5-point difference between the scores.

posted Monday, December 14, 2009 1:35am  |  Comments (56)
Twitter

My Twitter profile is HERE.

posted Sunday, December 13, 2009 3:05pm  |  Comments (6)
X-7
My first blog on here in who knows how long. COD MW2 and L4D2

I just preordered COD MW2 on Amazon. I got $20 bucks off my next game I order on there. Which will probably be L4D2. I also got RE5 for my PC. (BTW I will get all these on PC)

posted Monday, November 9, 2009 6:40pm  |  Comments (2)

Band Hero Demo Preview
posted Sunday, October 25, 2009 2:16am  |  Comments (2)
Wow I haven't posted here...

What's up guys, many of you know that I am still active elsewhere. For those gamers that are interested I stop posting here because of the limitaions on video's.

For those who wish to follow me you can visit here. Peace guys.

posted Tuesday, July 21, 2009 12:13pm  |  Comments (3)
I still make videos on youtube

Hey you guys in case you were wondering if I still make blogs than yes I do I make them on youtube. I have been making videos for awhile on youtube. So if you want to know what I have bought or how I am doing than please check out my youtube channel here is my link to my channel.

http://www.youtube.com/user/CoolMan2423

P.S I still update my game lists on GS so if you want to know what games I have bought then you can check out my GS games list.

posted Sunday, June 14, 2009 12:54pm  |  Comments (2)
It's time for farewell my friends... My Final Blog.

Well it's been about 3 or 4 weeks since my last blog and I figured that my life here is kind of dying slowly. So, I thought I would just give you an update or two into things in my life, in what will be one of my last blogs here.

I guess I started my blogs here not realising what it would turn into. I started off with my first blog back in September 2005 with a kind of rant about San Andreas. About how angry the game made me plus a mega-rant about the final mission. I then started blogging every time I levelled up here on Gamespot and my blogs kind of developed more and more into my personal life. And looking back on them, I've noticed a vast improvement in my grammar, typing speed, and that I've made some great friends here on Gamespot.

I suppose the deeper I went with my blogs into my personal life the harder it is to take steps backwards away from it and so I think with this blog I will give you my last few weeks and what I hope to come in my life.

Well my last blog was about The Dark Knight which has quickly become one of my favourite movies and has me almost infatuated with Heath Ledger... Who now ranks highly with my favourite actors... Christian Bale, Ben Affleck, Mark Wahlberg, Clive Owen, Gary Oldman, Jean Reno, and the list could go on and on. So, I have now seen The Dark Knight 5 times; once more and I match my personal best of Small Soldiers in the cinema when I was younger.

The job at the local Cinema is going well, its long hours and pay isn't pretty but getting a payday every fortnight is nice and you can't really argue with free films!

Oh and I almost forgot, yesterday our A Level results were out. Considered I could have tried harder if I were aiming for going to University or something, I am quite pleased with my 3 passes But as I said, right now, I'm not interested in going to Uni, when you start earning money Full Time, it's tough to go back to learning and earning less. With my first paycheck I thought I'd treat myself so I bought a limited edition hand-painted 10" Joker figurine... With only 6000 being made and officially licensed this is surely an investment at £80 too. So that was a sort of congratulations to myself, not many people nowadays apply for their first ever job, and get a first ever interview and get the job! I have since bought a new cell phone/mobile phone with my paychecks and about 40 new DVDs in the space of a month

In future, I intend to rent a place with my best friend and just have a lot of studentesque fun. Obviously right now I am happy with the money I am earning considered the type of work I do, but will certainly try to further myself into something more suited for my abilities. I don't know what I will be doing with my life as nobody can predict the future, which is scary, but I gotta hope for the best I suppose.

So, I suppose this is goodbye.

It has been a rollercoaster ride over the past 3 years or so but overall it's been a nice outlet for me to spend some free time, chat about some of my favourite things in life.. Games all over the site, films at The DVD Collectors Union, and all things British in the UK Discussion

I would like to thank everyone who ever took an interest in my blogs and life, and in some cases for your kind and inspiring words. Thanks to you all and I wish you all the best in whatever you do in your lives.

Much Love...

Le-Beave

P.S: Keep on gamin' dudes & dudettes!

posted Friday, August 15, 2008 4:09am  |  Comments (26)
Happy 4th of July everyone!!!!!!!!!

Sorry it's so late but I had to work tonight, earlier before work went to my girlfriend's parents cookout....pretty fun with good food...well hope you all had fun cooking out and watching fireworks..have a great weekend

posted Saturday, July 5, 2008 2:53am  |  Comments (5)
Data Warehouse Clear Gif