Sunday, 30 June 2013

The ratings game


We're all used to rating things. Whether that's out of five, ten, or 100, we do it instinctively in this online world. Rate books; rate films; rate recipes; and even ratings for more unsavoury things from the slimy depths of the Internet.

Generally we assume we know what different scores mean. We may have our own system, or we may check what the site says scores mean. But overall I think we like to assume that equivalent scores mean the same. And if I give a book three stars on one site, I would also give it three stars on another - after all, they mean the same, right?

Unfortunately, they don't.

I was interested in how different book sites that all use a five star system interpret each score. My books are often reviewed and rated on these sites, and some readers are kind enough to put their ratings and reviews on more than one site. I found some interesting things. Come with me as we delve deeper.


Okay, let's start with Amazon. If you hover over a star rating when adding a review you'll see what that rating equates to, according to Amazon.


Amazon
1 star = I hate it
2 stars = I don’t like it
3 stars = It’s OK
4 stars = I like it
5 stars = I love it

And now Goodreads.



Goodreads
1 star = Did not like it
2 stars = It was ok
3 stars = Liked it
4 stars = Really liked it
5 stars = It was amazing

And Shelfari...


Shelfari
1 star = I hated it
2 stars = I didn’t like it
3 stars = I liked it
4 stars = I really liked it
5 stars = I loved it

And finally LibraryThing.


However, LibraryThing doesn’t define what the stars mean. This could actually be the best approach, just leaving it up to the reader!

Let's put all of that together.


There are lots of things to spot here.
  • Amazon has 3 stars as ok/neutral/average. There is no such rating in Shelfari, which jumps from ‘didn’t like’ to ‘like’, negative to positive. There is no option for a neutral score.
  • 3 stars on Shelfari is the same as 4 stars on Amazon.
  • 1 star on Amazon is deeply negative, hate. 1 star on Goodreads is just the absence of liking, and is nowhere near as negative. In fact, Goodreads has no rating for hating a book. 
  • 1 star on Goodreads is the same as 2 stars on Amazon; 2 stars on Goodreads is the same as 3 on Amazon; ditto with 3 on Goodreads/4 on Amazon. Basically any score on Goodreads means a less favourable judgement when pasted across to Amazon.
  • The rating 'I don’t like it' [Amazon & Shelfari 2 stars; Goodreads 1 star] is ambiguous. It is actually neutral, just showing the absence of a positive feeling, but that could mean it is neither good nor bad - just average. However, many people interpret this as meaning 'I dislike it' - which isn't technically true. So this term leads to problems.
So the way scores are interpreted on all the main book review sites varies quite a bit. If you paste your reviews and scores identically on multiple sites, you are really saying different things.

I don't think we, as users, should change that behaviour and alter our scores depending on what site we post a rating on, since any of these sites could redefine what the stars mean at any point. It would make much more sense for the sites to all use the same system. The fact that they don't is even stranger when we consider the fact that Shelfari is owned by Amazon, and they have now also bought out Goodreads! So three of the biggest book review sites are under one umbrella, yet are inconsistent in how they interpret the same scoring scale.

If you rearrange the descriptive terms from negative to positive, you see this:
  • I hate it [Amazon & Shelfari 1 star; no Goodreads equivalent]
  • I don’t like it [Amazon & Shelfari 2 stars; Goodreads 1 star]
  • It’s OK [Amazon 3 stars; Goodreads 2 stars]
  • I like it [Amazon 4 stars; Goodreads & Shelfari 3 stars]
  • I love it [Amazon  & Shelfari 5 stars; Goodreads 4 stars]
  • It was amazing [no Amazon equivalent; Goodreads 5 stars]
  • [Note that Shelfari also has 4 stars 'I really liked it’ which would seem to be the same as love it, but for that they have 5 stars, so theirs is the most confusing. ]
Bearing all that in mind, I would propose this as a star system for all sites:

1 star = I hated it
2 stars = I disliked it
3 stars = It was okay
4 stars = I liked it
5 stars = I loved it

If you use any of the sites above then contact them and ask them why they aren't being consistent and all using the same system. Feel free to point them here for background, and make suggestions in the comments below for how you think it should work. I will contact the sites too, and post any updates here.

By the way: sometimes the words are more important than the score. I'd rather have a perceptive 4 star review than a 5 star "omg awesome!!!" (Even though that's lovely too).

Update 1st July 2013: Amazon responded thusly: "Thank you for writing to Amazon with your valuable suggestion about the rating system for Kindle books. I’ve forwarded your suggestion as feedback to our development team, who will carefully review your comments and will consider your feedback as we plan further improvements. It is always important for us to hear how customers react to all aspects. Your valuable feedback will help us to improve the selection and service we provide and we're glad you took time to write to us. As always, please feel free to contact us if you have future suggestions, questions or comments. We have team of experts who takes into consideration each and every feedback of our customers and brings them into practice."

Update 16th July 2013: Nothing else from Amazon (e.g. what they are actually doing); and not even a reply from Shelfari, LibraryThing or Goodreads.
Share:

4 comments:

Anonymous said...

Hmm thought provoking. I have, over time, mellowed a bit and realised that authors have feelings too, and I think the really really strong negative use of 'hate' is a bit harsh. If someone hated a book so much it's likely they would have stopped before they finished it. Therefore I would be tempted to not have such a negative and mean score.

I would therefore be more inline with Goodreads, which I use, but, I have found rating a book as only 2 stars when it's ok looks a bit mean as well, so I sometimes bump it up to a 3 stars. Two looks like it failed, or there were major flaws but the text relating to 2 (It was ok) fits my sentiments: it was ok, not brilliant, but I wasn't really that bad.

So, I'm not sure where that leaves me/you/the ratings! Ah, a brain wave! How about 1=did not like, 2 = it was not my thing but I recognise it's a good book, 3 =it was ok, 4=like it (a lot), 5 = amazing! Gosh, it's hard, as that doesn't seem perfect either.

Karl said...

You make a good point, maybe we shouldn't encourage a culture of hating, and should go with the Goodreads scale. I also often feel that two stars is a fail on my own books, even though it translates as 'okay' - the psychology of perception behind scoring is quite emotional.

Maybe the simplest option is the Librarything option, just have stars and don't try and define what they mean?

Shaun Horton said...

I actually was considering discussing this myself, but you've done it a lot better than I think I could. The only thing I think I can add is that humans generally like things being symmetrical, which is where I think some confusion comes in.

A lot of people on Goodreads seem to think three stars is ok, while two and four stars are didn't like/like and one and five are hated it/loved it. That does make more sense than a didn't like/ok/like/really like/loved system to me, at least.

Karl said...

Plus you'll be busy promoting monsters all this month, Shaun! I agree with your comments on perceptions and a good way forward.