December 11, 2008

What you need to consider, if you’re considering rating widgets

Previously posted on the Capstrat blog. –Todd

You see ‘em everywhere. They’re on CitySearch; they’re on Yelp; they’re tucked away in the footers of knowledgebase articles. Content ratings. Those little 5-star widgets and their kin. For such an innocuous little widget, the decision to put them on your page brings up all kinds of philosophical questions.

So, if you’re considering putting these things on your site, read further. Below is a primer on what goes through my head when I hear the feature uttered.

Business goal
What should the rating achieve? In corporate intranets, for example, rating might be used as a community-generated way to notify content authors that their document needs to be updated or deleted.

In a review site like Yelp, it might be a way to help bubble-up restaurants that other people liked. This helps distinguish Yelp from its competition and, potentially increases its value to the user.

For a recommendation service like Netflix, this might be a way to help develop a profile of what a user and a community likes.

Takeaway: determine what problem you’re trying to solve and then choose the proper rating widget and rating experience.

Axis of rating
What does the rating mean? Conducting usability tests on our sites, we often ask “What did you rate that content on?” The answers are eye opening-they include usefulness, to relevance, to accuracy, to personal agreement.

The first two are especially compelling. Usefulness and relevance are tied to what the user came there to accomplish. A technically accurate piece of content-but one that is not relevant to a user’s task-is likely to be rated irrelevant. That’s no good.

Takeaway: establish up front what that rating represents.

User momentum
Most of the time, someone rating a page is diverging from the task they came to your site to do. If you want people to rate content, you need to put up the minimum amount of barriers for them to do so. AJAX-enabled ratings as almost a necessity, since users can rate content quickly without breaking stride with a page refresh.

But consider if the people rating need to log in. That’s a big ol’ speedbump in their experience and will absolutely impact the volume of ratings you receive.

Takeaway: require the absolute minimum overhead to rate content.

Who gets to rate
In concert with user momentum, in some cases you might only want authorized users to rate content. In others, you may take a more laissez-faire approach and open it up to everyone.

But consider what may happen if you choose either. If you require users to log in to rate content, many aren’t going to go through that trouble.

If anyone can rate, you have to deal with people skewing the results by rating more than once. Plus, you can’t build user profile data off it, since those users are untracked.

Takeaway: consider the balance between volume of ratings and the need to impose access restrictions.

Content Volatility
If the owner of rated content decides to change it, is the community rating still accurate?

This is especially relevant in user-generated content is rated by a community. If it changes, should it be reset to zero? For example, if I rated something as very useful, then the author changed in a materially adverse way, I might get irritated that my rating no longer represents how I feel about that content.

Conversely, if I am a content author, I might get aggravated if you don’t let me update the material I contributed. This is beyond a buzzkill. It can be a community killer, simply because users lose the privilege to manage the content they took the time to give you.

One tactic is to take the middle road. For any content that is provided, some of it remains editable and some doesn’t. I’m calling this the “material change” pattern and it suggests that some parts of the content cannot be changed, but others can. If I create a recipe, for example, maybe I can change the description of it but not fiddle around with the directions or ingredients.

The problem here is that users don’t always know why some things can be changed and others can’t. That can degrade user experience and their satisfaction.

Takeaway: consider the impact of allowing ratings on content that can change.

Incentives
This is a bit of a corollary to “momentum” above. Namely, what are you doing to encourage people to rate?

Many studies suggest that only a small slice of visitors rate anything, period. Couple that with the fact that those who do rate, usually only do so for a few items at a time. If you have a ton of traffic, that might not be a big deal. However, for those of us who aren’t Amazon, getting an accurate pulse on the community’s reaction to content can be statistically difficult.

Netflix and Pandora are models of good incentives. Both tie user ratings back into their recommendation engine, serving you more accurate suggestions the more that you rate. Both impose little overhead, so it’s quick and pleasurable to give your one-star salute.

Take away: pay it forward. Give users a reason to rate, whether it’s tying the action back to their community ranking or giving them more relevant content.

The object and scope of the rating
What exactly is being rated? Is my vote cast for the page, the site as a whole, or just this specific little nugget of content? Proximity and visual cues play a big part is drawing the relationship between the rating widget and its scope.

Takeaway: consider where you place rating widgets. Use text and design to clarify the scope of the search. This can be as simple as “How helpful was this article?”

Anchoring effect
This is something I borrowed from behavioral economics. If you provide the community’s overall rating when the user rates the content, it often biases their rating. People rate things higher when other people rate them higher.

Takeaway: be wary when interpreting results. Ratings are usually not statistically valid evidence about how people feel about your content.

Motivation to rate
People tend to rate anything when they have very strong opinions about it and less often when they are indifferent. Ratings, then, can often inaccurately reflect the sentiments of all users simply because some people hated or loved the material enough to take the time and express their opinion. This goes back to the anchoring effect, as well

Takeaway: again, we careful in the interpretation. You’re likely to be skewed.

So, those are just some of the first things that come to my head when thinking about ratings.

PS: Rate the needless minutae in this article (1 – 5, where 5 represents “chock full” and 1 represents “none at all”.)