Consider the Source

We all know (or at least suspect) that when you survey a group and allow people to reply if they are moved to do so that you will generally get responses at either end of the spectrum and not many from the middle. People will really only go to the trouble of filling out a survey if they love or hate the survey subject.

Over on’s Machinist blog, they dealt with the same issue with online rating systems like TripAdvisor, IMDb and (Read the comments section on the entry for a lot of other insight into other weaknesses in these rating systems.)

“To see how an Amazon star-rating compares to society’s “true” opinion, Hu, Pavlou and Zhang conducted their own survey of one product, singer-songwriter Jason Mraz’s 2005 album, “Mr. A-Z.” In a survey of 66 college students, about two-thirds gave the album three or four stars. There were also a bunch of twos, some ones, and very few fives. On Amazon the picture is completely different. More than half of reviewers judge “Mr. A-Z” a five-star CD, while there are only a small number of threes, twos and ones.

Pavlou explains the lovefest by citing a specific kind of response bias, what he calls “purchasing bias.” In order to review something, you must have already purchased it. But people buy stuff they think they’re going to like — that’s why they buy stuff…Purchasing bias, Pavlou points out, is related to the price of a product; a higher price reduces the probability that someone who is unlikely to enjoy a product will buy it and review it anyway….If the Jason Mraz album was $200 rather than $11, then only die-hard fans would buy it and rate it, skewing its average review higher…The more expensive a product, Pavlou says, the more you should discount its high reviews.”

The article talks about a company called Summize which is translating all the star ratings into a thermometer bar like this one for the aforementioned Mraz album. Clicking on the various colors representing the good, bad and ugly number of ratings gives you direct access to the reviews with those ratings. Seeing all the reviews of each star category together rather than interspersed with ratings of other stars aids a little more in decision making. It can also reveal if marketing departments have tried to seed in good reviews at intervals when comments with similar syntax and spelling errors pop up side by side.

As the entry also points out, considering the source is still paramount when dealing with critiques. A reviewing site called Yelp allows you to cross reference reviewers with other reviewers of similar minds to evaluate if you share their taste and thus, have a higher degree of confidence in their opinions.

Refining software to compare our taste to those of others for us is what Web 3.0 is projected to be all about. (Web 2.0 is user generated content like blogging, Wikipedia, YouTube.) It is speculated that the next generation of web applications will search the internet for what we want and like a TIVO, will gradually learn what our preferences are in order to make suggestions. Presumably, we will be able to trade these specs with loved ones to aid them in Christmas shopping for us.

I imagine that as with Tivo, advertisers will be scrambling to figure out how to position their products in ways that the next generation of search agents will suggest them to consumers. (I am guessing they will pay software developers to have the agents favor them.)

The potential good news for arts organizations is that even if they don’t try to be manipulative in the type of Metawords they use in their web design, the artificial intelligences of the search agents may inform their masters that they have a high degree of confidence that they will enjoy a performance based on the years of criteria the agent has indexed even though they have never gone to see a show before.

I am sure that large corporations will see to it that software is developed enabling the agent to inform the arts organization website that this is the first time its master has purchased tickets to a show allowing the arts organization to offer great seats at reduced prices and perhaps flag the purchaser to receive free background information about the show and special attention by the front of house staff.

If the companies that develop these agents are smart, they won’t allow the users to be so specific in their criteria that they close themselves off from seemingly out of left field recommendations synthesized by the agent based on a profile it has compiled.

About Joe Patti

I have been writing Butts in the Seats (BitS) on topics of arts and cultural administration since 2004 (yikes!). Given the ever evolving concerns facing the sector, I have yet to exhaust the available subject matter. In addition to BitS, I am a founding contributor to the ArtsHacker ( website where I focus on topics related to boards, law, governance, policy and practice.

I am also an evangelist for the effort to Build Public Will For Arts and Culture being helmed by Arts Midwest and the Metropolitan Group. (

My most recent role was as Executive Director of the Grand Opera House in Macon, GA.

Among the things I am most proud are having produced an opera in the Hawaiian language and a dance drama about Hawaii's snow goddess Poli'ahu while working as a Theater Manager in Hawaii. Though there are many more highlights than there is space here to list.


1 thought on “Consider the Source”

  1. As someone who has previously advocated a Tripadvisor type system for theatre reviews I found the piece very interesting. Personally I’ve had nothing but success booking hotels through Tripadvisor, but I agree that you have to look beyond the rating. To some extent Tripadvisor does this by also asking reviewers to say whether a hotel is suitable for a particular type of guest – families, young couples, people with disabilities etc, but you do also have to consider the background of the reviewer for the full picture. For example the differing thoughts between european and american guests when describing the size of rooms in a city centre Florence hotel. I tend to treat such sites as I treat Sat Nav in my car – a great resource but I’m not going to follow it blindly and stop looking at roadsigns.

    The idea of “purchasing bias” is one that has caused me problems in the past with our theatre review site – View From The Stalls. We just do it as a bit of fun for the shows we pay to go and see – no press freebies for us. As a result we choose shows we see carefully and only go to ones we think we’ll like, with the consequence that the majority of our reviews are positive – more so than a professional press critic sent to comment on a random piece of theatre regardless of personal taste. It became a bit of an issue for us, and we didn’t want to be seen as overly positive, so had to write an editorial post “Are You Positive” explaining the mechanism described here as “purchasing bias”

    I do like the idea of Tivo style suggestions, but would always be wary of sponsored links etc, and at the end of the day there’s little to beat spending time getting to know the personal taste of a reviewer – press or online – and bearing that in mind when weighing up their comments.


Leave a Comment