What Do You Know About Propensity Score Matching?

While it was relatively quiet in the office over the holidays, I made an attempt to catch up on reading reports that I had downloaded and bookmarked over the last few months.

In the process, I came across Measuring Cultural Engagement: A Quest for New Terms, Tools, and Techniques which is a summary of a symposium of “Cultural researchers, practitioners, and policymakers from the U.S., the UK, and other countries” held in June 2014.

Instead of telling you about what I read and evaluating it, I actually wanted to ask- Does anyone know anything about Propensity Score Matching?

Well, obviously I guess I probably should do a little explanation for people.

It is a statistical method that has been around for about 30 years, but this is the first I have heard of it. It’s application to the arts is discussed on page 18 and sounds pretty interesting, but I am not quite sure if it is something an individual arts organization could engage in themselves.

According to Measuring Cultural Engagement (MCE):

“The Norman Lear Center adapted PSM to evaluate the impact of media and arts programming. The idea is to isolate a piece of media or arts programming to assess whether audience members who were exposed to it were more likely to demonstrate a shift in knowledge, attitude, or behavior compared to very similar people who did not encounter the programming”

The reason this technique can be valuable to the arts is because it is often difficult and expensive to identify a representative sample group of people who have participated in a niche event. Yet arts groups often need to gather data from people in support of grants and it is often difficult to get the data you really need: (my emphasis)

One key problem in measuring cultural engagement is confusing outputs with outcomes. It is easier to tell funders how many seats or tickets were sold or the number of “likes” on Facebook than whether a particular arts or cultural event had a substantial impact on an individual or a community. Since many cultural agencies and organizations, including the NEA, talk about the benefit or value of arts and culture to individuals and communities, it is essential that the research community develop pragmatic tools to help these groups demonstrate that their mission is being accomplished. Using PSM in this way, arts organizations can focus on outcomes instead of outputs, measuring the impact of their work on individuals and communities.”

The example used in MCE is evaluating whether people who saw the movie Food, Inc had a experienced a change in knowledge and attitude. The Normal Lear Center used surveys distributed through social media groups and email lists affiliated with the film and production company. They received about 20,000 responses.

MCE acknowledges that one of the weakness of Propensity Score Matching is that it requires a pretty large sample size, but that the Lear Center has been able to get good results from as few as 1,000 surveys. This is one of the reasons I was wondering if it is at all viable for an individual arts organization.

Being able to get results focused on outcomes rather than outputs sounds great–if it is something that can reasonably be done. Has anyone out there had any experience with Propensity Score Matching?

Something MCE mentioned that intrigued me but wasn’t expounded upon enough was (my emphasis):

“Seventeen statistically significant variables were identified that predicted the likelihood of seeing a film like Food, Inc. Of these, only three were demographic. This surprised the film’s marketing team as demographics usually form the basis of film marketing. The three variables focused on whether a survey participant was employed in certain industries or had children. Individuals were more likely to see the film if they did not have children. This was contrary to what the marketers expected.”

I really wanted to know what the nature of the other 14 significant variables were if they weren’t demographic. Arts marketing focuses pretty heavily on demographics as well so it would be really interesting to know what types of factors made up the majority of the significant variables if they weren’t demographic.

About Joe Patti

I have been writing Butts in the Seats (BitS) on topics of arts and cultural administration since 2004 (yikes!). Given the ever evolving concerns facing the sector, I have yet to exhaust the available subject matter. In addition to BitS, I am a founding contributor to the ArtsHacker (artshacker.com) website where I focus on topics related to boards, law, governance, policy and practice.

I am also an evangelist for the effort to Build Public Will For Arts and Culture being helmed by Arts Midwest and the Metropolitan Group. (http://www.creatingconnection.org/about/)

My most recent role was as Executive Director of the Grand Opera House in Macon, GA.

Among the things I am most proud are having produced an opera in the Hawaiian language and a dance drama about Hawaii's snow goddess Poli'ahu while working as a Theater Manager in Hawaii. Though there are many more highlights than there is space here to list.

CONNECT WITH JOE


3 thoughts on “What Do You Know About Propensity Score Matching?”

  1. Hi Joe, yes, I have used propensity score matching in a different venue than the arts. It was in a study looking at workers’ compensation pension outcomes. When you have a subject where there are selection biases (for example, that the more educated are more likely to participate in the arts) then propensity scoring can help to control for the outcome to more precisely estimate the effect on outcomes. I think you explained it pretty well to a lay audience. I imagine it would be useful to use when you have a lot of data on attendees and non-attendees (or season ticket holders and not is more likely). If anyone has data they want to “play” with, let me know. I’m interested in doing more studies on socioeconomic phenomena in the arts. –Heather

    Reply

Leave a Comment