Home Page

What Do You Know About Propensity Score Matching?

January 6, 2015

by:

Joe Patti

While it was relatively quiet in the office over the holidays, I made an attempt to catch up on reading reports that I had downloaded and bookmarked over the last few months.

In the process, I came across Measuring Cultural Engagement: A Quest for New Terms, Tools, and Techniques which is a summary of a symposium of “Cultural researchers, practitioners, and policymakers from the U.S., the UK, and other countries” held in June 2014.

Instead of telling you about what I read and evaluating it, I actually wanted to ask- Does anyone know anything about Propensity Score Matching?

Well, obviously I guess I probably should do a little explanation for people.

It is a statistical method that has been around for about 30 years, but this is the first I have heard of it. It’s application to the arts is discussed on page 18 and sounds pretty interesting, but I am not quite sure if it is something an individual arts organization could engage in themselves.

According to Measuring Cultural Engagement (MCE):

“The Norman Lear Center adapted PSM to evaluate the impact of media and arts programming. The idea is to isolate a piece of media or arts programming to assess whether audience members who were exposed to it were more likely to demonstrate a shift in knowledge, attitude, or behavior compared to very similar people who did not encounter the programming”

The reason this technique can be valuable to the arts is because it is often difficult and expensive to identify a representative sample group of people who have participated in a niche event. Yet arts groups often need to gather data from people in support of grants and it is often difficult to get the data you really need: (my emphasis)

“One key problem in measuring cultural engagement is confusing outputs with outcomes. It is easier to tell funders how many seats or tickets were sold or the number of “likes” on Facebook than whether a particular arts or cultural event had a substantial impact on an individual or a community. Since many cultural agencies and organizations, including the NEA, talk about the benefit or value of arts and culture to individuals and communities, it is essential that the research community develop pragmatic tools to help these groups demonstrate that their mission is being accomplished. Using PSM in this way, arts organizations can focus on outcomes instead of outputs, measuring the impact of their work on individuals and communities.”

The example used in MCE is evaluating whether people who saw the movie Food, Inc had a experienced a change in knowledge and attitude. The Normal Lear Center used surveys distributed through social media groups and email lists affiliated with the film and production company. They received about 20,000 responses.

MCE acknowledges that one of the weakness of Propensity Score Matching is that it requires a pretty large sample size, but that the Lear Center has been able to get good results from as few as 1,000 surveys. This is one of the reasons I was wondering if it is at all viable for an individual arts organization.

Being able to get results focused on outcomes rather than outputs sounds great–if it is something that can reasonably be done. Has anyone out there had any experience with Propensity Score Matching?

Something MCE mentioned that intrigued me but wasn’t expounded upon enough was (my emphasis):

“Seventeen statistically significant variables were identified that predicted the likelihood of seeing a film like Food, Inc. Of these, only three were demographic. This surprised the film’s marketing team as demographics usually form the basis of film marketing. The three variables focused on whether a survey participant was employed in certain industries or had children. Individuals were more likely to see the film if they did not have children. This was contrary to what the marketers expected.”

I really wanted to know what the nature of the other 14 significant variables were if they weren’t demographic. Arts marketing focuses pretty heavily on demographics as well so it would be really interesting to know what types of factors made up the majority of the significant variables if they weren’t demographic.

Audience Relations, Marketing/PR, State of the Arts, Terms and Ideas

3 Comments

You Can Build A Toilet, But You Can’t Make Me Use It!

January 5, 2015

by:

Joe Patti

There may be no greater evidence that increasing arts in schools won’t create more arts patrons/lovers than the fact people in India won’t use toilets.

Yes, while that statement may be a cheap ploy at getting you to read the rest of this post, there is some truth to it.

According to a CityLab article, even though the government of India has gone on a huge toilet building campaign in order to improve sanitation, so many people refuse to use the commodes that the government is pondering a monitoring program to make sure people do. (my emphasis)

In a recent survey of 3,200 rural households by Delhi-based Research Institute for Compassionate Economics, half of respondents who didn’t have a toilet believed that “defecating in the open is the same or better for health than using a latrine.” Most people who owned a government-constructed latrine still chose to use the outdoors. Some end up using their loo for storage or extra living space.

Many people in the article talk about the use/non-use as a factor of cultural and religious values and advocate for everything from education campaigns to social pressure and spying in order to achieve the goal of no public defecation by 2019.

It can be perplexing to read about the difficulties this effort faces despite the clear and nearly immediately demonstrable health benefits of sanitary practices (Not to mention the government will do the construction.)

We might think that the benefits and importance of using a toilet would be self evident, but apparently that is not the case in India. In that context, saying something that is far from self-evident like arts education improves test scores starts to sound a little weak as an argument.

Comparing toilets in India to Arts in America is probably rife with more flaws than I am immediately imagining. But there is a similarity in the societal inertia that needs to be overcome. Because India is a situation outside our experience, we can examine it with some objectivity and recognize some of the common issues we face.

Parents and schools providing arts experiences to kids certainly contributes to socializing young people to participate when they get older. When I look at India, it reinforces my sense that any effort to help people recognize the presence and importance of arts in their lives is going to involve a lot more than just advocating for more arts teachers in the schools.

Advocacy, Art(ist) Education, Arts in an Age of Technology, State of the Arts

3 Comments

Info You Can Use: Good Fellowship Opportunity

December 31, 2014

by:

Joe Patti

Tomorrow as the new year dawns and you make a resolution to be the best darn arts administrator you can be, you might want to consider starting an application to Oregon Shakespeare Festival’s new arts administration/leadership fellowship.

According to American Theatre, the Paul Nicholson Arts Management Fellowship is meant to increase diversity among arts leadership.

The seven-month fellowships are for professionals interested in a career in arts management and artistic leadership. Candidates need only be interested in leading an arts organization; a background in theatre is not required. The chosen fellow will work in close capacity with OSF managing director Cynthia Rider, and will lead projects across multiple administrative departments.

Johka emphasizes that while the Paul Nicholson Fellowship is designed to give potential leaders of color an opportunity to develop professionally, the application process is open to all, not just candidates of color.

“You can’t make a claim to have a commitment to diversity and inclusion while maintaining that in order to bring people in, they must have a long history in a field that has been traditionally all white,” Johka adds. “You have to open up your criteria to accept and value all kinds of transferable skills.”

Just in case you breezed over the sentence I bolded above, it is not necessary that you have a background in theatre.

Deadline is February 1 and the fellowship, which includes housing, transportation and a stipend, begins March 31, 2015. Application is accepted online.

Seems like a great opportunity. My only nitpick is that the application process requires three letters of recommendation to be attached at the time of submission. I realize this is likely to be a highly competitive process and there needs to be a weeding out process. However, if the goal is to attract people of diverse backgrounds, requiring three letters of recommendation may end up excluding people who have advanced along the less traditional path with “all kinds of transferable skills” they are seeking.

Getting quality letters of recommendation for your application file in a timely manner is often even a difficult task for university doctoral students who operate in a culture in which they are de rigueur.

Arts in an Age of Technology, State of the Arts, Working In The Arts

No Comments

Not As Simple As Adding an “A” To STEM

December 30, 2014

by:

Joe Patti

When the concept of teaching a STEM curriculum in schools started to take off a few years back, I was among those who quickly started advocating for turning STEM into STEAM.

However, a thoughtful piece on Education Week gave me pause and made me realize blithely calling for the arts to be inserted without really understanding what the curriculum was all about is somewhat akin to deciding an English teacher directing a play after school is sufficient to meet arts standards.

According to author Anne Jolly, the arts isn’t something you can or should simply plug in to STEM like a Lego block. (my emphasis)

Recently, the idea of adding the arts to STEM programs has been gaining momentum. Surprisingly, I’ve heard push-back from both camps:

1. From STEM proponents: STEM lessons naturally involve art (for example, product design), language arts (communication), and social studies and history (setting the context for engineering challenges). STEM projects do not deliberately exclude the arts or any other subject; rather, these subjects are included incidentally as needed for engineering challenges.The focus of STEM is developing rigorous math and science skills through engineering. How can you focus on other subjects (such as art) without losing the mission of STEM or watering down its primary purpose?

2. From arts proponents: Engineering and technology can certainly serve the artist and help create art. But if we’re talking about how one can use art in engineering… as an artist, it seems we’re missing the point and devaluing, or not realizing, art’s purpose and importance. We have it backwards.

Jolly goes on to make some suggestions about how the arts can have a place in a STEM program, but none of them really feel like a clean melding of ideals, and she admits as much. As she points out numerous times in her writing, a good STEM program will never exclude an artistic component.

“Just one word of caution, though. Art is often touted as a method of adding creativity to STEM—but keep in mind that engineers are rarely lacking for creativity and ingenuity. Just look at the world around you for proof. The purpose of STEAM should not be so much to teach art but to apply art in real situations. Applied knowledge leads to deeper learning.”

This is one of those articles where every commenter has something insightful to add to the conversation. No one manages to come up with the definitive approach, but they draw attention to the issues that need to be considered in order that neither Arts or STEM get short shrift in a relationship.

Advocacy, Art(ist) Education

4 Comments