Evaluation Time: Your Social Impact Entertainment Report

You probably know this anecdotally: Good advertising influences people’s drink preferences, their electronics purchases and more. But what if we told you the media itself affects our thoughts and beliefs on socioeconomic and political issues?

We’re big fans of all kinds of Social Impact Entertainment (SIE) here at FUNKY BROWN CHICK. Oh, hell yeah! But how do we truly know it’s effective? Can media messages really change the way that people think and behave?

What we’re really talking about here is social behavior change communication. If you’ve ever heard the phrases “media impact”, “narrative change”, or “culture change”, these are all describing a similar concept. Whether it’s a film, tv show, game, or social media platform, the media we engage with is both a result of and enhances our views on “the big issues” — and each other.  

You may be thinking, “yeah, this sounds great, but how can we measure how this stuff actually works?” Well, hell. Here’s an idea: learn from the experiences of those who have already been doing the work. Read on! This social impact entertainment report helps you discover what several groups have learned about narrative change and evaluation strategies.

1. THERE’S NO ONE “RIGHT” WAY TO MEASURE SOCIAL IMPACT ENTERTAINMENT

Our founder’s degrees are in social science. We know there are many ways to measure impact. Your approach may depend on the type of media you’re working with. In case it’s helpful, here are some resources to guide you.

  • Media Impact Funders has a lot of info on how to evaluate different types of media. They, too, do not believe in a “one size fits all” approach. Per their report, metrics that are evaluated will depend on the project itself. They suggest that those doing evaluation consider tracking: community engagement activities
    • media coverage
    • political or legal outcomes
    • increases in diversity and representation
    • attention from influencers
    • internal capacity building
  • Impact Guide specifically deals with evaluating documentary film. Look through the toolkit to learn more about what “great” evaluation looks like. There’s also a handy-dandy virtual toolbox detailing the kinds of indicators and techniques that could be useful. 
  • #PopJustice: Volume 3 measured the impact of pop culture on how we view race by evaluating TV, radio, video games, and the internet. The report used the experimental method to review “a targeted group of academic studies and research reports” on this topic. The report also has a helpful section covering newer or soon-to-be-released evaluation tools.

2. DO MORE THAN SIMPLY LOOK AT NUMBERS WHEN EVALUATING A SOCIAL IMPACT ENTERTAINMENT REPORT

In fact, this Wired article mentions that while they may be easier to measure, “…simple metrics often aren’t enough when it comes to quantifying success … [t]here is more to assessing a complex system than looking at its growth, efficiency, and the handful of other qualities that can be quantified and thus measured.” 

So how can we assess a complex system? By looking at how cultural norms have changed. “Norm change” as data scientist Riki Conrey calls it, can be evaluated by looking at four distinct areas. These areas mainly focus on evaluating a community’s changed behaviors.

3. LEAVE ROOM FOR TRADITIONAL QUANTITATIVE ANALYSIS

  • This study by USC’s Norman Lear Center and public opinion tracker Zogby International relied almost exclusively on quantitative efforts. Researchers used surveys, “iterative clustering techniques,” and “statistical analysis techniques” to evaluate the link between Americans’ values, political views, and their entertainment habits. 
  • Consultants at Learning for Action have published a very thorough report about the impact of media projects have on individuals and systems. Using several case studies as examples, this report encourages people to use both qualitative (surveys and interviews) and quantitative (Google Analytics data, observation logs, stats from public records) measures to evaluate impact.
  • The authors of this guide maintain that evaluation around narrative change is a rather new concept. They suggest that interested parties should use information gleaned from different areas of research in a bid to develop metrics.

4. PAIR QUALITATIVE AND QUANTITATIVE METHODS WITH NEW TECHNIQUES TO EVALUATE NARRATIVE CHANGE

Writer, narrative strategist, and speaker Kirk Cheyfitz argues that we should think more broadly if we want to create new societal norms. He suggests looking at evaluation models like implicit association testing, indirect elicitation work, and cultural audits. These techniques rely more on psychology, measuring a person’s “affect, emotion, and unconscious associations.” These methodologies can be paired with more traditional efforts in order to direct and measure social change. 

There is no agreed-upon way to conduct evaluations on the impact of SIE right now. However, the evaluation of narrative change projects is an exciting and ever-evolving area. As researchers become more aware of the effects of narrative change, they are developing better tools to measure its impact. 


At FUNKY BROWN CHICK, Inc., we approach evaluation from a data-driven, sociological perspective. Our founder has both a B.A. and an M.A. in Sociology. She has also written extensively about the social sciences. So, yeah, we’ll say it again, we love data! Feel to contact us if you have any questions on SIE evaluation or program evaluation in general.