Guest Post: Measuring Success at the Base of the Pyramid

This post first appeared on Acumen investee VisionSpring’s blog, Business in a Bag. We’ll be cross-posting with Business in a Bag from time to time.

The post’s author is Tim Johnson-Aramaki, a student at the University of Michigan’s Ross School of Business, who spent the summer with VisionSpring India working on a data-collection methodology to measure the long-term impact of VisionSpring’s work on the lives of Vision Entrepreneurs and customers. His project is part of a multi-year impact study conducted by Professor Ted London at Michigan’s William Davidson Institute.

Over the last few months, I’ve been working to develop a survey instrument with the VisionSpring team here in Hyderabad and the William Davidson Institute team in Ann Arbor. The first step was survey pre-testing, which involved conducting countless interviews in rural village throughout the state of Andhra Pradesh. These interviews are meant to help us discover whether the questions we’ve come up with are understood by respondents with varying semantic and cultural backgrounds, and if they prompt valid and appropriate responses. Some of the results we’ve gathered have been really interesting.

For example, one of the most critical pieces of data in measuring VisionSpring’s impact is the income of its Vision Entrepreneurs and customers. It also happens to be one of the most difficult things to measure as there are challenges when it comes to discussing money. Through our interviews, we’ve found that while people are relatively open in assigning a number to their income, that number may not be accurate. There are a variety of reasons for this, but one is that they fear the income figures may be passed on to state or national agencies, potentially jeopardizing the public assistance they receive. To avoid this, they often provide income figures lower than that which they actually earn.

Responses can also be skewed because interviewees seem to identify the interviewer as higher up on the social ladder. With that in mind, the interviewees often attempt to please the interviewer with superlatively positive answers about product or service quality and refuse to make any constructive criticism. Moreover, in areas of the survey that allow for subjective self-measurement, many interviewees have attempted to “impress” the interviewer with potentially inflated responses. For example, we had to alter an entire section that measured an individual’s self-assessed capacity in tasks such as interpersonal communication because nearly all the responses to these questions were either a four (”agree”) or five (”strongly agree”). There would have been no improvement to measure!

Working through these issues to get an effective impact-assessment tool for the base of the pyramid has been challenging but rewarding. What’s so great about this project is that while it’s being made for and by VisionSpring, ultimately the survey and survey-development practices developed here in Hyderabad could really benefit other organizations working at the base of the pyramid in helping them measure the success of their work.