metrics

World Watches in Shock and Awe

Mark your calendars: June 16 (tomorrow) is World Metrics Day! Our declaration of an official World Metrics Day (or WMD, as we’ve been calling it) is a little (well, actually, a lot) tongue in cheek, but Acumen Fund’s commitment to metrics is absolutely serious. Figuring out how to measure social impact has long been a challenge for philanthropists, social investors, practitioners and the development sector. Our focus has been on finding ways to collect, understand and use data to (1) best support the enterprises in which we invest, (2) improve how Acumen Fund works and (3) create common measures for the sector - in that order of importance. (We’ve blogged here before about metrics, on what we like to call cost effective cost effectiveness and our best available charitable option methodology.)

If you follow our work, you know that we’ve been developing a platform called Pulse, discussed in BusinessWeek and The New York Times when it was announced last fall at the Clinton Global Initiative. Pulse helps social investors track, measure, and evaluate operational data to understand whether different approaches to solving the problems of global poverty are working. Although originally designed as a tool to help us manage our own portfolio, we’ve been working with Salesforce.com to make it available for adoption by peer organizations and to share data across the sector. We have also been collaborating with the Rockefeller Foundation, B-Lab, Deloitte and PwC to develop standard definitions of social and environmental performance that can enable comparability across portfolios (for more on this, see www.iris-standards.org). About forty organizations are currently testing Pulse, which we expect to be widely available to nonprofits later this year.

We chose June 16 as World Metrics Day for three reasons. First, last year on June 16th we convened a number of our partners and leading thinkers on the topic at the Rockefeller Foundation to map out our current strategy, and we are simply taking stock of where we have come in the year. Second, it’s Bloomsday, a day notable for its devotees’ commitment (obsession?) to a singular topic, perhaps the least linear novel ever written (Joyce’s Ulysses). Finally, it happens to be my father’s birthday (happy birthday, Dad!). My father is a pathologist and the naming of Pulse was inspired by the autopsy metaphor. Most measurement and evaluation in the social sector is like an autopsy: intrusive, inconclusive and too late to help. We wanted to build a metrics practice that could take the “pulse” of our investments, with simple but meaningful measures that can help us make course corrections real time.

So we will be spending the day thinking about and talking about and, indeed, celebrating metrics. Celebrate with us — June 16 is the first annual World Metrics Day. Remember, count something. and make it count!

Updated 6/18/2009 - thanks to Sara Olsen of SVT Consulting for her great blog post over at Social Edge on World Metrics Day. Count what counts, Sara!

Editor’s note: New contributor Brad Presner is Acumen Fund’s Metrics Manager. In this role, he manages the development of the Pulse social metrics platform and helps define Acumen Fund’s performance assessment strategy. He joined Acumen Fund in July, 2008, having worked closely with the Pulse team while an employee of Google.org. Brad holds a BS degree in Mechanical Engineering from Stanford University.

By Brad Presner

Last October, Brian Trelstad and I were fortunate enough to be invited to the Bill & Melinda Gates Foundation offices in Seattle. As a part of their own processes for engaging in thoughtful and actionable measurement, the Impact Planning and Improvement (IPI) Group brought together a set of leading thinkers to help inform their own thinking and potentially change their approach to measuring social impact at the Foundation.

Acumen Fund’s contribution to this meeting was a discussion of our BACO Methodology., an exercise whereby our Portfolio team compares the estimated social output of a potential investment we are considering with that of the best available charitable option that addresses the same issue. For us, this process helps us ascertain where philanthropic capital will be most effective – whether invested in the social entrepreneurs we seek to provide with patient capital or with an alternative charitable vehicle.

I think it’s safe to say that Brian and I got as much out of this meeting as we contributed. We were humbled to be in the room with peers such as Jed Emerson ofSROI fame and his former colleagues at REDF; Paul Brest of the Hewlett Foundation; Michael Weinstein from Robin Hood; Kat Rosqueta of the Center for High Impact Philanthropy, Sara Olsenof SVT Group, and others from banks, think tanks and consulting firms. The collection of thoughtful, engaged leaders was truly inspiring.

A little more than halfway through the meeting, Paul Brest made a point that all eight of the methodologies being discussed that day could be boiled down to a similar fundamental goal—calculating the expected return as a function of how you value the benefit of your program, weighted by probability of success, as a ratio of the cost of your program:

Expected Return = (Benefit X Probability of Success) / Cost

This keen observation underscores both the inherently simple goal—how much good came out of what we put in—and an abundantly complex set of questions that all eight of us in the room had endeavored to answer for ourselves – namely, how do you accurately estimate the benefit of your work?

I don’t think we came away with any definitive answers. In fact, the only thing we seemingly could agree on was that there was no single “right” answer (and that it may not even be desirable for there to be). But what this convening really highlighted is that this conversation is happening more and more. At conferences, in team meetings at Foundations, in phone calls among peers – we are seeing a convergence in the field around what we at Acumen like to think of as “cost effective cost effectiveness”. While our methods may never be the same and the metrics we track may differ, the conversations about how and why we seek to assess impact are happening with more frequency and greater depth. And while it will likely take us many more years to get there, we are light years ahead of a time not long ago when the conversations weren’t even happening.

We encourage you to take a look at the recently posted “Measuring and/or Estimating Social Value Creation Report” put together by Melinda Tuan and sponsored by the Gates Foundation. We are grateful to have been included in their process and hope that in some small way, we were able to contribute to our shared goal of advancing the field towards more cost effective cost effectiveness.

In addition to holding the “billionaire title,” these three individuals share a commitment to a new movement of social change, according to Matthew Bishop and Michael Green, co-authors of Philanthrocapitalism: How the Rich Can Save the World.  Bishop and Green believe that Gates, Clinton and Jolie are part of a new generation of wealthy, powerful people who are changing the role of philanthropy by combining focused, charitable donations with “big-business-style” investment strategies. Their book profiles several of these new philanthropists, including those listed above and other influentials like George Soros and U2’s Bono.

Bishop, American Business Editor/New York Bureau Chief for The Economist, explained the basis for focusing on the super-rich as change agents in a recent interview with Alliance Magazine: “The point we’re making is that there are more super-rich people around than ever before and they have all sorts of problem-solving talents developed in their business lives that they are now looking to bring to bear on some of the world’s big problems.”

But while these philantrocapitalists may be willing to take greater risks with their checkbooks as compared to traditional philanthropic institutions, many of them have a deep interest in accountability and rigorous performance measurement. Bishop’s and Green’s Values blog recently published an entry on philanthrocapitalism’s need for robust impact measurement systems.  Acumen Fund’s new portfolio management system, called Pulse, was pointed to as one innovative “work-in-progress” solution aimed at addressing this challenge.

Bishop also recently interviewed Acumen Fund CEO Jacqueline Novogratz at the Clinton Global Initiative about Acumen Fund’s emphasis on market-based solutions to solving the problems of global poverty (the video can be viewed below or on The Economist’s website).

In reviewing the list of names and organizations included in Bishop’s and Green’s book, it is clear that philanthrocapitalism has as many distinctions in as it does similarities.  Though the movement is unlikely to settle on a precise spot along the scale of pure philanthropy to pure capitalism, an exciting, unifying theme emerges: there is great promise in connecting high net worth individuals with innovative intermediaries to move from social investment to long-term impact.

We’ve previously mentioned our Portfolio Database Management System (PDMS), the tool we are developing (with the help of many) to aggregate and evaluate data about social enterprises. Yesterday was a big day for PDMS: the social metrics platform was announced yesterday at the Clinton Global Initiative, and articles about it appeared in both BusinessWeek and The New York Times.

Also announced at the Clinton Global Initiative was the launch of the ANDE network, which Brian blogged about here. The PDMS is currently being tested by a number of beta users, and we anticipate launching officially at the beginning of 2009. We’d love to launch with a name better than the bland PDMS acronym — any suggestions?

This post first appeared on Acumen investee VisionSpring’s blog, Business in a Bag. We’ll be cross-posting with Business in a Bag from time to time.

The post’s author is Tim Johnson-Aramaki, a student at the University of Michigan’s Ross School of Business, who spent the summer with VisionSpring India working on a data-collection methodology to measure the long-term impact of VisionSpring’s work on the lives of Vision Entrepreneurs and customers. His project is part of a multi-year impact study conducted by Professor Ted London at Michigan’s William Davidson Institute.

Over the last few months, I’ve been working to develop a survey instrument with the VisionSpring team here in Hyderabad and the William Davidson Institute team in Ann Arbor. The first step was survey pre-testing, which involved conducting countless interviews in rural village throughout the state of Andhra Pradesh. These interviews are meant to help us discover whether the questions we’ve come up with are understood by respondents with varying semantic and cultural backgrounds, and if they prompt valid and appropriate responses. Some of the results we’ve gathered have been really interesting.

For example, one of the most critical pieces of data in measuring VisionSpring’s impact is the income of its Vision Entrepreneurs and customers. It also happens to be one of the most difficult things to measure as there are challenges when it comes to discussing money. Through our interviews, we’ve found that while people are relatively open in assigning a number to their income, that number may not be accurate. There are a variety of reasons for this, but one is that they fear the income figures may be passed on to state or national agencies, potentially jeopardizing the public assistance they receive. To avoid this, they often provide income figures lower than that which they actually earn.

It has been a busy summer here at the Acumen Fund New York office, where we’ve been working hard to implement website and blog changes that make it easier for our community to access data on and insights from our work.

On the website, it’s now possible to view metrics reported by many of our investees. These investment metrics are fed directly from our Portfolio Data Management System. We are not able to share all the available data, and some of our new investments are still in the process of collecting data, but we are committed to sharing as much information as we can, as often as we can – and this is a step in that direction.

Many of us in the base of the pyramid community, myself included, often wonder, is this business really making a social impact? Sure, there are real indicators of success, but what action drove that particular outcome? As I ponder the social impact of business, I’m reminded of an old marketing adage: We know at least 50 percent of our efforts are working “we just don’t know which half. (Hat tip to Brian Trelstad for bringing this up in a meeting.)

In order to build truly inclusive businesses, our sector must start tracking impact over time. This we can probably all agree on. But the challenge is not so much in creating buy-in around the idea of measurement, but in finding a way to integrate an effective and user-friendly system for doing so in an already resource-constrained work environment.

Because of these and other stumbling blocks in creating social metrics, Rob Katz and I were particularly intrigued by the newly released Measuring Impact Framework developed by The World Business Council for Sustainable Development (WBCSD). This framework, which has been in the works for nearly two years, is designed to guide companies from small enterprises to large multinationals through the process of measuring and assessing impact, and making better-informed future decisions within the context of a larger development paradigm.

The WBCSD framework is designed to be applicable across sectors and at varying points throughout a company’s life cycle. The core business activities suggested for analysis — governance & sustainability (including corporate governance and environmental management), assets (infrastructure, products and services), people (jobs, skills and training), and financial flows (procurement and taxes) are flexible, open-ended and non-exclusive.

As someone with a limited background in program and policy evaluation, I can see how this framework could be helpful in conducting an internal impact audit. The WBCSD is effective in outlining and defining the various parts of any good analysis: understanding the relationships between a series of business activities, company resources, direct and indirect outcomes and large-scale impact. Assuming the same individuals conduct the impact analysis over consistent time intervals, organizations can use the Measuring Impact Framework as a tool for identifying goals, achieving internal benchmarks and measuring their own progress over time.

But to gauge an organization’s positive and negative contributions within a broader development context, it will become increasingly important to consider counterfactuals and to establish standard indicators as part of the assessment. By measuring against peer organizations as well as against itself over time an organization can protect itself from its own bias and see itself as part of a greater competitive landscape. Only through this lens will business leaders be able to make integrated and inclusive decisions for change.

It may be a long time before the social business sector (a vague and multi-faceted term in-and-of-itself) can agree on a set of standard metrics; but the work of WBCSD and others such as the Aspen Network for Development Entrepreneurs is encouraging. If nothing else, the Measuring Impact Framework is a tool that businesses can use to improve reporting processes, identify and communicate priorities with stakeholders. Hopefully, it will also serve as a catalyst for shared effort and continued development on sector-wide measurement tools.