metrics

At SoCap09 two weeks ago, I spoke on a moderated panel called “True Tales of Amazement and Horror from the Fundraising Circuit” with Don Shaffer, Shari Berenbach, Penelope Douglas and moderator Stuart Davidson. It was a great group - one I was honored to be a part of - and it was a successful panel in no small part thanks to Stuart’s questions.

One of those questions was directed at me; Stuart asked, “Do metrics sell on the fundraising circuit?” Must have been an ear-perking question, because my answer has been tweeted and re-tweeted a dozen times. Here’s how it came out, at least to the Twitter-verse:

@elizabethu: Brian Trelstad, Acumen Fund: our donors care more that we care about metrics than they care about the metrics themselves #socap09

As other tweeple picked up on this response, I’ve been mulling over the limits of 140-character messages; frankly, some important context got lost in translation. In retrospect, here’s what I was trying to get at, more or less:

I remember my corporate finance professor saying that valuing a public company was a combination of cash and hope. Cash flows from current businesses and hope for the future. Cash plus hope = corporate valuation. Start-ups and social enterprises are a little heavier on the hope than the cash in their early stages, but this is how you measure value in any enterprise. In our fundraising, in the early stages of our business and in the early stages of our field, we have also relied more on hope than cash in that these are early stage businesses with more promise than current performance. And when we go out to fundraise, our metrics are not really the selling point. So at this point, I think that our donors care more that we care about metrics than they care about the metrics themselves. But as competition increases and the field matures, I believe this will change and we need to be ready for some tough questions as a fund and as a field about the impact of our work and the social metrics.

This is also a riff on an article in the Stanford Social Innovation Review from back in 2004. In it, the authors cite research on high net worth individuals in venture philanthropy, concluding that the obsession with performance metrics is actually somewhat overblown, and metrics are not a gating factor for contribution decisions.

Hopefully that clarifies. Thanks for all the tweets, folks - and for caring enough about metrics to send 140-character musings out to the ‘verse. And as much as I can - which is not often - I tweet at @trelstad

Mike McCreless is currently serving as a summer associate on the Portfolio team.  He is a joint-degree student at Harvard Business School and Harvard Kennedy School’s MPA-ID program. He has worked as a consultant for Monitor Company in New York, Madrid, Paris and Morocco, and as a Research Associate for Professor Michael Porter at Harvard Business School, where he wrote case studies and research notes about economic development in Rwanda, foreign aid and other topics.

A couple of years ago, Larry Summers gave a guest lecture at a class I was taking on international development. He delivered a fascinating 20-minute lecture as if reading it from a transcript in his mind’s eye, and then took questions for an hour. I asked him why foreign aid had delivered so much less ‘development,’ by any definition, than expected, and what should we be doing with our time and money instead?

He took a long pause, staring into space, and then said slowly that that was perhaps the most profound question in economics, causing me simultaneously to swell with pride and recoil at my own pretentiousness. He listed several of the standard arguments for aid’s ineffectiveness, but what really stuck with me was his contention that advances in technology, not foreign aid, would do most to eradicate poverty in the long run.

Well, that made sense, but I wasn’t sure what to do about it. It was not as if I had a stockpile of socially valuable technological innovations saved up for just such an occasion. I was a white twenty-something from the American ‘burbs, with no technological expertise and limited experience in Africa, but enough experience with the foreign aid system to give me misgivings about a career with the big multilaterals.

I didn’t have any technological innovations, but I did have ideas about how to make the foreign aid system more effective. Before enrolling in grad school, I spent time in Rwanda, researching economic development and in particular, the efforts of aid agencies on the ground. As I traveled around the country and met its people, I was astonished at the level of deprivation that they endured despite the vast inflows of aid. It appeared that aid had delivered surprisingly little ‘development,’ though of course the counterfactual might have been much worse. Moreover, the academic and popular press was awash with possibilities to improve the management practices of aid agencies. I hoped that better measurement of project outcomes, and alignment of organizational structures and staff incentives with those outcomes, would help donors and recipients alike get more out of the $100 billion spent on aid every year.

I am in a three-year program in which students work in internship programs during the two intervening summers. Last summer, I had the opportunity to test some of these ideas during an internship at a large aid institution. My role was to help design the next version of an internal system to track project outcomes. I worked with project managers to understand how they select projects, manage them over time, and measure their impact. I tried to assess the potential for better management practices at U.S. headquarters to improve results on the ground in Africa.

I also tried to understand the culture of the organization and how I might fit in. I tried to understand managers’ jobs, how they spent their time day-to-day, and more importantly, how they felt about their jobs. What did their work mean to them? Did they feel like they were making a difference? Did they believe that somewhere out there, in some forgotten urban slum or rural village, someone’s life was improved, their burdens eased by the work we did behind our desks in the U.S.?

During these conversations, I was often reminded of a quote from Liar’s Poker, Michael Lewis’ memoir of his time on Wall Street during the 1980s: “Goodness was not taken into account on the trading floor. It was neither rewarded nor punished. It just was. Or it wasn’t.” On a trading floor, this is as it should be, but I did not expect it to be as true of aid institutions in 2008 as it was of Salomon Brothers in 1987.

Many of the managers I spoke to care deeply about development, and think critically and often about whether their work actually helps anyone. Others wish they worked at a purely commercial enterprise, where they would not have to be bothered by concerns about social impact. To some, a job in the foreign aid system simply meant a comfortable life in the U.S. and enhanced status in the socio-political elite of their home country. Most staff simply assume that what they do helps people, without really checking to make sure. Their hearts are in the right place, and they have faith in the system to direct their efforts toward socially valuable ends. In fairness, it is extraordinarily difficult to know whether one’s activities in U.S. headquarters actually help anybody on the ground. I was nevertheless taken aback at how little people cared to find out.

I realized that my ideas about reforming management practices at aid agencies were a technocratic solution to a cultural problem. Managerial fixes such as systems to track results and incentive plans aligned with results help on the margins, but do not directly address the fundamental cultural issues that constrain the effectiveness of foreign aid.

For my second summer internship, I focused on smaller organizations that I believed shared my values, and my interest in experimenting with new modes of foreign assistance. To make a long story short, this June I found myself at Acumen Fund. The fourth day of my internship was “World Metrics Day,” declared as such by Acumen Fund and its partners to celebrate the progress they have made in measuring the social impacts of their investments. As we sat down to a World Metrics Day teleconference, Acumen Fund Chief Investment Officer Brian Trelstad turned to me. “World Metrics Day! What better way to inaugurate your first week!” He stared at me expectantly. I wasn’t used to this level of excitement about measuring social impact. I stared dumbly back, trying to figure out if he was joking or not, if I should laugh or nod. He has a wry sense of humor, but he wasn’t joking. The man really cares to know whether what he does actually helps anybody. It was going to be a different kind of summer.

I have spent half of my time working on a deal to use innovative biomass technology to electrify rural off-grid villages in India, and half of my time thinking about how to integrate the many approaches Acumen takes to measuring the social impacts of its investment. The biomass project has vividly demonstrated the potential for innovation to create tremendous social impact. This social impact may be difficult to measure, but Acumen Fund’s investment in impact measurement also demonstrates that an organization that truly wants to measure its impact can find a way to do so.

In the end, Larry Summers was right. My quest for an organization that was serious about high-efficiency social impact led me to an organization that invests in innovations—innovations not only in technology, but also in distribution and delivery of essential products and services to the poor.

On the surface it may seem like Acumen Fund, and venture philanthropy in general, seems to just be the hot new thing these days. They blog, and tweet, and use fancy words like ‘philanthro-capitalism.’ They are undoubtedly in fashion. That’s great, because it gets new people excited about the work they do. But look past all that. Acumen Fund could fall out of fashion tomorrow—no more headlines, no more keynote speeches—and what would be left is a handful of people who actually care whether what they do is valuable. Not to the blogosphere and the editors of The Economist, but to poor people in India, Pakistan, Kenya, Uganda and Tanzania. That is my experience of Acumen Fund this summer: a small group of people who are in it for the right reasons, a handful of villages in India that have access to clean electricity for the first time (and the prospect of bringing electricity to many more), and all for much less than it would have cost my former employer to accomplish the same. The fashion may fade, but the impact will remain.

World Watches in Shock and Awe

Mark your calendars: June 16 (tomorrow) is World Metrics Day! Our declaration of an official World Metrics Day (or WMD, as we’ve been calling it) is a little (well, actually, a lot) tongue in cheek, but Acumen Fund’s commitment to metrics is absolutely serious. Figuring out how to measure social impact has long been a challenge for philanthropists, social investors, practitioners and the development sector. Our focus has been on finding ways to collect, understand and use data to (1) best support the enterprises in which we invest, (2) improve how Acumen Fund works and (3) create common measures for the sector - in that order of importance. (We’ve blogged here before about metrics, on what we like to call cost effective cost effectiveness and our best available charitable option methodology.)

If you follow our work, you know that we’ve been developing a platform called Pulse, discussed in BusinessWeek and The New York Times when it was announced last fall at the Clinton Global Initiative. Pulse helps social investors track, measure, and evaluate operational data to understand whether different approaches to solving the problems of global poverty are working. Although originally designed as a tool to help us manage our own portfolio, we’ve been working with Salesforce.com to make it available for adoption by peer organizations and to share data across the sector. We have also been collaborating with the Rockefeller Foundation, B-Lab, Deloitte and PwC to develop standard definitions of social and environmental performance that can enable comparability across portfolios (for more on this, see www.iris-standards.org). About forty organizations are currently testing Pulse, which we expect to be widely available to nonprofits later this year.

We chose June 16 as World Metrics Day for three reasons. First, last year on June 16th we convened a number of our partners and leading thinkers on the topic at the Rockefeller Foundation to map out our current strategy, and we are simply taking stock of where we have come in the year. Second, it’s Bloomsday, a day notable for its devotees’ commitment (obsession?) to a singular topic, perhaps the least linear novel ever written (Joyce’s Ulysses). Finally, it happens to be my father’s birthday (happy birthday, Dad!). My father is a pathologist and the naming of Pulse was inspired by the autopsy metaphor. Most measurement and evaluation in the social sector is like an autopsy: intrusive, inconclusive and too late to help. We wanted to build a metrics practice that could take the “pulse” of our investments, with simple but meaningful measures that can help us make course corrections real time.

So we will be spending the day thinking about and talking about and, indeed, celebrating metrics. Celebrate with us — June 16 is the first annual World Metrics Day. Remember, count something. and make it count!

Updated 6/18/2009 - thanks to Sara Olsen of SVT Consulting for her great blog post over at Social Edge on World Metrics Day. Count what counts, Sara!

Editor’s note: New contributor Brad Presner is Acumen Fund’s Metrics Manager. In this role, he manages the development of the Pulse social metrics platform and helps define Acumen Fund’s performance assessment strategy. He joined Acumen Fund in July, 2008, having worked closely with the Pulse team while an employee of Google.org. Brad holds a BS degree in Mechanical Engineering from Stanford University.

By Brad Presner

Last October, Brian Trelstad and I were fortunate enough to be invited to the Bill & Melinda Gates Foundation offices in Seattle. As a part of their own processes for engaging in thoughtful and actionable measurement, the Impact Planning and Improvement (IPI) Group brought together a set of leading thinkers to help inform their own thinking and potentially change their approach to measuring social impact at the Foundation.

Acumen Fund’s contribution to this meeting was a discussion of our BACO Methodology., an exercise whereby our Portfolio team compares the estimated social output of a potential investment we are considering with that of the best available charitable option that addresses the same issue. For us, this process helps us ascertain where philanthropic capital will be most effective – whether invested in the social entrepreneurs we seek to provide with patient capital or with an alternative charitable vehicle.

I think it’s safe to say that Brian and I got as much out of this meeting as we contributed. We were humbled to be in the room with peers such as Jed Emerson ofSROI fame and his former colleagues at REDF; Paul Brest of the Hewlett Foundation; Michael Weinstein from Robin Hood; Kat Rosqueta of the Center for High Impact Philanthropy, Sara Olsenof SVT Group, and others from banks, think tanks and consulting firms. The collection of thoughtful, engaged leaders was truly inspiring.

A little more than halfway through the meeting, Paul Brest made a point that all eight of the methodologies being discussed that day could be boiled down to a similar fundamental goal—calculating the expected return as a function of how you value the benefit of your program, weighted by probability of success, as a ratio of the cost of your program:

Expected Return = (Benefit X Probability of Success) / Cost

This keen observation underscores both the inherently simple goal—how much good came out of what we put in—and an abundantly complex set of questions that all eight of us in the room had endeavored to answer for ourselves – namely, how do you accurately estimate the benefit of your work?

I don’t think we came away with any definitive answers. In fact, the only thing we seemingly could agree on was that there was no single “right” answer (and that it may not even be desirable for there to be). But what this convening really highlighted is that this conversation is happening more and more. At conferences, in team meetings at Foundations, in phone calls among peers – we are seeing a convergence in the field around what we at Acumen like to think of as “cost effective cost effectiveness”. While our methods may never be the same and the metrics we track may differ, the conversations about how and why we seek to assess impact are happening with more frequency and greater depth. And while it will likely take us many more years to get there, we are light years ahead of a time not long ago when the conversations weren’t even happening.

We encourage you to take a look at the recently posted “Measuring and/or Estimating Social Value Creation Report” put together by Melinda Tuan and sponsored by the Gates Foundation. We are grateful to have been included in their process and hope that in some small way, we were able to contribute to our shared goal of advancing the field towards more cost effective cost effectiveness.

In addition to holding the “billionaire title,” these three individuals share a commitment to a new movement of social change, according to Matthew Bishop and Michael Green, co-authors of Philanthrocapitalism: How the Rich Can Save the World.  Bishop and Green believe that Gates, Clinton and Jolie are part of a new generation of wealthy, powerful people who are changing the role of philanthropy by combining focused, charitable donations with “big-business-style” investment strategies. Their book profiles several of these new philanthropists, including those listed above and other influentials like George Soros and U2’s Bono.

Bishop, American Business Editor/New York Bureau Chief for The Economist, explained the basis for focusing on the super-rich as change agents in a recent interview with Alliance Magazine: “The point we’re making is that there are more super-rich people around than ever before and they have all sorts of problem-solving talents developed in their business lives that they are now looking to bring to bear on some of the world’s big problems.”

But while these philantrocapitalists may be willing to take greater risks with their checkbooks as compared to traditional philanthropic institutions, many of them have a deep interest in accountability and rigorous performance measurement. Bishop’s and Green’s Values blog recently published an entry on philanthrocapitalism’s need for robust impact measurement systems.  Acumen Fund’s new portfolio management system, called Pulse, was pointed to as one innovative “work-in-progress” solution aimed at addressing this challenge.

Bishop also recently interviewed Acumen Fund CEO Jacqueline Novogratz at the Clinton Global Initiative about Acumen Fund’s emphasis on market-based solutions to solving the problems of global poverty (the video can be viewed below or on The Economist’s website).

In reviewing the list of names and organizations included in Bishop’s and Green’s book, it is clear that philanthrocapitalism has as many distinctions in as it does similarities.  Though the movement is unlikely to settle on a precise spot along the scale of pure philanthropy to pure capitalism, an exciting, unifying theme emerges: there is great promise in connecting high net worth individuals with innovative intermediaries to move from social investment to long-term impact.

We’ve previously mentioned our Portfolio Database Management System (PDMS), the tool we are developing (with the help of many) to aggregate and evaluate data about social enterprises. Yesterday was a big day for PDMS: the social metrics platform was announced yesterday at the Clinton Global Initiative, and articles about it appeared in both BusinessWeek and The New York Times.

Also announced at the Clinton Global Initiative was the launch of the ANDE network, which Brian blogged about here. The PDMS is currently being tested by a number of beta users, and we anticipate launching officially at the beginning of 2009. We’d love to launch with a name better than the bland PDMS acronym — any suggestions?

This post first appeared on Acumen investee VisionSpring’s blog, Business in a Bag. We’ll be cross-posting with Business in a Bag from time to time.

The post’s author is Tim Johnson-Aramaki, a student at the University of Michigan’s Ross School of Business, who spent the summer with VisionSpring India working on a data-collection methodology to measure the long-term impact of VisionSpring’s work on the lives of Vision Entrepreneurs and customers. His project is part of a multi-year impact study conducted by Professor Ted London at Michigan’s William Davidson Institute.

Over the last few months, I’ve been working to develop a survey instrument with the VisionSpring team here in Hyderabad and the William Davidson Institute team in Ann Arbor. The first step was survey pre-testing, which involved conducting countless interviews in rural village throughout the state of Andhra Pradesh. These interviews are meant to help us discover whether the questions we’ve come up with are understood by respondents with varying semantic and cultural backgrounds, and if they prompt valid and appropriate responses. Some of the results we’ve gathered have been really interesting.

For example, one of the most critical pieces of data in measuring VisionSpring’s impact is the income of its Vision Entrepreneurs and customers. It also happens to be one of the most difficult things to measure as there are challenges when it comes to discussing money. Through our interviews, we’ve found that while people are relatively open in assigning a number to their income, that number may not be accurate. There are a variety of reasons for this, but one is that they fear the income figures may be passed on to state or national agencies, potentially jeopardizing the public assistance they receive. To avoid this, they often provide income figures lower than that which they actually earn.

It has been a busy summer here at the Acumen Fund New York office, where we’ve been working hard to implement website and blog changes that make it easier for our community to access data on and insights from our work.

On the website, it’s now possible to view metrics reported by many of our investees. These investment metrics are fed directly from our Portfolio Data Management System. We are not able to share all the available data, and some of our new investments are still in the process of collecting data, but we are committed to sharing as much information as we can, as often as we can – and this is a step in that direction.

Many of us in the base of the pyramid community, myself included, often wonder, is this business really making a social impact? Sure, there are real indicators of success, but what action drove that particular outcome? As I ponder the social impact of business, I’m reminded of an old marketing adage: We know at least 50 percent of our efforts are working “we just don’t know which half. (Hat tip to Brian Trelstad for bringing this up in a meeting.)

In order to build truly inclusive businesses, our sector must start tracking impact over time. This we can probably all agree on. But the challenge is not so much in creating buy-in around the idea of measurement, but in finding a way to integrate an effective and user-friendly system for doing so in an already resource-constrained work environment.

Because of these and other stumbling blocks in creating social metrics, Rob Katz and I were particularly intrigued by the newly released Measuring Impact Framework developed by The World Business Council for Sustainable Development (WBCSD). This framework, which has been in the works for nearly two years, is designed to guide companies from small enterprises to large multinationals through the process of measuring and assessing impact, and making better-informed future decisions within the context of a larger development paradigm.

The WBCSD framework is designed to be applicable across sectors and at varying points throughout a company’s life cycle. The core business activities suggested for analysis — governance & sustainability (including corporate governance and environmental management), assets (infrastructure, products and services), people (jobs, skills and training), and financial flows (procurement and taxes) are flexible, open-ended and non-exclusive.

As someone with a limited background in program and policy evaluation, I can see how this framework could be helpful in conducting an internal impact audit. The WBCSD is effective in outlining and defining the various parts of any good analysis: understanding the relationships between a series of business activities, company resources, direct and indirect outcomes and large-scale impact. Assuming the same individuals conduct the impact analysis over consistent time intervals, organizations can use the Measuring Impact Framework as a tool for identifying goals, achieving internal benchmarks and measuring their own progress over time.

But to gauge an organization’s positive and negative contributions within a broader development context, it will become increasingly important to consider counterfactuals and to establish standard indicators as part of the assessment. By measuring against peer organizations as well as against itself over time an organization can protect itself from its own bias and see itself as part of a greater competitive landscape. Only through this lens will business leaders be able to make integrated and inclusive decisions for change.

It may be a long time before the social business sector (a vague and multi-faceted term in-and-of-itself) can agree on a set of standard metrics; but the work of WBCSD and others such as the Aspen Network for Development Entrepreneurs is encouraging. If nothing else, the Measuring Impact Framework is a tool that businesses can use to improve reporting processes, identify and communicate priorities with stakeholders. Hopefully, it will also serve as a catalyst for shared effort and continued development on sector-wide measurement tools.