What does all this mean to the citizens?
By Thomas I Miller, Ph.D., President, National Research Center, Inc.
It’s not so much that you can’t always get what you want. It’s more often that your residents may not care about you’ve got even when they see it. The business of measuring performance is fraught with doubt. Many of the most dedicated performance measurers suffer bouts of insecurity, fearing that their jurisdiction can rise to the best in every category of performance, but their customers, the residents of their communities, just won’t notice. Or then there’s the problem of measuring the mounds of “stuff” that local government does, only to be characterized as “bean counters,” as if the mounds amount to nothing more than a hill of beans. Such characterizations suggest a smug superiority when cast by other members of the staff or public or a self‐loathing when uttered in candid conversations among performance measurers themselves.
Don’t despair. Every social scientist or human service professional suffers uncertainty about her effectiveness. Ask mediators or therapists if their work helps people lead happier lives. What economist, these days, is confident that the dismal science can predict the cost of living and what political scientist can tell you how to end a civil war? Do performance measurers improve the quality of community life? Answers to the efficacy of this “touchy‐feely” stuff are tough. Brain surgery, by comparison, is cake.
Ignore those epithets claiming that what we count is just beans. Much of what is counted in performance measurement is the meat of a story that can be told about a jurisdiction. Good management requires not just efficiency, but the knowledge that you are being efficient. This knowledge requires honest and consistent assessment and everyone in the performance measurement business understands that this kind of measurement demands a precision that would swell the heart of a watchmaker. But after all the time has been expended in the service of counting programs, costs, speeds, personnel and ratios relevant to each of these, the story uncovered seems to be missing an ending.
The best ending we have to the account told by performance measures comes from the narrative offered by residents. This narrative reflects the results of resident opinion gathered from scientific surveys conducted periodically. These surveys tell you if all your efforts in removing trash, speeding to crime scenes or circulating more library books has been noticed by the very people for whom you’ve made a cleaner, safer, smarter place to live. Survey results emphasize the reality that it’s no longer how hard we try. It’s how well we do. And how well we do is in the mind of the people for whom we do it. It’s true that what residents think doesn’t always square with what other performance data show. Outcomes and outputs don’t always correlate and, in a way, each is a different window into the same room. Each offers a different view of performance. But shouldn’t we be seeing the same dirty socks on the chair even if we look from the front and the side window? Sometimes performance outputs show a pristine room with polished floors and survey results describe a dorm room after homecoming. For example, how can it happen that City A, with one of the highest percent of cleared Part I crimes, also has among the lowest percent of residents who feel safe in their neighborhood at night? Or City Z has the most Parks and Rec FTEs per 1000 population but only middling ratings of overall park and recreation quality?
Research suggests a couple of answers. First, while there is indication that “objective” measures don’t always correlate with “subjective” measures, there is growing evidence that residents can distinguish genuine quality from its absence. In New York City there has been shown a strong relationship between what residents think about their street cleanliness and ratings of cleanliness by trained citizen observers (1). In a study of perceptions about water quality in two small cities in southern India, it was shown that resident opinion about their water, flowing untreated to a standpipe in their street every few days for only a few hours each day, is worse than the opinions of residents in every one of hundreds of places in the U.S. Residents give much lower ratings to snow removal after a catastrophic snow storm. Residents rate dirt streets much lower than paved streets (2).
These findings may seem trivial, but they do point to the fact that residents can distinguish junk from treasure. When the relationship between what we do and what residents recognize is not strong, one must ask if what we are measuring as outputs (what we do) are the things that matter when residents make a judgment about service quality. For example, litter on streets correlates with ratings of street cleanliness, but if residents think a street is clean when no rats are under foot, then the count of paper wrappers wouldn’t correlate with resident ratings of street cleanliness.
Let’s be sure to look at what residents think about the quality of work provided (an outcome) and when the correlations aren’t strong between outputs and outcomes, let’s think first if we are measuring the performance (outputs) that people care about. For example, in focus groups conducted in New York City, it was found that ratings of police were built, in part, on the observed number of sworn officers of Color (3). Make that a performance measure (at least in New York) and perceptions of police could link more closely to output tracking. Those who assert that performance measures should be derived from conversations with the public are part of the Citizen‐Informed Performance Assessment group (4) supported in large part by The Sloan Foundation; or they could just be strategic planners. For sake of self esteem or sheer sanity, it’s worth keeping one eye on those folks and another on the opinions of your own residents. (Keep your third eye on traditional measures of performance.)
References
(1) Gregg G. Van Ryzin, Stephen Immerwahr and Stan Altman. “Measuring Street Cleanliness: A
Comparison of New York City’s Scorecard and Results from a Citizen Survey.” Public Administration
Review. October 2006.
(2) Unpublished white papers by National Research Center, Inc. staff.
(3) Berman, Cohn, Barbara J. Listening to the Public. Fund for the City of New York. 2005.
(4) See for example, School of Public and Environmental Affairs Indiana University. Center for Urban
Policy and the Environment (04‐C20). A Quick Guide to Citizen‐Initiated Performance Assessment for Local Governments. 2004
Note: This article also appears in ICMA Center for Performance Measurement™ FY 2008 Parks and Recreation, page 339. Complete dataset, comments, and chapter PDFs are available only to CPM participants at http://icma.org/cpm.