Methodology
Grading the States 2008 Methodology
Grading the States 2008 builds on a rich supply of information about state government. To make its evaluations, the Government Performance Project (GPP) collected data from over 12,000 different sources—including surveys, written documents and interviews—and comprehensively analyzed them to paint a complete portrait of government performance in each of the 50 states.
The grading process for Grading the States 2008 report included the following steps:
- Grading against criteria.
The GPP graded the states against criteria, not against each other. The team’s analysts began by carefully identifying four management areas – Information, Infrastructure, Money, and People – that are closely tied to achieving policy goals. In each management area, they then identified the characteristics of effectively managed governments. These criteria, defined by the best research in the field, established the grading standards.
- Refining the criteria.
The GPP research team then identified the subcriteria that comprise each criterion. For example, a state that manages its money well would maintain structural balance through matching current revenues and expenditures; a revenue structure that accommodates economic fluctuations; contingency devices that help address economic downturns; and limited use of one-time revenues.
- Collecting the most important information on the criteria and subcriteria.
The GPP research team then collected the data that provided the best information about the criteria and subcriteria. Researchers assembled much of the data from existing sources, including information posted on state Web sites and published in government reports. Some of the information came from a Web-based survey, in which state officials provided information requested by the GPP. And some of the information came from interviews conducted by the team of reporters from Governing magazine and the Pew Center on the States.
- Drilling down to agency level.
In addition to examining overall state government, we conducted additional research on two specific state agencies—corrections and the state agency with responsibility for child protective services—both as indicators of a state’s performance at the agency level, and as critical functions of state government in their own right. This research was factored into the criteria in each of the four management areas.
- Analyzing the information through a collaborative process.
The research team of scholars and journalists then jointly analyzed and discussed the data and its implications. They combined their information and together agreed on the grades. Continuing a trend begun with the 2005 report, their evaluations focused almost exclusively on states’ abilities to demonstrate tangible results. Plans, proposals, and data (while all important) were given less weight than examples that improved program or overall state performance.
- Conducting the process in transparent fashion.
From the very beginning, the GPP committed to a process of transparency. The GPP published the criteria and subcriteria before launching data collection. The grades published in Governing include extensive explanation of the grades and of the information used to produce them. In addition to the narratives published in the magazine, the GPP Web site contains deeper explanation. In the coming months, more data and analysis from the project will be made available on the Web site. The result will be an unprecedented repository of information about state government management.
This is the fourth time the Government Performance Project has graded the states. Fulfilling a promise made in 2005, this year’s grades are comparable with the grades issued then, with one important caveat: While the criteria used in this year’s analysis are essentially the same as in the last report card, the state of the art in government performance has advanced significantly over the past three years. Innovations in best practices have pushed the bar higher in a number of areas—most obviously in the use of information and information technology in decision-making and program management, but also in other areas such as strategic human capital management. Thus, a state that has held steady in a management area could see its grade in that area decline.
The grades assess the capacity of state governments as a whole to produce results—including the cumulative skills and leadership of elected and appointed officials, career civil servants, and not-for-profit and private-sector partners. The grades do not represent a judgment of any individual, branch, or department within state government. States vary widely in their political, economic, policy, and labor-management cultures. Accordingly, many different considerations factor into these assessments, including the nature of state constitutional and legal processes, the structure of state policies and programs, the relationships among elements of the state government, and the crucial connections between state government and the public.
