Methodology
Grading the States 2008 Methodology
Grading the States 2008 builds on a rich supply of information about state government. To make its evaluations, Pew staff collected data from more than 12,000 different sources—including surveys, written documents and interviews—and comprehensively analyzed them to paint a complete portrait of government performance in each of the 50 states.
The grading process for Grading the States 2008 report included the following steps:
- Grading against criteria.
Pew graded the states against criteria, not against each other. Pew analysts began by carefully identifying four management areas—Information, Infrastructure, Money and People—that are closely tied to achieving policy goals. In each management area, Pew staff then identified the characteristics of effectively managed governments. These criteria, defined by the best research in the field, established the grading standards.
- Refining the criteria.
Pew staff then identified the subcriteria that comprise each criterion. For example, a state that manages its money well would maintain structural balance through matching current revenues and expenditures; a revenue structure that accommodates economic fluctuations; contingency devices that help address economic downturns; and limited use of one-time revenues.
- Collecting the most important information on the criteria and subcriteria.
Pew staff then collected the data that provided the best information about the criteria and subcriteria. Researchers assembled much of the data from existing sources, including information posted on state Web sites and published in government reports. Some of the information came from a Web-based survey, in which state officials provided information requested by Pew staff. And some of the information came from interviews conducted by the team of reporters from Governing magazine and the Pew Center on the States.
- Drilling down to agency level.
In addition to examining overall state government, Pew staff conducted additional research on two specific state agencies—corrections and the state agency with responsibility for child protective services—both as indicators of a state’s performance at the agency level, and as critical functions of state government in their own right. This research was factored into the criteria in each of the four management areas.
- Analyzing the information through a collaborative process.
The research team of scholars and journalists then jointly analyzed and discussed the data and its implications. They combined their information and together agreed on the grades. Continuing a trend begun with the 2005 report, their evaluations focused almost exclusively on states’ abilities to demonstrate tangible results. Plans, proposals, and data (while all important) were given less weight than examples that improved program or overall state performance.
- Conducting the process in transparent fashion.
From the very beginning, Pew committed to a process of transparency. Pew published the criteria and subcriteria before launching data collection. The grades published in Governing include extensive explanation of the grades and of the information used to produce them. In addition to the narratives published in the magazine, the Pew Center on the States' Web site contains deeper explanation.
This is the fourth time Pew has graded the states. Fulfilling a promise made in 2005, the 2008 grades are comparable with the 2005 grades, with one important caveat: While the criteria used in 2008's analysis are essentially the same as the 2005 report card, the state of the art in government performance advanced significantly. Innovations in best practices pushed the bar higher in a number of areas—most obviously in the use of information and information technology in decision-making and program management, but also in other areas such as strategic human capital management. Thus, a state that held steady from 2005 to 2008 in a management area could have seen its grade in that area decline.
The grades assess the capacity of state governments as a whole to produce results—including the cumulative skills and leadership of elected and appointed officials, career civil servants and not-for-profit and private-sector partners. The grades do not represent a judgment of any individual, branch or department within state government. States vary widely in their political, economic, policy and labor-management cultures. Accordingly, many different considerations factor into these assessments, including the nature of state constitutional and legal processes, the structure of state policies and programs, the relationships among elements of the state government and the crucial connections between state government and the public.
