PROGRAM EVALUATION AND POLICY ANALYSIS

May 21, 2009

    Every year, about $200 million is allocated for program evaluation or policy analysis of about $200 billion worth of social programs in the U.S.  This lecture provides the basics needed to understand and hopefully prepare such research proposals.  Program evaluation is defined as the use of research to measure the effects of a program in terms of its goals, outcomes, or criteria. Policy analysis is defined as the use of any evaluative research to improve or legitimate the practical implications of a policy-oriented program. Program evaluation is done when the policy is fixed or unchangeable. Policy analysis is done when there’s still a chance that the policy can be revised. A number of other terms are probably deserving of definition:

·         action research — just another name for program evaluation of a highly practical nature

·         applied research — a broad term meaning something practical is linked to something theoretical

·         continuous monitoring or audit — determining accountability in activities related to inputs

·         cost-benefit analysis — determining accountability in outputs related to inputs

·         evaluability assessment — determining if a program or policy is capable of being researched

·         feasibility assessment — determining if a program or policy can be formulated or implemented

·         impact analysis or evaluation — finding statistically significant relationships using a systems model

·         needs assessment — finding service delivery gaps or unmet needs to re-establish priorities

·         operations research — another name for program evaluation using a systems model

·         organizational development — research carried out to create change agents in the organization

·         process evaluation — finding statistically significant relationships between activities and inputs

·         quality assurance — ensuring standards for data collection appropriate for program objectives

·         quality circles — creating employee teams that conduct research on organizational quality problems

·         quality control — ensuring data utilized for decision making is appropriate

·         total quality management — evaluation of core outputs from a consumer-driven point of view

    Patton (1990) and many other good books, websites, and seminars on evaluation or grant and proposal writing will contain additional definitions. Don’t take any of my definitions (above) as gospel since there’s not much agreement among professionals on terminology. The basic principles and practices of evaluation are well-established, however. 

    In criminal justice, unless you’re willing to shop for sponsors in the private sector among charitable foundations, the source of most research funding is the National Institute of Justice. NIJ is the research branch of the U.S. Department of Justice. It is totally unlike the old LEAA/LEEP (now NCJRS) in that discretionary block grants (enabling topics of your own choosing) are a thing of the past. Since 1984 and every year since, NIJ comes out with their Research Plan for the upcoming year. It usually sets 7-8 areas they want evaluative research done in. These priorities represent a compromise between the personal preferences of the Attorney General and what NIJ senior staffers believe to be the most critical areas. There’s almost always something available for research with drugs, domestic violence, victims, community crime prevention, sentencing, and in recent years, technology. Not much NIJ-sponsored research supports criminological research, and what little there is is eaten up by think tanks or well-organized academic research teams. NIJ seems to favor proposals that involve randomized experimental designs and a clear indication that both academics and practitioners will be working closely together. Juvenile grants are handled by another agency, OJJDP (Office of Juvenile Justice and Delinquency Prevention), and most of the priorities there tend to heavily involve collaboration with lawyers or law-related education. 

SYSTEMS MODELS

    To understand evaluative research, one must understand the systems model approach. Other words for this approach are theory-driven or criterion-driven. The basic idea is that any program, organization, or functional unit of society can be represented as an interrelated series of parts that work together in cybernetic or organic fashion. In fact, computer language is often used to describe the things an evaluator looks for, such as:

·         Inputs — any rules, regulations, directives, manuals, or operating procedures, including costs, budgets, equipment purchases, number of authorized and allocated personnel

·         Activities — anything that is done with the inputs (resources) such as number of cases processed, kinds of services provided, staffing patterns, use of human and capital resources 

·         Events — things that happen shortly before, after, or during the evaluation period such as Supreme Court decisions, changes in legislation, economic/political changes, natural disasters, etc.

·         Results — specific consequences of activities or products produced, such as number of cases closed or cleared, productive work completed, or completed services provided

·         Outcomes — general consequences or accomplishments in terms of goals related to social betterment such as crime rate declines, fear of crime reductions, deterrence, rehabilitation, etc.

·         Feedback — any recycling or loop of information back into operations or inputs, such as consumer satisfaction surveys, expansion or restriction due to demand, profitability, etc.     

    In addition, evaluation research requires a fairly good grasp of administrative or management models, particularly those that have to do with planning, organizing, and control functions. At a minimum, the researcher should be familiar with the following generic model of all organizations:

MISSION
|
GOALS
|
OBJECTIVES
|
BEHAVIOR

    The mission is, of course, easily obtainable in the form of a mission statement or some declaration of strategic purpose. It’s the one thing that the organization holds up to the outside world, or external environment, as it’s connection to goals of social betterment. Goals are those general purposes of the functional divisions of the organization. For example, to improve traffic flow, to deter crime, to solve cases, and to return stolen property are broadly-stated goals. They’re the things that clientele or consumers of the agency relate to. Objectives are specific, measurable outcomes related to goals, such as a 30% improvement in traffic flow, a 25% reduction in crime, or $500,000 in returned property. They are also usually time-specific, and are what employees most relate to. Behavior is the ordinary productivity of employees. Accountability of behavior to objectives is the personnel function, and of behavior to goals or mission is oversight. Feedback loops normally exist between behavior and goals in most communication channels, however. Evaluators usually identify forces in the external environment (events), like demography, technology, economics, and politics, as well as who the clientele are of the organization (it’s not the same as consumers), as well as the degree to which the organization is extracting power resources from its community vis-a-vis some competitive organization.

THE STEPS OF PROGRAM EVALUATION

    Program evaluation uses less of what I’ve mentioned above than policy analysis. Like grants, program evaluations are often expected to involve some original research design, and sometimes what is called a triangulated strategy, which means 2-3 research designs in one (for example, a quantitative study, some qualitative interviews, and a secondary analysis of data collected from a previous program evaluation). The basis steps are the same as the scientific inference process: (1) hypothesize; (2) sample, (3) design, and (4) interpret. 

    The hypothesis step is crucial. The evaluator must dream up hypotheses that not only make sense for the kind of practical, agency-level data to be collected, but ones that are theoretically significant or related to issues in the field as evidenced from a review of the extant literature. Evaluators draw upon a wide range of disciplines, from anthropology to zoology. In Justice Studies, it’s common to draw hypotheses having to do with offenses from the field of criminal justice and hypotheses having to do with offenders from the field of criminology. 

    Sampling is often not random since few organizations are willing to undergo experiments or deny treatments to some of their customers for the sake of research. Quasi-experiments or time-series may be the best that the evaluator can do. Sometimes, ANCOVA, or post-hoc statistical controls can be used as substitutes for experiments. 

    The design step usually involves replication of instruments, indexes, or scales used by previous researchers in studies of similar organizations. Occasionally, the evaluator will develop, pretest, and validate their own scale. 

    Interpretation results in the production of at least three documents: a lengthy evaluation report geared for a professional audience which contains the statistical methodology; a shorter report geared for laypeople which simplifies or summarizes the study; and an executive summary which is usually two or three pages in length and can also serve as a press release. In addition, evaluators often do interim and progress reports.

THE STEPS OF POLICY ANALYSIS

    Policy analysis generally takes more time than program evaluation. Whereas all the above can be accomplished by a seasoned evaluator in six months or less, policy analysis may take a year or more. It may even take that long just to find out exactly when, how, and where the policy is articulated. A key operating assumption among policy analysts (at the beginning of their work) is that policy is never directly measurable. If it were a simple matter of looking up the statutory authority for the agency, there would be no reason for policy analysis, although it’s common for policy analysts to at least pay homage to “policies enacted in law.”  Some policy changes are incremental but others are non-incremental (called paradigm shifts).  It’s also customary for policy analysis to have a theory of some kind, whether using any pre-existing theory like the models of public choice, welfare economics, corporatism, pluralism, neo-institutionalism, or statism (Howlett & Ramesh 2003), coming up with one’s own theory; e.g., through inductive versus deductive theory construction, and/or a taxonomy of typical policy styles with respect to specific areas of government activity by individual, group, and institutional level of analysis.  

    Policy analysis is most useful when you’ve got a large, troubled organization manifesting a Roshomon Effect (employees disagree or don’t know the policies) and decoupled subenvironments (parts of the organization going off some direction on their own). There’s also usually some public concern about the morality or “goodness” of the organization, and policy analysis is not concerned with disinterested description, but with recommending something on the basis of moral argument. It’s unfortunate that policy studies and policy analysis are typically limited to graduate school education. Learning the art and craft of policy analysis can be of enormous benefit to undergraduates, and the sooner they learn it the better since far too many people wait until graduate school. The basic steps of systematic policy analysis are as follows:

·         Problem Identification — finding the public interests and issues involved

·         Criteria Selection — determining what criteria to evaluate the organization on

·         System Assessment — analysis of boundaries, feedback, and power dynamics

·         Strategies and Tactics — examining decision-making and delivery mechanisms

·         Feasibility Assessment — formulation and implementation analysis     

    A problem is defined as some predicted condition that will be unsatisfactory without intervention. They can stem from problems that everyone is already aware of, but in most cases, are derived the policy analyst’s perception of what’s appropriate for the organization. This is sometimes called a constrained maximization approach because it involves reflection on the broader societal purpose (public interest) of the organization as well as what the top decision-makers are cognizant of. The analyst often derives a perception of the public interest from a literature review, or engages in what is called agenda building, which is similar to how sociologists approach a social problem or issue by analyzing the criminalization or medicalization of something. Intended and unintended functions of the organization as well as formal and informal agendas are looked at.

    The selection of criteria depends on the public interests being served. It’s time for some overdue examples, so let’s take a typical code of ethics for an organization. Almost all of the verbiage in these can be reduced to the values of liberty, equality, and fraternity. The analyst would therefore try to construct formulas to measure liberty, equality, and fraternity in terms of the inputs, activities, and outcomes we became familiar with in our earlier discussion of program evaluation. I’m not kidding; actual mathematical formulas are constructed to measure abstract concepts. Much of the science behind it comes from the way economists measure opportunity cost. It makes more than a few people quantiphobic, so I’ll explain a select few of the simplest ones:

    A system assessment in policy analysis usually involves creating a flowchart showing how outcomes and feedback are produced by the organization, and then showing this flowchart to employees in the agency to see if they agree or disagree with the depicted process. At this time, the analyst is concerned with the problem-outcome connections, and relevant discrepancies or assertions by employees in this review are converted into testable hypotheses that involve estimates of Type I or Type II error. This refers to whether or not there is an informal employee “culture” that accepts or rejects the policies inferred by the analyst. One of the assumptions of policy analysis is that practices do not make policy, but sometimes it seems like they do, so the analyst has to look into the goodness of fit, if you will, with the intentions of those practices and if the discrepant employees are what are called knowledgeables, or those people who are capable of thinking one step up to the abstract principles or interests behind policy. It may well be that the organization is fulfilling broader interests like health, well-being, or quality of life.

    Systems, of course, also intimately involve boundary mechanisms, both externally to the environment and internally. It’s the analyst’s job to determine if these boundaries are open or closed, as this matters for the adaptive capacity of the organization and its readiness to accept change. The general rule is that permeability is healthy. Open organizations are more adaptive and changeable. Feedback loops are also open or closed, depending on the amount and use of information from outside consumers or clientele or internally from employees only. Many criminal justice agencies have criminal consumers, so it’s customary to see terms like semi-permeable or semi-closed.  Another term which is somewhat murky is the concept of “stakeholder.”  There are many types of stakeholders and many different ways to do stakeholder analysis (Friedman & Miles 2006), but generally, any person or organization who is (or can be) positively or negatively impacted by a project is a stakeholder.

    Strategies and tactics involve measuring the delivery mechanisms of the organization. In this regard, the analyst is concerned with informal and formal power structures. The powerless informal people will have their satisficing (settling for less) behavior measured. The powerful decision makers will have the degree to which they say (and they always do) they can improve with more time, money, and people measured. It’s the job of the analyst to determine what are called the optimization levels for resource allocation.  Many good books have been written on Pareto optima, Markov chains, queuing theory, PERT, and cost-benefit analysis, and I won’t try to explain them here, but they all essentially involve running a series of simultaneous equations (or difference equations) through a computer in order to determine probabilistic permutations or combinations that represent various break-even points.    

    Feasibility assessment involves timeline and matrix analysis. A timeline is a graphical way of depicting events, and a formulation and implementation matrix depict the political action plans, both historically and in terms of the analyst’s recommendations. A legend for the symbols goes like this: O = origin of some policy initiative; I = implementation of some action plan; P = program operation started; E = evaluation of program; T = termination of program. 

Best Practices for Curbing Corruption in Asia

Curbing Corruption in Asia is a turning point in the debate over corruption. Professor Jon S.T. Quah combines broad comparative concepts with a detailed account of corruption and reforms in six countries to spell out critical policy choices Asian societies face. His analysis of three patterns of corruption control is a welcome departure from the many reform proposals that emphasize the same factors everywhere, and thus do not really fit anywhere. Not only can corruption be controlled; Professor Jon S.T. Quah shows how several Asian societies have done it. At the same time, he identifies critical points at which controls can fail. Anyone concerned with corruption and development in Asia should give the lessons in his article a very careful thought.

 

The government is not the sole responsible for curbing corruption, citizen of a particular country also have duties to watch over their leaders. Reading this column entitled Best Practices for Curbing Corruption in Asia” has been an eye opener in many ways. Different countries are faced with different scenario of corruption and each needs a special attention.

 

The Philippines for example, even if a separate agency will be put up for monitoring corruption what’s the assurance that this agencies will really do its function. It happened before, the Philippine Government establish the PCGG but lately they‘re   found out that the most of its members were corrupt, this only mean that in order for a government to solve corruption the root of the problem should be looked up.

 

Also according to the column a certain government should punish those guilty of corruption, this is something true, unfortunately here in the Philippines by the times Former President Estrada was pronounced guilty of the crime Plunder he was sent to prison but several years later the government just to pleas all the followers/supporters of the former President decided to free Erap of his crimes.

 

The primary reason why there is a big difficulty for countries to curb corruption is because there is no unity to achieves maximum tolerance for corruption. A lot of civilians and government employees sees corruption as a daily occurring thing that is not in any way wrong.