08 monitoring policy outcomes
TRANSCRIPT
-
8/4/2019 08 Monitoring Policy Outcomes
1/47
11
MONITORING POLICY OUTCOMES
Discussant: Samuel O. Parami, MM., MPD.
DDA 326: Public Policy Analysis
-
8/4/2019 08 Monitoring Policy Outcomes
2/47
2 2
Presentation Outline
A. IntroductionB. Monitoring in Policy Analysis
1. Sources of Information2. Types of Policy Outcomes
3. Types of Policy Actions4. Definitions and Indicators
C. Approaches to Monitoring1. Social Systems Accounting2. Social Experimentation3. Social Auditing4. Research & Practice Synthesis
D. Techniques for Monitoring
-
8/4/2019 08 Monitoring Policy Outcomes
3/47
3 3
The Power of Measuring Results
If you do not measure results, you can not tellsuccess from failure
If you can not see success, you can notreward it
If you can not reward success, you are probablyrewarding failure
If you can not see success, you can not learnfrom it
If you can not recognize failure, you can notcorrect it
If you can demonstrate results, you can winpublic support
Adapted from Osborne & Gaebler, 1992
-
8/4/2019 08 Monitoring Policy Outcomes
4/47
4 4
Introduction
The consequences of policy actions are neverfully known in advance and, for this reason, it
is essential to monitor policy actions after theyhave occurred. In fact, policy recommendations may be
viewed as hypotheses about the relationshipbetween policy actions and policy outcomes:if action A is taken at time t 1, outcome O willresult at time t 2 .
-
8/4/2019 08 Monitoring Policy Outcomes
5/47
5 5
A. Monitoring in Policy Analysis Monitoring is the policy-analytic procedure
used to produce information about the causesand consequences of public policies.
Monitoring, since it permits analysts todescribe relationships between policy-program operations and their outcomes, is theprimary source of knowledge about policy
implementation. Monitoring is primarily concerned about with
establishing factual premises about public
policy.
-
8/4/2019 08 Monitoring Policy Outcomes
6/47
6 6
A. Monitoring in Policy Analysis Monitoring performs at least four major
functions:
1. Compliance: monitoring helps determine
whether the actions of programadministrators, staff, and other stakeholdersare in compliance with standards andprocedures imposed by legislatures,regulatory agencies, and professional bodies.
2. Auditing: monitoring helps determine whetherresources and services intended for certaintarget groups and beneficiaries have actuallyreached them.
-
8/4/2019 08 Monitoring Policy Outcomes
7/477 7
A. Monitoring in Policy Analysis Monitoring performs at least four major
functions:
3. Accounting: monitoring produces
information that is helpful in accountingfor social and economic changes thatfollow the implementation of broad sets ofpublic policies and programs over time.
4. Explanation: monitoring also yieldsinformation that helps to explain why theoutcomes of public polices and programsdiffer.
-
8/4/2019 08 Monitoring Policy Outcomes
8/478 8
A. Monitoring in Policy Analysis
1. Sources of information . Information must be:
a. Relevant. Macronegative versus micropositive.
b. Reliable.
Observations precise and dependable.c. Valid.
Information about policy outcomes shouldmeasure what we think it is measuring.
-
8/4/2019 08 Monitoring Policy Outcomes
9/479 9
A. Monitoring in Policy Analysis
1. Sources of information . Information on policy outcomes.
Historical Statistics of theUnited States
Current Opinion
Statistical Abstract of theUnited States United States Census of Population byStates
County and City Data Book Congressional District Data Book
Census Use Study National Opinion Research CenterGeneral Social Survey
Social and EconomicCharacteristics of Students
Survey Research Center NationalElection Studies
Educational Attainment inthe United States
National Clearinghouse for MentalHealth Information
Current Population Reports National Clearinghouse for Drug Abuse
Information
-
8/4/2019 08 Monitoring Policy Outcomes
10/4710 10
A. Monitoring in Policy Analysis
1. Sources of information . Information on policy outcomes.
Current Population Reports National Clearinghouse for Drug AbuseInformation
The Social and EconomicStatus of the BlackPopulation in the UnitedStates
National Criminal Justice ReferenceService
Female Family Heads Child Abuse and Neglect ClearinghouseProject
Monthly Labor Review National Clearinghouse on RevenueSharing
Handbook of Labor Statistics Social Indicators
Congressional Quarterly State Economic and Social Indicators
-
8/4/2019 08 Monitoring Policy Outcomes
11/471111
A. Monitoring in Policy Analysis
1. Sources of information .
When information is not available fromexisting sources, monitoring may becarried out by some combination of
questionnaires, interviewing, fieldobservation, and the use of agencyrecords.
-
8/4/2019 08 Monitoring Policy Outcomes
12/4712 12
A. Monitoring in Policy Analysis
2. Types of policy outcomes.a. Consequences:
Policy outputs the goods, services, orresources received by target groups andbeneficiaries.
Policy impacts actual changes in behaviorand attitudes that result from policy outputs.
b. Populations: Target groups individuals, communities, ororganizations on whom policies and programsare designed to have an effect.
Beneficiaries
groups for whom the effects ofolicies are beneficial or valuable.
-
8/4/2019 08 Monitoring Policy Outcomes
13/4713 13
A. Monitoring in Policy Analysis
3. Types of policy actions. Policy actions havetwo major purposes.
a. Regulation actions designed to ensurecompliance with certain standards orprocedures.
b. Allocation actions that require inputs ofmoney, time, personnel, and equipment.
Regulative and allocative actions may haveconsequences that are distributive orredistributive.
-
8/4/2019 08 Monitoring Policy Outcomes
14/4714 14
A. Monitoring in Policy Analysis
4. Types of policy actions.a. Policy inputs the resources time,
money, personnel, equipment, andsupplies used to produce outputsand impacts.
b. Policy processes the
administrative, organizational, andpolitical activities and attitudes thatshape the transformation of policyinputs into policy outputs andimpacts.
-
8/4/2019 08 Monitoring Policy Outcomes
15/4715 15
A. Monitoring in Policy Analysis
ISSUEAREA
POLICY ACTIONS POLICY OUTCOMES
Inputs Processes Outputs Impacts
CriminalJustice.
Expendituresfor salaries,equipment,maintenance.
Illegal arrestsas percentagesof total arrests.
Criminalsarrested per100,000known crimes.
Criminalsconvicted per100,000known crimes.
MunicipalServices.
Expendituresfor sanitation
workers andequipment.
Morale amongworkers.
Totalresidences
served.
Cleanliness of streets.
Social welfare. Number of social workers.
Rapport withwelfarerecipients.
Welfare casesper socialworker.
Standard of living of dependentchildren.
-
8/4/2019 08 Monitoring Policy Outcomes
16/4716 16
A. Monitoring in Policy Analysis
4. Definitions and indicators. Variables versus constants. Definitions.
Constitutive definitions
gives meaning towords used to describe variables by usingsynonymous words.
Provide no concrete rules or guidelines foractually monitoring changes.
Operational definitions gives meaning to avariable by specifying the operations necessaryto experience and measure it.
Indicators directly observable characteristics.
-
8/4/2019 08 Monitoring Policy Outcomes
17/4717 17
B. Approaches to Monitoring
APPROACHTYPES OFCONTROL
TYPE OFINFORMATIONREQUIRED
Social systems
accountingQuantitative
Available and/or new
information
Social experimentationDirect manipulationsand quantitative
New information
Social auditingQuantitative and/orqualitative New information
Research and practicesynthesis
Quantitative and/orqualitative
Available information
-
8/4/2019 08 Monitoring Policy Outcomes
18/47
18 18
B. Approaches to Monitoring
POLICY POLICY POLICY POLICYINPUTS PROCESSES OUTPUTS IMPACTS
PRECONDITIONS UNFORESEEN SIDE-EFFECTSEVENTS AND
SPILLOVERS
CONTROLLED OUTCOMESMANIPULABLE ACTIONS
UNMANIPULABLE CAUSES UNCONTROLLED EFFECTS
In1In2...Ini
P 1P 2...P j
O 1O 2...O m
Im 1Im 2...Im n
PC 1PC 2...
PC p
E 1E 2...
E q
SES 1SES 2...
SES r
-
8/4/2019 08 Monitoring Policy Outcomes
19/47
19 19
1. Social Systems Accounting
An approach and set of methods that permit
analysts to monitor changes in objective and
subjective social conditions over time.
-
8/4/2019 08 Monitoring Policy Outcomes
20/47
20 20
1. Social Systems Accounting
The major analytic element of socialsystems accounting is the social indicator .
Statistics that measure socialconditions and changes therein overtime for various segments of apopulation. By social conditions, wemean both the external (social andphysical) and the internal (subjectiveand perceptional) contexts of human
existence in a given society.
1 S i l i
-
8/4/2019 08 Monitoring Policy Outcomes
21/47
2121
1. Social systems accounting
AREA INDICATORHealth and illness Persons in state mental hospitals
Public safety Persons afraid to walk alone at night
Education High school graduates aged 25 and older
Employment Labor force participation by women
Income Percent of population below the poverty lineHousing Households living in substandard units
Leisure and recreation Average annual paid vacation inmanufacturing
Population Actual and projected population
Government and politics Quality of public administration
Social values andattitudes
Overall life satisfaction and alienation
Social mobility Change from fathers occupation
Physical environment Air pollution index
Science and technology Scientific discoveries
Representative social indicators
-
8/4/2019 08 Monitoring Policy Outcomes
22/47
22 22
1. Social Systems Accounting
Must assume that the reasons forindicator change are related to policyactions. This is not always the case.
Advantages. Identify areas of insufficient information.
When indicators provide reliableinformation, possible to modify policies.
-
8/4/2019 08 Monitoring Policy Outcomes
23/47
23 23
1. Social Systems Accounting
Limitations. Choice of indicators influenced by values
of analysts. Social indicators frequently do not cover
manipulable policy actions. Most social indicators are based available
data about objective (rather than
subjective) social conditions. Social indicators provide little information
about how inputs are transformed intooutcomes.
-
8/4/2019 08 Monitoring Policy Outcomes
24/47
24 24
1. Social Systems Accounting
-
8/4/2019 08 Monitoring Policy Outcomes
25/47
25 25
1. Social Systems Accounting
-
8/4/2019 08 Monitoring Policy Outcomes
26/47
26 26
Coefficients a
13.516 2.561 5.277 .000
.340 .243 .322 1.400 .17971.419 11.060 6.458 .000-1.551 .403 -.705 -3.849 .002-6.745 22.133 -.305 .7641.218 .494 .525 2.465 .025
(Constant)
Number of years(Constant)Number of years(Constant)Number of years
Model1
1
1
Breaks indisaster patterns1950 - 1971
1972 - 1988
1989 - 2006
B Std. Error
UnstandardizedCoefficients
Beta
StandardizedCoefficients
t Sig.
Dependent Variable: Total Disaster Declarationsa.
Model Summarya
Breaks in disaster patterns R R Square Sig.1950 - 1971 0.322 0.1031972 - 1988 0.705 0.497 ***1989 - 2006 0.525 0.275 *aPredictors: (Constant), Number of years
*p
-
8/4/2019 08 Monitoring Policy Outcomes
27/47
27 27
1. Social Systems Accounting
-
8/4/2019 08 Monitoring Policy Outcomes
28/47
28 28
2. Social Experimentation Use of social indicators often leads to random
innovation rather than systematic change.
Social experimentation is the process ofsystematically manipulating policy actions inway that permits more or less preciseanswers to questions about the sources ofchange in policy outcomes.
-
8/4/2019 08 Monitoring Policy Outcomes
29/47
29 29
2. Social Experimentation
Characteristics and procedures. Direct control of experimental treatment. Comparison (control) groups. Random assignment.
Goal: internal validity the capacity ofexperiments and quasi-experiments tomake valid causal inferences about theeffects of actions on outcomes.
-
8/4/2019 08 Monitoring Policy Outcomes
30/47
30 30
2. Social Experimentation
Threats to internal validity. History. Maturation. Instability. Instrumentation and testing.
Mortality. Selection. Regression artifacts.
-
8/4/2019 08 Monitoring Policy Outcomes
31/47
3131
2. Social Experimentation
Social experimentation is weakestin the area of external validity orgeneralizability.
Neglects policy processes.
-
8/4/2019 08 Monitoring Policy Outcomes
32/47
32 32
3. Social Auditing
One of the limitations of social systemsaccounting and social experimentationis that both approaches neglect oroversimplify policy processes.
-
8/4/2019 08 Monitoring Policy Outcomes
33/47
33 33
3. Social Auditing
Social auditing explicitly monitorsrelations among inputs, processes,outputs, and impacts in an effort to trace
policy inputs from disbursement to finalrecipient. Social auditing helps to determine
whether ineffective outcomes are theresult of inadequate inputs or processesthat divert resources or services from
intended target groups or beneficiaries.
-
8/4/2019 08 Monitoring Policy Outcomes
34/47
34 34
3. Social Auditing
Processes monitored of twomain types:
Resource diversion (from targetor beneficiary groups).
Resource transformation(changes in meaning of policyactions from administrator torecipient).
-
8/4/2019 08 Monitoring Policy Outcomes
35/47
35 35
Social Audit Steps
The three phases of a social audit
Phase 1: Design and data collection
a. Clarify the strategic focusb. Design instruments, pilot test
c. Collect information from householdsand key informants in a panel ofrepresentative communities
-
8/4/2019 08 Monitoring Policy Outcomes
36/47
36 36
Social Audit Steps
Phase 2: Evidence-based dialogue and
analysisa. Link household data with information from
public services
b. Analyze findings in a way that points toaction
c. Take findings back to the communities for
their views about how to improve thesituation
d. Bring community members into discussionof evidence with serviceproviders/planners.
-
8/4/2019 08 Monitoring Policy Outcomes
37/47
37 37
Social Audit Steps
Phase 3: Socialization of evidence forpublic accountability
Work-shopping
Communication strategy Evidence-based training of planners
and service providers Media training Partnerships with civil society
-
8/4/2019 08 Monitoring Policy Outcomes
38/47
38 38
4. Research and practice synthesis.
An approach to monitoring that involves thesystematic compilation, comparison, andassessment of the results of past efforts toimplement public policies.
Two primary sources of available information.a. Case studies of policy formulation and
implementation.
b. Research reports that address relationsamong policy actions and outcomes.
-
8/4/2019 08 Monitoring Policy Outcomes
39/47
39 39
4. Research and practice synthesis.
Two methods.a. Case survey method a set of
procedures used to identify andanalyze factors that account forvariations in the adoption andimplementation of policies.
Requires case coding scheme, a set ofcategories that capture key aspects ofpolicy inputs, processes, outputs, andimpacts.
-
8/4/2019 08 Monitoring Policy Outcomes
40/47
40 40
Research and practice synthesis
Two methods.b. Research survey method a set of
procedures used to compare andappraise results of past research onpolicy actions and outcomes. Yields empirical generalizations about sources
of variation in policy outcomes, summaryassessments of the confidence researchershave in these generalizations, and policyalternatives or action guidelines that are impliedin these generalizations.
Requires the construction of a format forextracting information about research reports.
-
8/4/2019 08 Monitoring Policy Outcomes
41/47
4141
4. Research and practice synthesis
Advantages.
a. Comparatively efficient way to compile andappraise an increasingly large body of cases andresearch reports on policy implementation.
b. The case survey method is one among severalways to uncover different dimensions of policyprocesses that affect policy outcomes.
c. The case survey method is a good way toexamine both objective and subjective conditions.
-
8/4/2019 08 Monitoring Policy Outcomes
42/47
42 42
4. Research and practice synthesis
Limitations. All related to reliability and validity of
information.
-
8/4/2019 08 Monitoring Policy Outcomes
43/47
43 43
4. Research and Practice Synthesis
Example
David H. Greenberg; Charles Michalopoulos; Philip K. Robins.
Do Experimental and NonexperimentalEvaluations Give Different Answers about theEffectiveness of Government-Funded TrainingPrograms?
-
8/4/2019 08 Monitoring Policy Outcomes
44/47
44 44
4. Research and Practice Synthesis: Exampl e
Abstract
This paper uses meta-analysis to investigatewhether random assignment (or experimental)
evaluations of voluntary government-fundedtraining programs for the disadvantaged haveproduced different conclusions thannonexperimental evaluations.
Information includes several hundredestimates from 31 evaluations of 15 programsthat operated between 1964 and 1998.
-
8/4/2019 08 Monitoring Policy Outcomes
45/47
45 45
4. Research and Practice Synthesis: Example
The results suggest that experimentaland nonexperimental evaluations yieldsimilar conclusions about theeffectiveness of training programs, butthat estimates of average effects for
youth and possibly men might havebeen larger in experimental studies.
-
8/4/2019 08 Monitoring Policy Outcomes
46/47
46 46
4. Research and Practice Synthesis: Example
The results also suggest that variationamong nonexprimental estimates ofprogram effects is similar to variationamong experimental estimates for menand youth, but not for women (for whomit seems to be larger), although small
sample sizes make the estimateddifferences somewhat imprecise for allthree groups.
-
8/4/2019 08 Monitoring Policy Outcomes
47/47
C. Techniques for Monitoring
APPROACHGRAPHICDISPLAYS
TABULARDISPLAYS
INDEXNUMBERS
INTER-RUPTEDTIME-SERIESANALYSIS
CONTROL-SERIESANALYSIS
REGRES-SION DIS-CONTI-NUITYANALYSIS
Socialsystemsaccounting X X X X X O
Socialauditing x x x x x oSocialexperimen-
tation x x x x x xResearchandpracticesynthesis x x o o o o