C R E S S T / U C L A

Center for the Study of EvaluationNational Center for Research on Evaluation, Standards, and Student Testing

(CRESST)

New Models of Technology Sensitive Evaluation: Giving Up Old Program Evaluation

Ideas

Eva L. Baker and Joan L. Herman

SRIFebruary 25, 2000

C R E S S T / U C L A

Goals of Presentation

Outline Purposes and Challenges of Technology Evaluation

Describe Present Limitations of Technology Evaluation

Suggest Improvements

C R E S S T / U C L A

Purposes of Technology Evaluation

Soothe Anxiety

Justify Expenditures

Judge Impact

Identify Short-Falls

Improve Outcomes

Shore Up Managers’ Images

Demonstrate Technology Use in Evaluation

C R E S S T / U C L A

Limitations of Current Approaches to the

Evaluation of Technology

Conception of Evaluation

Designs

Measures

Validity of Results

That About Covers It

C R E S S T / U C L A

Limitations: Concept of Evaluation

The Scholarly Overlay of “Formative” and “Summative” Evaluation Makes Little Sense in Education in General and No Sense in Technology Implementations

Focus on “Value Added” Using New Outcomes Instead of Limiting Measures to Lowest Common Denominator

Evaluation Should Match Cycle-Time of Technology, e.g., No Five-Year Studies

C R E S S T / U C L A

Limitations: Designs

Existing Designs Are Usually Messy, Volunteer Studies of Available Classrooms

Randomized Treatment Allocations Are Possible, But Compensation from Home and Other Environments as Well as Pressures for Equal Access Make Them Impractical in the Long Run Without Creative Strategies

Treatments Need to Be Reconceptualized in Terms of Control and Uniformity

C R E S S T / U C L A

Limitations: Design/Procedures

Need for Collective Bargaining Agreements for Participation in Evaluation—Data Provision, Types of Information, Ability to Monitor Children and Adults

Human Subjects and Informed Consent

C R E S S T / U C L A

Limitations: Measures of Technology Effects

Opinion, Implementation, Smile-Tests

Student Performance Measures Insensitive to Implementations Mismatch Between Desired Goals and Measures Standardized Measures Mom and Pop Measures Lacking Technical and Credible

Qualities Masked by “Standards-Based” Rhetoric

C R E S S T / U C L A

Families of Cognitive Demands

Self-RegulationCommunication

ContentUnderstanding

ProblemSolving

Teamwork and

CollaborationLearning

C R E S S T / U C L A

Cross-Walk to Guide the Simultaneous Design of Assessment

and Instruction

Cognitive Models (Task Specification, Scoring Guides) Become Implemented Different Subject Matters

Domain-Independent and Domain-Dependent Components

Used for Design and/or Administration and/or Scoring

C R E S S T / U C L A

Next Generation: Authoring Systems for Multiple Purposes

Not an Item Bank

Capture Content, Task, Process

Instant Scoring and Feedback

Expert-Based

Beyond the Screen

C R E S S T / U C L A

Limitations: Validity of Results

Source: Inappropriate Measures, Unclear Assignment, Treatment Vagaries

Even in the Best Case: Generalizing to What? By the Time We Complete a Study, the Treatments Are Obsolete

C R E S S T / U C L A

Suggestions for Improvement

Defining the Objectives of Evaluation

Fixing Measures

Addressing the Concept of Evaluation

C R E S S T / U C L A

Distributed EvaluationCharacteristics and Functions

Conceptually Congruent with Distribution and Decentralization

Provides Information for Users and Program Managers

Allows Flexibility in Implementation Measures, e.g., Measures of Engagement, While Raising the Standards for Validity

C R E S S T / U C L A

An Indicators Approach

Flexibility

Longitudinal

Data Capture

Disaggregation and Aggregation

Data Export/Import

Local Engagement and Feedback

C R E S S T / U C L A

An Indicators Approach

Report Generation to Appropriate Constituencies

Updatable

Operational

Creates the Right “Management, High Tech” Patina

C R E S S T / U C L A

Quality School Portfolio Longitudinal

Database

Standards-Based

Multi-Purpose

Multi-User

Multiple Occasions

Local Goals

Automated Reports

C R E S S T / U C L A

Key Attributes of Distributed Evaluation

Measures: Fixed and Flexible

Some Common Performance Measures

And “Indicator” Mentality for Outcome Measures from Archival Sources

Common and Site Specific Implementation Measures

Fixed and Flexible Data Collection Schedule

Feedback Is a Feature in Every Design

C R E S S T / U C L A

More Characteristics

Local Site Is a Participant in Rather Than a Recipient of Evaluation

Software Based for Easy Data Entry Feedback, and Update

Tailored Reports

Design Independent