designing and conducting summative evaluations

Designing and Conducting Summative Evaluations

“Education is our most important national investment. It commands the time and attention of 60 million Americans-3 citizens in 10. It consumes an annual public and private expenditure in excess of 120 billion. Every citizen has a vital personal stake in this investment. Our ability to advance both economically and technologically, our country’s entire intellectual and cultural life depend on the success of our great educational enterprise.” Jimmy Carter October 17, 1979

Michael W. Moore cwid # 010028336


Click icon to add picture

“Summative evaluations is the process of collecting data and information in order to make decisions about the acquisition or continued use of instruction. The distinction between formative and summative evaluation became important several decades ago when advocates debated public school curriculum and each new media delivery system that was claimed to be better. Studies were conducted and the winners were determined.”


Describe the purpose of summative evaluation. Describe the purpose of two phases of summative evaluation and the decisions resulting from each phase. Design a summative evaluation for comparing alternative sets of candidate instructional materials.


“Summative Evaluation is a process of collecting data and information in order to make decisions about the acquisition or continued use of some instruction. While formative evaluation is the process of collecting data and information in order to improve the effectiveness of instruction.” Dick et.al. 2005


“The distinction between formative and summative evaluation became important several decades ago when advocates for each new public school curriculum and each new media delivery system claimed that it was better than its competitors.” Dick et.al. 2005


“Studies were conducted as soon as possible to determine the ‘winner.’ Often the innovation did not do as well as traditional instruction. This came as no surprise to experienced evaluators, who knew that the innovation was really in draft form while the traditional instruction had been used, and revised for many years. Dick et.al. 2005


“Persuasive arguments were made to postpone such comparisons until an innovation had been formatively evaluated and revised to the point that all the major problems were removed, and it was suitable for routine use.” Dick et.al. 2005


“Finally then it would be appropriate to compare the innovation to other forms of instruction or to document exactly what the new innovation could do in terms of learner performance, attitudes, instructor reactions, costs, durability, and compatibility with other instruction in a curriculum or organization.” Dick et.al. 2005


“It is not necessary to have a comparison between or among instructional products to have a summative evaluation. A summative study can be done to document the effects of a single innovation.” Dick et.al. 2005


“ For the designer, the fundamental importance of formative evaluation for the effective use of the systematic design process cannot be overstated. Every designer should be able to conduct formative evaluations with confidence.” Dick et.al. 2005


“Such is not the case with summative evaluations as they were originally conceived. When it came to an impartial study of the effects of an innovative delivery system or an innovative curriculum, or both, most decision makers did not want the study to be conducted by a developer or advocate for one of the competing formats of instruction: thus, external or third-party evaluators were often hired to conduct summative evaluations.” Dick et. al. 2005


“In recent years the tenor of summative evaluations has changed. The question is no longer , ‘Which is better? Instead it is, ‘Did the intervention, including the instruction, solve the problem that led to the need for instruction in the first place ?’ In other words, instruction is being considered as a solution to a problem, and the ultimate summative evaluation question is , ‘Did it solve the problem?’” Dick et.al. 2005

Designing and Conducting Summative Evaluations“Summative evaluation is defined as the design of evaluation studies and the collection of data to verify the effectiveness of instructional materials with target learners. Its main purpose is to make go-no-go decisions about maintaining currently used instructional materials or about adopting materials that have the potential for meeting an organization’s defined instructional needs.” Dick et.al. 2005


“A summative evaluation has two main phases: expert judgment and field trial. The purpose of the expert judgment phase is to determine whether currently used instruction or other candidate instruction has the potential for meeting an organization’s defined instructional needs.” Dick et.al. 2005


“The purpose of the field trial phase is to document the effectiveness of promising instruction with target group members in the intended setting. The analyses and decisions to be made during each phase and the evaluation activities supporting each one are listed.


“The activities undertaken in the expert judgment phase to decide whether candidate instruction is promising include (1) evaluating the congruence between the organization’s instructional needs and candidate instruction, (2) evaluating the completeness and accuracy of candidate instruction, (3) evaluating the instructional strategy contained in the candidate instruction, (4) evaluating the utility of the instruction, and (5) determining current users’ satisfaction with the instruction.” Dick et.al. 2005


“ The field trial phase has two components. The first is outcomes analysis, which involves determining the effect of instruction on learners’ skills, on the job (transfer), and on the organization (need resolution). “ Dick et.al. 2005


“The second component, called management analysis, includes assessing instructor and supervisor attitudes related to learner performance, implementation feasibility, and costs. The field trial for the summative evaluation includes documenting learner performance and attitudes, documenting instructor /implementer attitudes and documenting procedures and resources required to implement the instruction.” Dick et.al. 2005


Summary: “Summative evaluations are conducted to make decisions about whether to maintain or adopt instruction. The primary evaluator in a summative evaluation is rarely the designer or developer of the instruction; the evaluator is frequently unfamiliar with the materials, the organization requesting the evaluation, or the setting in which the materials are evaluated.” Dick et.al. 2005

MICHAEL WADE MOORE2937 Alta Road Montgomery, Alabama

36110 Home Phone (334) 593-2548 Cell Phone (334) 306-7502 Personal E-mail: [email protected] School E-mail:[email protected] Skype Account: michael.moore 238 Dropbox.com:[email protected]

mailto:[email protected]