Welcome to Test Item Construction and Evaluation Learning Objectives Upon completion of this lesson, participants will be able to: Distinguish between assessment and evaluation. Recognize examples and non-examples of appropriate multiple-choice test item construction. Summarize the guidelines for constructing multiple-choice test items. Define principles of evaluation. Identify the criteria of a good evaluation. Identify appropriate evaluation methods for specific outcomes. Identify different types of course evaluation and evaluation data. Describe methods for evaluation of a training module. Prepare an end of course report. Identify and plan self development needs as an instructor. Definitions Educational Assessment: is the process of documenting, usually in measurable terms, knowledge, skills, attitudes, and beliefs. Educational Evaluation: is the evaluation process of characterizing and appraising some aspect/s of an educational process. Course Evaluation: is a questionnaire, which requires a written or selected response answer to a series of questions in order to evaluate the instruction of a given course. The term may also refer to the completed survey form or a summary of responses to questionnaires. TEST ITEM CONSTRUCTION Why Multiple-Choice? MULTIPLE-CHOICE VERSATILITY RELATION OF ITEM TYPE TO TEST OBJECTIVES FACTORESSAY or ORALSHORT ANSWERMULTIPLE-CHOICE Ability to organize +++- Potential diagnostic value Easily and reliably scored Takes little time to prepare ++- Measures higher mental processes ++- Broad content sampling Measures application +++ Adequate sampling of objectives Thorndike and Hagen (1969) have summarized the advantages of various essay and objective test formats. Positive attributes have been designated by plus signs in the table above, while distinct disadvantages have minus signs. Overview MULTIPLE-CHOICE ITEMS Anatomy Construction guidelines Examples TEST Anatomy of a Test Item Stem + Alternatives = Test Item What is primarily responsible for the increase in the average length of life in the USA during the last fifty years? a.Compulsory health and physical education courses in public schools. b.The reduced death rate among infants and young children. * c.The safety movement. Which has greatly reduced the number of deaths from accidents. d.The substitution of machines for human labor. Stem Alternatives Distractor Distractor Distractor Answer Advantages and Limitations Advantages of multiple-choice test items Good for many subject areas Takes less time to answer Less guessing Easy to score Advantages and Limitations Limitations of multiple-choice items Not good for displaying thought Still subject to some guessing Construction Guidelines 1.Constructed using a single written objective 2.Stem presents definite problem 3.Use simple and clear wording 4.Avoid clue words 5.Avoid grammatical giveaways 6.Alternatives of equal length 7.Highlight key words 8.Positively stated stems 9.Use as much wording as possible in stem 10.One clear best answer 11.Distractors are plausible and related Examples Construct each item to assess a single written objective. Items that are not written with a specific objective in mind often end up measuring lower-level objectives exclusively or covering trivial material that is of little educational worth. Examples BETTER EXAMPLE POOR EXAMPLE California: a.Contains the tallest mountain in the United States. b.Has an eagle on its state flag. c.Is the second largest state in terms of area. d.Was the location of the Gold Rush of 1849.* What is the main reason so many people moved to California in 1849? a.California land was fertile, plentiful, and inexpensive. b.Gold was discovered in central California.* c.The east was preparing for a civil war. d.They wanted to establish religious settlements. Examples POOR EXAMPLE BETTER EXAMPLE The term hypothesis, as used in research, is defined as: a. A conception or proposition formed by speculation or deduction or by abstraction and generalization from facts, explaining or relating an observed set of facts, given probability by experimental evidence or by factual or conceptual analysis but not conclusively established or accepted. b. A statement of an order or relation of phenomena that so far as is known is invariable under the given conditions, formulated on the basis of conclusive evidence or tests and universally accepted, that has been tested and proven to conform to facts. c. A proposition tentatively assumed in order to draw out its logical or empirical consequences and so test its accord with facts that are known or may be determined, of such a nature as to be either proved or disproved by comparison with observed facts. * The term hypothesis, as used in research, is defined as: a. An assertion explaining an observed set of facts that has not been conclusively established. b. A universally accepted assertion explaining an observed set of facts. c. A tentative assertion that is either proved or disproved by comparison with an observed set of facts. * The alternatives in the poor example above are rather wordy, and may require more than one reading before the student understands them clearly. In the better example, the alternatives have been streamlined to increase clarity without losing accuracy. Examples POOR EXAMPLE BETTER EXAMPLE A word used to describe a noun is called an: a.Adjective.* b.Conjunction. c.Pronoun. d.Verb. A word used to describe a noun is called: a.An adjective.* b.A conjunction. c.A pronoun. d.A verb. Examples POOR EXAMPLE BETTER EXAMPLE Which of the following is the best indication of high morale in a supervisors unit? a. The employees are rarely required to work overtime. b. The employees are willing to give first priority to attaining group objectives, subordinating any personal desires they may have. * c. The supervisor enjoys staying late to plan the next day. d. The unit gives expensive birthday presents to each other. Which of the following is the best indication of high morale in a supervisors unit? a. The employees are rarely required to work overtime. b. The employees willingly give first priority to attaining group objectives. * c. The supervisor enjoys staying late to plan for the next day. d. The unit members give expensive birthday presents to each other. Examples EXAMPLE Which of the following is the BEST example of treating an open wound? a.Stop the bleeding and apply a dressing. b.Stop the bleeding, apply a dressing and see a doctor. c.Clean the wound, stop the bleeding and apply a dressing. * Examples Positively stated stems Negatively-worded items are those in which the student is instructed to identify the exception, the incorrect answer, or the least correct answer. Examples POOR EXAMPLE BETTER EXAMPLE If the pressure of a certain amount of gas is held constant, what will happen if its volume is increased? a. The temperature of the gas will decrease. b. The temperature of the gas will increase. * c. The temperature of the gas will remain the same. If you increase the volume of a certain amount of gas while holding its pressure constant, its temperature will: a. Decrease. b. Increase. * c. Remain the same. Examples POOR EXAMPLE BETTER EXAMPLE The United States should adopt a foreign policy based on: a. A strong army and control of the North American continent. b. Achieving the best interest of all nations. c. Isolation from international affairs. d. Naval supremacy and undisputed control of the worlds sea lanes. * According to Alfred T. Mahan, the United States should adopt a foreign policy based on: a. A strong army and control of the North American continent. b. Achieving the best interest of all nations. c. Isolation from international affairs. d. Naval supremacy and undisputed control of the worlds sea lanes. * Examples POOR EXAMPLE Which of the following artists is known for painting the ceiling of the Sistine Chapel? a.Warhol. b.Flintstone. c.Michelangelo.* d.Santa Claus. BETTER EXAMPLE Which of the following artists is known for painting the ceiling of the Sistine Chapel? a.Botticelli. b.da Vinci. c.Michelangelo.* d.Raphael Write your objectives in this box. Keeps test item tied to a single objective. Write your test items in this box. Guidelines on the left will help you stay away from pitfalls that plague test writers. Worksheet Interim Summary MULTIPLE-CHOICE ITEMS Anatomy Construction guidelines Examples Critical Elements Data Judgment Educational objective Predetermined standard Types of Evaluation Placement Diagnostic Formative Summative Criteria for Evaluation Validity Reliability Objectivity Comprehensiveness Differentiation Organization of the Test Arrangement of test items Arrangement of alternatives Review test for accuracy Change test periodically Interim Summary Types of evaluation Criteria of evaluation Organization of the test Effective Course Evaluation Overview Effective Course Evaluation Lesson Outline: What is evaluation? The evaluation life cycle Portions of course to be evaluated Types of evaluation Collecting evaluation data Responding to significant feedback End of course report Instructor development What is Evaluation? Identifies and illuminates what is actually occurring in the area or activity that is the subject of evaluation Produces sufficient information to enable the evaluator, using agreed upon criteria, to make valid and useful judgments about what is being evaluated The Evaluation Life Cycle Planning Data Collection Analysis Action Portions to Evaluate Appropriateness of objectives for the learning group Organization and sequencing of content Effectiveness of the instructional methods Quality of teaching and learning aids used Validity of assessment methods used Performance of students in the course Overall course outcomes Types of Evaluations Process/Monitoring Outcome Impact Needs Assessment Structural Cost-Benefit Cost Effectiveness Formative Summative Data Collection SurveysInterviewsObservation Learner Feedback As learners are reaching the end of teaching or training, their experiences are particularly useful in terms of course evaluation. Learner Performance 1st1st 2nd2nd 3rd3rd 4th4th 5th5th The following approach is recommended in evaluating test scores. Accurately record trainee and class (average) scores. Compare class results against the standards. Check effectiveness of objectives. Make notes for later analysis. Identify students that bring down class average. If the class is achieving or surpassing standard, continue to evaluate. Self Evaluation In evaluating lessons, the following questions are a good starting point. What worked very well in terms of student learning, and why was it so effective? What am I least satisfied in the lesson and why? How would I change it for next time? Responding to Feedback The following list shows some typical issues that may come up during evaluation. Students find certain objectives too difficult to meet Some objectives are outdated Course materials are not sufficiently comprehensive Certain parts of the course materials are confusing Instructional methods may be uninteresting Assessments are not fully testing all learning objectives Assessment items are not correctly constructed Preparing End of Course Report This feedback is critical to the development of the course. Content of Report Format of Report Instructor Development Address the following questions. What are the specific competencies that need to be developed? What options are available to develop these competencies? How will I monitor and evaluate progress? What indicators will I use to determine success? Summary What is evaluation? The evaluation life cycle Portions of course to be evaluated Types of evaluation Collecting evaluation data Responding to significant feedback End of course report Instructor development Next Session Oct 9 10:30-12:00 Methods of instruction and classroom delivery Teaching techniques Questioning techniques Use of Video Case Studies Close Create a program you can be proud of!!