don’t guess... know when (and how) to assess william f. mccomas, ph.d. director of science...
TRANSCRIPT
Don’t Guess . . . Know When (and How)
to Assess
William F. McComas, Ph.D.Director of Science Education Programs
Fellow, Center for Excellence in Teaching
Rossier School of Education
University of Southern California
Los Angeles, CA 90089-0031
What is Assessment?Assessment comes from the Latin meaning to “sit with” a learner and verify that the responses really mean what they seem to meanIn most assessment plans such
assurance is lackingMuch assessment is suspect in
terms of validity and reliability
What Can We Assess?
Knowledge
Attitudes
Skills
Assessment Plans Must:
Be free of biasReflect what is (or should have been) taughtProvide information to enhance instruction (teaching or curriculum)
Reveal misconceptions
Adapted from Champagne, Lovitts and Calinger (1990). Assessment in the Service of Instruction. Washington, AAAS.
Assessment should:
Be a learning experience for both students and teachersInvolve the evaluation of reasonable standards or benchmarks known in advance to the learnersBe based on a well-designed table of specifications The table of specifications lists the learning goals
associated with the number, kind, and nature of the individual assessment items
Timing is EverythingDiagnostic Evaluation (pretest) Used for gauging prior conceptions on which
to base instructional strategies
Formative Evaluation (in process) Useful for midcourse corrections
Summative Evaluation (posttest) A way of seeing how well you did as a teacher
Sorry, Wrong Number!
Norm-referenced; measures of performance against that of others Scores are typically reported in percentiles “Jane was in the 95th percentile on the SAT”
Criterion-referenced; absolute measures Scores are typically reported in percents “Bob got 82% of the questions correct”
Problems occur when instructors confuse the two types of assessment goals
Major Assessment Issues
High Stakes Test; a single assessment the result of which (positive or negative) will have a large impact on the future of the examinee Certification, licensure, entrance exams, etc.
Low Stakes Test; a single assessment the result of which does not have a large impact on the future of the examinee Most teacher-made tests
Major Issues in Assessment Design
Reliability; a measure of the consistency of the results of a given instrument (mathematically determined)
Validity; a measure of the degree to which the instrument measures what it is designed to measure (qualitatively determined)
What is Validity?
Validity refers to the accuracy of a measure
A measurement is valid when it measures what it is designed to measure and performs the functions that it purports to perform.
Does an indicator accurately measure the variable that it is intended to measure?
What is Reliability?Reliability refers to the dependability and consistency of the results provided by an instrument or technique.
A measurement is reliable when it produces the same result repeatedly with the same examinees or those from the same sample.
Does an indicator repeatedly report the same measure that it is intended to measure?
Forms of Assessment
Enhanced: new forms of style, substance and
goals
Traditional: typical forms of style and substance
Traditional Assessment
Goals; assignment of grades, student progress reporting and fault finding
Target; learners
Timing; summative
Methods; objective exams at the recall level of the knowledge domain
Enhanced AssessmentGoals; assignment of grades, student and program assessment
Target; learners, instructors and the curriculum
Timing; formative and summative
Methods; objective exams and expanded methods targeting KSA at all levels
Enhanced Assessment Includes
More data points (not just summative assessment) More domains (not just knowledge) but attitudes, skills, creativity, etc.,Higher levels of all domains (such as synthesis rather than memorization)More techniques such as portfolio, authentic assessment and empirically derived exams
Why Expand Assessment?
The nature and focus of assessment tell students (and teachers) what is important
Enhanced assessment encourages students to achieve in nontraditional realms (ex; creativity) and in nontraditional ways
Expanded assessment results provides the data necessary to make more thoughtful decisions about the curriculum, instruction, and student progress
Multi-Domain Assessment
Knowing and Understanding
Exploring and Discovering (process skills -- particularly in math and science)
Using and Applying Knowledge
Imagining and Creating
Feeling and Valuing (attitudes)
Understanding the nature of the discipline (such as science)
Domain: Knowledge & Understanding
Facts
Information
Concepts
Laws (principles)
Explanations and theories
Processes inherent to the discipline
Discipline-Based Skills and Processes
Primary Science Process SkillsObserving
Using Space/Time Relationships
Classifying (Grouping and Ordering)
Using Numbers (Quantifying)
Measuring
Communicating
Inferring
Predicting
Integrated Science Process Skills
Controlling and Identifying Variables
Interpreting Data
Formulating Hypotheses
Defining Operationally
Experimenting
Discipline-Based Skills and Processes
Domain: Using & Applying
Seeing learned concepts in everyday lifeApplying learned concepts to everyday lifeEvaluating reports of scientific developmentsMaking personal decisions based on legitimate knowledgeBecoming involved in science-related pursuitsTaking actions based on what has been learned
Domain: Imagining & Creating
Visualizing by producing mental images
Combining objects and ideas in unusual or useful new ways
Producing alternate uses for objects or ideas
Pretending
Converging, Diverging and Synthesizing
Domain: AttitudesValues and Feelings
Developing positive attitudes toward science in general and school scienceExploring and discussing both sides of an issueExpressing personal feelings in a constructive fashionMaking decisions based on values
Domain: Understanding the Nature of the Discipline
Appraising the strengths and limitations of the discipline and its methods
Evaluating and applying appropriate knowledge-production modes
Using criteria to place boundaries on the discipline (what is in and what is out)
Understanding the social aspects of work in the field
Bloom’s Taxonomy of the Cognitive Domain
Knowledge (Recall of data)
Comprehension (Understanding)
Application (Using data)
Analysis (Separating ideas into their parts)
Synthesis (Building something out of parts)
Evaluation (Making judgments)
Technique:Authentic Assessment
These are modes of assessment in which examinees are asked to perform in ways highly related to “real” situations (or situations in which the learning originally occurred)“Real” situations are those that exist in study, work, or life itselfThe best authentic assessments are both faithful (life-like) and comprehensive (wide-ranging)
Authentic Assessment Involves:
Worthy problems or questions of importance
Faithful representations of real situations
Options, constraints, and access to resources that are legitimate and not arbitrary
Problems that require a combination of knowledge, judgment and creativity
Judgment standards that are clearly stated in advance
Adapted from Wiggins (Nov / 93) Phi Delta Kappan
Technique: Empirically-Derived Exams
1 Determine the standards or bench-marks for achievement
2 Create an open-ended assessment tool3 Collect data and collapse similar
responses into typical response items4 Now, use the typical responses on a
multiple choice type examination
Defining science is difficult because science is complex and does many things. But, MAINLY science is:
A the study of fields like biology, physics, etc.B a body of knowledge (laws and theories) which
explain the world around usC exploring the unknown and making discoveriesD carrying out experiments to solve problemsE inventing or designing things like computersF an organization of people (scientists) who have
ideas for discovering new knowledgeG None of these choices fits my basic viewpoint
Technique: Performance Assessment
Practical examinations,Performances,ExhibitsThe task must be legitimate and contextualized, not artificial and remote in physical science - students measure the
temperature of a liquid vs. in ecology - students determine the chemicals in a
sludge sample
Technique: Portfolio Assessment
“. . . a systematic and organized collection of evidence used by the teacher and student to monitor growth of the student’s knowledge, skills, and attitudes in a specific subject area.”
Vavrus, L. (August, 1990). Put portfolios to the test.
Instructor. pp. 48-53.
What Can Portfolios Contain?Artifacts; tests, lab reports, photographs, meaningful journal entries
Reproductions; documents about typical events not usually captured (videos of presentations, photos of projects, etc.)
Attestations; something produced by someone else (i.e. notes from the teacher)
Productions; documents prepared for the portfolio (goal statement or reflection)
The Portfolio ProcessCollect materials for the portfolio
Select materials to be in the portfolio
Reflect on why those materials were selected
Project make a presentation based on the portfolio contents
Respect the contents and presentation
Show the Effect of the portfolio
Issues in Enhanced Assessment
How do we evaluate the results of the new schemes?
How do we report the results?
What do we do with the results in terms of advisement and promotion?
What are the solutions to questions of time and expense?
Typical Small Toolbox of Assessment Methods
Midterms and FinalsMultiple Choice Items
True False ItemsSummative Assessment
Philosophy
Larger Toolbox of Assessment Methods
Summative Formative
True/FalseMultiple ChoiceTerm PapersLabs / Practical exams
• Not graded• Perhaps on-line• Feedback to all concerned• Immediate feedback • Allow for real time adjustments
• Graded• End of a Unit• Feedback to students
Minute PaperEmpty OutlinesApproximate AnalogiesEtc.
Don’t Guess . . . Know you Know How
to Assess
William F. McComas, Ph.D.Director of Science Education Programs
Fellow, Center for Excellence in Teaching
Rossier School of Education
University of Southern California
Los Angeles, CA 90089-0031