©2003-2005 subjective metrics, inc. 05.11.20051 waypoint online assessment and structured peer...
TRANSCRIPT
![Page 1: ©2003-2005 Subjective Metrics, Inc. 05.11.20051 waypoint online assessment and structured peer review COAS: Drexel University Andrew J. McCann visiting](https://reader030.vdocuments.us/reader030/viewer/2022032802/56649e165503460f94b007ff/html5/thumbnails/1.jpg)
1©2003-2005 Subjective Metrics, Inc.05.11.2005
waypointonline assessment and structured peer review
COAS: Drexel University
Andrew J. McCannvisiting professor of english, drexel university
founder and president, subjective metrics, inc.www.gowaypoint.com
215.713.9393
![Page 2: ©2003-2005 Subjective Metrics, Inc. 05.11.20051 waypoint online assessment and structured peer review COAS: Drexel University Andrew J. McCann visiting](https://reader030.vdocuments.us/reader030/viewer/2022032802/56649e165503460f94b007ff/html5/thumbnails/2.jpg)
2©2003-2005 Subjective Metrics, Inc.05.11.2005
Agenda
• Background
• Introduce waypoint
– Brief overview
• Drexel applications
• Live Demonstration
– Evaluate
– Manage
– Libraries
![Page 3: ©2003-2005 Subjective Metrics, Inc. 05.11.20051 waypoint online assessment and structured peer review COAS: Drexel University Andrew J. McCann visiting](https://reader030.vdocuments.us/reader030/viewer/2022032802/56649e165503460f94b007ff/html5/thumbnails/3.jpg)
3©2003-2005 Subjective Metrics, Inc.05.11.2005
Problem: Feedback
• Handwritten and/or manually typed
• “Lost”
• Little accountability
• Consistency issues
• No data
• Needed: not AI – but a technological tool
![Page 4: ©2003-2005 Subjective Metrics, Inc. 05.11.20051 waypoint online assessment and structured peer review COAS: Drexel University Andrew J. McCann visiting](https://reader030.vdocuments.us/reader030/viewer/2022032802/56649e165503460f94b007ff/html5/thumbnails/4.jpg)
4©2003-2005 Subjective Metrics, Inc.05.11.2005
Waypoint: Evaluation Tool
• Encourages pre-written feedback:
clearer, more detailed explanations
of key concepts
• Facilitates sharing of assessments
amongst instructors
• Quantifies evaluations by skill
• Archives all feedback
Web-based:
• cross-platform
• backed up
• zero maintenance
![Page 5: ©2003-2005 Subjective Metrics, Inc. 05.11.20051 waypoint online assessment and structured peer review COAS: Drexel University Andrew J. McCann visiting](https://reader030.vdocuments.us/reader030/viewer/2022032802/56649e165503460f94b007ff/html5/thumbnails/5.jpg)
5©2003-2005 Subjective Metrics, Inc.05.11.2005
Waypoint Process
![Page 6: ©2003-2005 Subjective Metrics, Inc. 05.11.20051 waypoint online assessment and structured peer review COAS: Drexel University Andrew J. McCann visiting](https://reader030.vdocuments.us/reader030/viewer/2022032802/56649e165503460f94b007ff/html5/thumbnails/6.jpg)
6©2003-2005 Subjective Metrics, Inc.05.11.2005
Develop Assessment
![Page 7: ©2003-2005 Subjective Metrics, Inc. 05.11.20051 waypoint online assessment and structured peer review COAS: Drexel University Andrew J. McCann visiting](https://reader030.vdocuments.us/reader030/viewer/2022032802/56649e165503460f94b007ff/html5/thumbnails/7.jpg)
7©2003-2005 Subjective Metrics, Inc.05.11.2005
Develop Assessment
![Page 8: ©2003-2005 Subjective Metrics, Inc. 05.11.20051 waypoint online assessment and structured peer review COAS: Drexel University Andrew J. McCann visiting](https://reader030.vdocuments.us/reader030/viewer/2022032802/56649e165503460f94b007ff/html5/thumbnails/8.jpg)
8©2003-2005 Subjective Metrics, Inc.05.11.2005
Receive Paper or Exam(Hard copy shown here)
![Page 9: ©2003-2005 Subjective Metrics, Inc. 05.11.20051 waypoint online assessment and structured peer review COAS: Drexel University Andrew J. McCann visiting](https://reader030.vdocuments.us/reader030/viewer/2022032802/56649e165503460f94b007ff/html5/thumbnails/9.jpg)
9©2003-2005 Subjective Metrics, Inc.05.11.2005
Receive Paper or Exam(Hard copy shown here)
![Page 10: ©2003-2005 Subjective Metrics, Inc. 05.11.20051 waypoint online assessment and structured peer review COAS: Drexel University Andrew J. McCann visiting](https://reader030.vdocuments.us/reader030/viewer/2022032802/56649e165503460f94b007ff/html5/thumbnails/10.jpg)
10©2003-2005 Subjective Metrics, Inc.05.11.2005
Evaluate Against Skills
![Page 11: ©2003-2005 Subjective Metrics, Inc. 05.11.20051 waypoint online assessment and structured peer review COAS: Drexel University Andrew J. McCann visiting](https://reader030.vdocuments.us/reader030/viewer/2022032802/56649e165503460f94b007ff/html5/thumbnails/11.jpg)
11©2003-2005 Subjective Metrics, Inc.05.11.2005
Evaluate Against Skills
![Page 12: ©2003-2005 Subjective Metrics, Inc. 05.11.20051 waypoint online assessment and structured peer review COAS: Drexel University Andrew J. McCann visiting](https://reader030.vdocuments.us/reader030/viewer/2022032802/56649e165503460f94b007ff/html5/thumbnails/12.jpg)
12©2003-2005 Subjective Metrics, Inc.05.11.2005
Respond to Student
![Page 13: ©2003-2005 Subjective Metrics, Inc. 05.11.20051 waypoint online assessment and structured peer review COAS: Drexel University Andrew J. McCann visiting](https://reader030.vdocuments.us/reader030/viewer/2022032802/56649e165503460f94b007ff/html5/thumbnails/13.jpg)
13©2003-2005 Subjective Metrics, Inc.05.11.2005
Respond to Student
![Page 14: ©2003-2005 Subjective Metrics, Inc. 05.11.20051 waypoint online assessment and structured peer review COAS: Drexel University Andrew J. McCann visiting](https://reader030.vdocuments.us/reader030/viewer/2022032802/56649e165503460f94b007ff/html5/thumbnails/14.jpg)
14©2003-2005 Subjective Metrics, Inc.05.11.2005
Snapshot Analysis
![Page 15: ©2003-2005 Subjective Metrics, Inc. 05.11.20051 waypoint online assessment and structured peer review COAS: Drexel University Andrew J. McCann visiting](https://reader030.vdocuments.us/reader030/viewer/2022032802/56649e165503460f94b007ff/html5/thumbnails/15.jpg)
15©2003-2005 Subjective Metrics, Inc.05.11.2005
Snapshot Analysis
![Page 16: ©2003-2005 Subjective Metrics, Inc. 05.11.20051 waypoint online assessment and structured peer review COAS: Drexel University Andrew J. McCann visiting](https://reader030.vdocuments.us/reader030/viewer/2022032802/56649e165503460f94b007ff/html5/thumbnails/16.jpg)
16©2003-2005 Subjective Metrics, Inc.05.11.2005
Longitudinal Cohort Analysis
Organizational Behavior: Research Competency1 to 4 scale; 110 students evaluated
0
10
20
30
40
50
60
70
1 2 3 4
# T
eam
s Feb '04Mar '04
Jun '04
![Page 17: ©2003-2005 Subjective Metrics, Inc. 05.11.20051 waypoint online assessment and structured peer review COAS: Drexel University Andrew J. McCann visiting](https://reader030.vdocuments.us/reader030/viewer/2022032802/56649e165503460f94b007ff/html5/thumbnails/17.jpg)
17©2003-2005 Subjective Metrics, Inc.05.11.2005
Process Summary
Organizational Behavior: Research Competency1 to 4 scale; 110 students evaluated
0
10
20
30
40
50
60
70
1 2 3 4
# T
eam
s Feb '04Mar '04
Jun '04
![Page 18: ©2003-2005 Subjective Metrics, Inc. 05.11.20051 waypoint online assessment and structured peer review COAS: Drexel University Andrew J. McCann visiting](https://reader030.vdocuments.us/reader030/viewer/2022032802/56649e165503460f94b007ff/html5/thumbnails/18.jpg)
18©2003-2005 Subjective Metrics, Inc.05.11.2005
Drexel Applications
• tDec Humanities (Dr. Valarie Arms)– Evaluation of all major writing assignments (600 students)
• Writing Center (Harriet Millan)– Customized assessments for WITs
• College of Engineering (Kevin Scoles & Adam Fontecchio)– Evaluation of lab reports with WITs and TAs
• College of Business (Frank Linnehan)– Structured response to student writing & accreditation data
generation
• Engineering Management (Mike Scheuerman)– Peer review
![Page 19: ©2003-2005 Subjective Metrics, Inc. 05.11.20051 waypoint online assessment and structured peer review COAS: Drexel University Andrew J. McCann visiting](https://reader030.vdocuments.us/reader030/viewer/2022032802/56649e165503460f94b007ff/html5/thumbnails/19.jpg)
19©2003-2005 Subjective Metrics, Inc.05.11.2005
Demonstration
• www.gowaypoint.com• Evaluate
– “Final Report”• Peer Review
– “Final Report”• Manage
– Quantification– Sorting– Data analysis
![Page 20: ©2003-2005 Subjective Metrics, Inc. 05.11.20051 waypoint online assessment and structured peer review COAS: Drexel University Andrew J. McCann visiting](https://reader030.vdocuments.us/reader030/viewer/2022032802/56649e165503460f94b007ff/html5/thumbnails/20.jpg)
20©2003-2005 Subjective Metrics, Inc.05.11.2005
Additional Features
• Self-assessment
• Collaborative assessment
– Multiple instructors can contribute to evaluation
![Page 21: ©2003-2005 Subjective Metrics, Inc. 05.11.20051 waypoint online assessment and structured peer review COAS: Drexel University Andrew J. McCann visiting](https://reader030.vdocuments.us/reader030/viewer/2022032802/56649e165503460f94b007ff/html5/thumbnails/21.jpg)
21©2003-2005 Subjective Metrics, Inc.05.11.2005
Questions and Discussion
![Page 22: ©2003-2005 Subjective Metrics, Inc. 05.11.20051 waypoint online assessment and structured peer review COAS: Drexel University Andrew J. McCann visiting](https://reader030.vdocuments.us/reader030/viewer/2022032802/56649e165503460f94b007ff/html5/thumbnails/22.jpg)
22©2003-2005 Subjective Metrics, Inc.05.11.2005
Testimonials
• “I've been doing this for twenty years and have my own system of grading and commentary…and found little need to improve. And then Waypoint came along. Somehow--and I really can't even explain it--but my grading time has been cut down by more than half and my students are actually thanking me for the in-depth commentary.”– Professor Ken Bingham
• “Usually I have trouble criticizing a peer’s paper if I’m not given certain criteria to judge. During this peer review, I was actually focused and excited about judging a peer’s paper.”– Erin Williams, COE 2008
• “If this peer review program was available in high school, I would have probably done a lot better in English.” – Ed Itaas, COE 2008
![Page 23: ©2003-2005 Subjective Metrics, Inc. 05.11.20051 waypoint online assessment and structured peer review COAS: Drexel University Andrew J. McCann visiting](https://reader030.vdocuments.us/reader030/viewer/2022032802/56649e165503460f94b007ff/html5/thumbnails/23.jpg)
23©2003-2005 Subjective Metrics, Inc.05.11.2005
Testimonials
• “An innovative teacher-friendly, student-friendly, efficient approach to grading writing—the most creative and time-saving method for evaluating writing I have ever found or used.”
– Gayle, a high school English teacher with 35 years’ experience
• “I don’t know how I ran a writing program for three years without it.”
– Harriet Millan, Director of the University Writing Program
![Page 24: ©2003-2005 Subjective Metrics, Inc. 05.11.20051 waypoint online assessment and structured peer review COAS: Drexel University Andrew J. McCann visiting](https://reader030.vdocuments.us/reader030/viewer/2022032802/56649e165503460f94b007ff/html5/thumbnails/24.jpg)
24©2003-2005 Subjective Metrics, Inc.05.11.2005
Quantitative Data
• Anonymous survey of Peer Review process
– 114 freshman engineers: 51 use waypoint, 63 the ‘old’ method
1-5 Scale (1 – strongly disagree, 5 – strongly agree)
wp (N=51)
Mean
‘old’ (N=63)
Mean
P value
(t-test, 95% CL)
1. I received clear and helpful criticism of my draft.
3.90 3.52 p<0.025
2. Evaluating other students’ papers helped me better understand the assignment and the play.
3.86 3.48 p<0.02
3. I made significant changes to my first draft based on peer review feedback.
3.81 3.25 p<0.005
4. I made significant changes to my first draft independent of peer review feedback.
3.49 3.43 p=0.38