Calibrated Peer ReviewTM

Writing, Collaboration, and Critical Thinking

http://cpr.molsci.ucla.edu

Arlene A. Russell, UCLA

University of Maryland

November 13, 2003

http://cpr.molsci.ucla.edu

Calibrated Peer Review (CPR)• was created under the Molecular Science Project,

an NSF-funded systemic reform project• is an instructional tool that uses writing and peer

review to teach higher-order thinking skills and peer collaboration

• manages the writing and anonymous review processes of instructor-selected assignments

• Integrate telecommunications and technology into instructional processes so that students learn to

• explore ideas independently• write about chemistry• collaborate• work to a mastery level• take responsibility for their own learning

Tenets of CPR

• Expository writing promotes understanding

• Clear writing demonstrates clear thinking

• Peer review and evaluation require higher-order, critical thinking skills

• Writing-to-learn Learning-to-write– But, writing-to-learn can (and should) include

attention to writing skills

Writing-to-Learn • Used extensively in humanities and social sciences

• Requires small classes

– Science and engineering have put their resources into labs

• Imposes intensive grading workload

~ ~• Seldom used in large science classes, because

– Workload too high, not the job of scientists, large classes, not objective,…….

Peer Review

• Accepted process for validating scientific research

• Essential skill required of all scientists

~ ~

• Seldom taught in science classes

• Not used in large lecture courses

Peer Review• “Peer Assessment Between Students in

Colleges and Universities” Review of Educational Research, Fall 1998, 68, pp 249-276

• Little use of computers except to submit work

Peer Review

• “Student Peer Assessment in Higher Education: A Meta-Analysis Comparing Peer and Teacher Marks” N. Falchikov and J. Goldfinch, Review of Educational Research, Fall 2000, 70, pp 287-322 – Peer assessments resembled teacher assessments most closely

“when global judgements based on well understood criteria were used.”

– Advanced students not better assessors than beginner students.

Calibrated Peer Review (CPR)

• Both writing and reviewing improve students’ understanding of the topic

– involves writing (reflective thinking)– involves reviewing (critical thinking)

Student’s Interaction with CPRStudents

– carry out a guided study of source material – write an essay (~300 words) about what they have

studied.– are trained as reviewers of this topic by studying a set of

“Calibration” essays and evaluating them on specific questions

– anonymously review three of their peers’ essays and provide written feedback to them

– apply their evaluation skills to self-review their own essays

Instructor Responsibility

• Selects the topics from the library of assignments

• Sets grading criteria and due dates for submission of essays and reviews

• Monitors student progress and performance

(Participates in the assignment as a student)

CPR Assignments (July 2003)

0

25

50

75

100

125

150

175

200

225

250

275

ChemistryEng CompBiology

Tech WritingEcon/Man

MathGeology

EthicsSociology

PsychMedicine

Proposal Writing

Sci EdEng Lit

High SchoolPhysics

CPR

AstronomyComp Sci

LawPoli SciBiochem

EngineeringNursing

Atmos SciOthers

Discipline

Assignments in Personal and Public Libraries

Complete Incomplete Complete Assignments = 889 Incomplete Assignments = 353

So What? Do CPR Assignments Make a Difference?

• Is content knowledge improved?

• Are evaluation skills developed?

• Do students learn to collaborate?

• What do students think about CPR?

Evaluation of CPR Economics - CSUN

• Comparison of intact classes

• Ten case-study assignments

• Three sections (same instructor) of Economics 200; 2 CPR based, 1 traditional with essays but no evaluation component

• Repeated over two semesters

Average score by quintile on midterm(Economics 200, Fall 1999)

0

10

20

30

40

50

60

70

80

q1 q2 q3 q4 q5

quintile

Average score on midterm

CPR

no CPR

Evaluation of CPR Biology - CSUF

• One class, one instructor• Time-series design• Ten topics, 5 taught through CPR, alternately with

5 taught through lecture and simulation

• Nancy J. Pelaez “Problem-based Writing with Peer Review Improves Academic Performance in Physiology” Advan Physiol Educ, Sep 2002; 26: 174 - 184.

y = 0.0007x + 0.1068R

2 = 0.4305

y = 0.001x - 0.208R2 = 0.5049

0%

20%

40%

60%

80%

100%

500 600 700 800 900 1000

Total points for BIO310

CPRconcepts

LectureConceptsLinear (CPRconcepts)Linear (LectureConcepts)

Mean multiple choice test scores were significantly better for topics taughtusing CPR compared with those taught using traditional instruction.

72+ 2% versus 65 + 2%, P < 0.001, Paired t-test.

Evaluation of CPRLife Science 1H - UCLA

• One class, one instructor (26 students)• Three CPR assignments• Multiple-choice midterm and final exam questions classified

as– relating directly to topics in CPR assignments – requiring transfer of understanding of CPR topics– addressing non-CPR topics.

• Mean performance: 84.8% on CPR topics, 79.7% on non-CPR topics (paired t-test: p = 0.006)

Total course points

1000900800700600

Percent correct on exams

110

100

90

80

70

60

50

CPR topics

non-CPR topics

Evaluation of CPRChemistry - UCLA

• General chemistry

• Midterm question on theory of two-component phase diagram

• Two quarters traditional lecture and problems

• One-quarter with an additional CPR assignment

Exam ScoresQuarter CPR Actual Percent

use grade• Winter 1998 No 6.2/10 62 %

• Winter 2000 No 9.2/15 61 %

• Spring 2000 Yes 15.1/20 71%

Evaluation of CPR-writing Skills Chemistry- UCLA

• Senior level environmental chemistry; all evaluation based on writing

• Midterm - 3 essays on issues -topics specified

• Final - 6 essays on issues, two on topics covered by CPR assignments

• 3 CPR assignments given between midterm and final

Environmental Chem Grades for Midterm and Final Exam Questions

0.0

5.0

10.0

15.0

20.0

25.0

30.0

35.0

40.0

45.0

50.0

Particulates

Ozone from NOx

AerosolsPAH's

Bioaccumulation

DioxinAcid Rain

PCB's DDT

Grade

Midterm:Non-CPR Final:Non-CPR Topics Final:CPR topics

Three CPR assignments completed after midtermOnly text entry for one CPRassignment finishedbefore midterm.

Midterm Final

average

average

average

Evaluation of CPR-Reviewing Skill

• Ten courses, taught by 8 instructors, from 8 different institutions, for 5 different subjects

• Paired t-test comparison of students’ ability to review assignments at beginning of term with performance on last assignment in course.

Change in Reviewing Skills for Students in 10 Different Courses, in 8 Different Schools(Chemistry: AP, Prep, Gen (4), Organic, Environmental; Remedial English; Economics)

0

1

2

3

4

5

6

AP* fresh fresh fresh* fresh fresh soph senior fresh soph

Course Level(* Not statistically significant difference)

RCI

(reviewer competency index)

first assignment last assignment

Peer Review FeedbackRating: 4; Explanation: You have the general concepts of base peak and molecular ionpeak down, but your explainations of how you obtain these peaks is incorrect. First, whenyou determine the molecular ion peak, you add the weights of the ions that were givenwith the problem (i.e. C=12, not 12.01, H=1, not 1.01, etc.). Hence, your molecular ion'sweight will be 84, not 84.93. Although your concept is correct, your calculation is wrong.Also, your explanation of the 86 m/z and 88 m/z readings is very difficult to follow. Younever directly tie the 86 m/z reading to CH2Cl35Cl37+.. You r explaination of thediffering levels of 86 m/z and 88 m/z is good in that you correctly state that it has to dowith the ratios that the isotopes occur in nature. The details of your explaination,however, are incorrect. The fact that there are two Cl37 atoms which have a 25% chanceof occuring in nature present in the molecule LOWERS the chance of that molecule beingpresent, since there are two "one in four" chances that have to be overcome instead of justone in the case of only one Cl37 atom being present. You do not add the 25% chancestogether and get a 50% chance. Also, you have a few spelling errors and you neglect toput the "plus radical (+.) after each ion. You a lso don't have a summary sentence.Additionally, you lack an explaination of why low pressure upheld in the machine (it isdone to reduce the amount of intermolecular processes). Overall, your generalunderstanding is good, but you need to work on the details.

Self Reviewing Skill

0

10

20

30

40

50

60

70

80

90

100

1 2 3 4 5

Text Quality

% Self-Assessing Accurately

W01 <6 W01 >6 Sp01 <6 Sp01 >6

Peer Review FeedbackRating: 6; Explanation: Small grammatical errors: -"rules can be bend if necessary" - "the rights ethics" -"Duty ethics focuses on that each person has a duty..." -"many similarity exist" - "When such incident arises" - "Iworked night shift " Fragments: "While in rights ethics,each person has the rights to life. " note: Only John StuartMill is not recognized. Also, no introduction at all!RATING: I gave this writer a 6 because I feel that thetheories were clearly described, despite the vast amount ofgrammatical errors. There was no introduction to thismemorendum at all. This is not an English class so I can'tmark down that much for these errors; however, I feel itimperative for engineers to be able to clearly write asentence in correct English. How else are they to writeproposals, reviews and test reports, etc. in the corporateworld, where English is the international language ofbusiness. By the way, very good explanation of thedilemma.

Introductory Organic Chemistry• Research University

– lecture three times/week

– no discussion

– separate lab course

– most students “premed”

• one CPR assignment: Study of Cycloalkanes– uses CHIME interactive tutorial to study ring strain

• given as extra credit assignment at end of quarter• feedback e-mail to instructor required for credit

Introductory Organic ChemistryStudent Buy-In

• 219 of 372 students did the assignment and submitted comments!

Student Perceptions

Student Comments Technical Issues

• “The chime plugin was a little problem because I too use netscape 6.1 so I ended up using Internet Explorer 6.0. It handled the plugin well and did not give me any problems. The web site performed decently with a 56K modem connection, except the chime illustrations took about 30sec to 45sec to load but acceptable. Now that the calibrations are available the calibration screens seem very user friendly but I would tell everyone that it is best to use a screen resolution of at least 800x600 because there is so much to info to display. The questions take half the screen and the essay takes up the other half.”

Source Material/Reviewing• “I thought that the CPR was really interesting. I liked the fact that

they made us look at 3D models before we could answer the questions because it gave us a good idea of the concept involved in the question. I think that the process we had to take before writing the essay was great because it took us step by step to the answer, allowing us to thoroughly comprehend the strain (angle and torsional) in cycloalkanes in comparison to their alkane counterparts and to other larger or smaller cycloalkanes…. I also like that we will get to grade other students’ essays and later see if our own essays were of good quality. I believe that it will let us know if we really understand the part that strain plays in cycloalkanes.”

Workload, Course Value• “This program is similar to something that my girlfriend is doing in

her Christian theology class at BIOLA. She writes a 400-word response to a "poll" question based on the 400-500 page reading assignment that week, every week. Then she reviews the work of three others (but not her own), followed by an in class discussion once a week. Of course, an in-class discussion is not appropriate for a chemistry course and the material covered requires more work than a simple read and response. This is why I feel that the CPR program would be more beneficial if it were assigned more than once a quarter. Something more like one assignment every 2-3 weeks. I really feel that this would improve the overall information retention of the class.

Writing

• By writing this assignment in a mini-essay form I was able to come at my own conclusions and express the material in my words. I believe that this was very helpful because in order to understand the material it is extremely useful to be able to describe and put the material one learned into his/her own words

Workload, Course Value

• I enjoyed the CPR assignment. I liked having another format to think about organic chemistry. If you do this again, I think optical rotation would be a great subject. I thought the writing assignment was a good length--not too much to read six examples of writing

Workload, Value, Reviewing

• I thought the idea of CPR was a very good idea because it helped expanding students' knowledge and developing insight into the materials we had studied. However, I didn't really enjoy this project. I didn't like it because there was too much research involved and questions were very hard to answer. This project itself was time consuming as well. It took me a long time to finish the practice before I moved onto the actual project. Another time-consuming part was the peer reviewing. It took me about an average of 10 minutes each to review one paper. It required a lot of concentration as well

Course Value

• I thought CPR to be good learning tool. The project made the participants review the course material. As a result a better understanding of the material is attained…. It’s like having two lectures instead of one. Making an environment where students have to think is a good way to make the material understandable.

Reviewing, Course Value

• Truthfully, I enjoyed CPR. Writing the essay was a great way to  review the specifics of cyclohxane strain, but when I reviewed the  peer essays I think I learned most. I was able to see what strong  points my essay had, and more importantly where I was lacking. I  think CPR can be a very useful tool for students to use if they take  it seriously and take the time to explore the website. Also, it is  another chance for points, which is always good. By the way, thanks  for the extra credit.

Timing, Workload

• this project was sort of like high school work. it was only  assigned to keep me busy. first of all i don't have a computer so i  had to go out of my way to use one. we studied cyclohexanes for the  first midterm. it would've been a lot better if it was material from  the current chapters.

Word Count, Technical Issues

• I just wanted to give you my opinion of cpr. I felt like it  was to much of a hassle. It was hard to log onto and frustrating to  deal with. When I logged on and began writing my text the HTML tutor  wasn't accessible so I didn't know the editing commands. I know that  we were supposed to be concise, but I felt that 230-390 words just  wasn't enough to truly answer the questions and write a good essay.  Anyways, I didn't think cpr was really worth it because it was to  much of a hassle.

• .just submitted my CPR essay. All in all it isn't that bad.  It's nice and short so that I ended up editting a lot of what I wrote  so that I reached the maximum length and that's better than trying to  think of crud to insert here and there to make a nice long essay.  After the hurdle of trying to sit down and writing something about  ochem, the experience itself is relatively brief. It would have been  nice to know who our target audience is. I know that other peers are  reading this, but for an example, are we supposed to clearly explain  what eclipsing bonds means? Initially I did what it is, but there  isn't enough word allowance for it so I took it out. The bad thing I  don't like is that the topic is a bit late. I wouldn't mind covering  a subject that is supposed to be covered in the next few weeks but I  think writing a paper about something we had covered for the first  midterm is a waste of time.

Foreign Students

• The calibrated peer review helped me to review many knowledges about the conformation of cyclohexane. I learned a lot from it

Technical Difficulties and Attitude to CPR

Positive NegativeComment Comment

Mentioned problems with CHIME 59% 41% (&/or UCR and CPR login)

No technical difficulties 82% 18%mentioned

• Integrates telecommunications and technology into instructional processes so that students learn to

explore ideas independentlywrite about chemistrycollaboratework to a mastery leveltake responsibility for their own learning

http://cpr.molsci.ucla.edu

Institutions with CPR Accounts

Sep-97 Sep-98

Jul-00

Aug-01

Sep-02

Jan-03

Jul-03

0

50

100

150

200

250

300

350

400

Feb-97 Jun-98 Oct-99 Mar-01 Jul-02 Dec-03

Date

Number

Courses Using CPR

Sep-97 Dec-98

Jul-00

Aug-01

Jul-02

Jan-03

Jul-03

0

200

400

600

800

1000

1200

1400

1600

Feb-97 Jun-98 Oct-99 Mar-01 Jul-02 Dec-03

Date

Number

Cumulative Student Accounts

Sep-97Dec-98

Jul-00

Aug-01

Jul-02

Sep-02

Jan-03

Jul-03

0

10000

20000

30000

40000

50000

60000

Feb-97 Jun-98 Oct-99 Mar-01 Jul-02 Dec-03

Date

Number

Acknowledgements

Funding sources• National Science Foundation

• Howard Hughes Medical Institute

• University of California, Office of the President

• University of California, Los Angeles

Acknowledgements

Colleagues• Orville L. Chapman• Tim Su• Krista Motschiedler Brand• Stephen Schimpf (Michael Fiore)• authors• students

Thank you