assessing quality of student

7/28/2019 Assessing Quality of Student

1/13

Assessing the Quality of Student Learning in Multimedia-Supported Project-based

Learning

Symposium ProposalAERA 2000

Creating Knowledge in the 21st

Century: Insights from Multiple Perspectives

Symposium Participants

Barbara Means, SRI International, CHAIR

Michael Simkins, Joint Venture: Silicon Valley

Gail Britt, Central Middle School, San Carlos, CAOtak Jump, Ohlone Elementary School, Palo Alto, CA

Karen A. Cole, Institute for Research on Learning

William R. Penuel, SRI InternationalRichard Lehrer, University of Wisconsin-Madison, DISCUSSANT

Objective of the Symposium

In this symposium we will present diverse approaches to addressing the problem

of how to measure the quality of learning in classrooms where students design

multimedia products in the context of project-based learning. We share studiesconducted as part of the evaluation of the Challenge 2000 Multimedia Project, a

federally-funded Technology Innovation Challenge Grant now in its fourth year of

implementation in California's Silicon Valley region. Through our symposium, we aim toprovide promising examples of assessment and evaluation practices that can both help

teachers improve student learning and provide researchers and policy makers with

knowledge of how the effective use of multimedia technology can impact studentachievement.

Our symposium draws upon three perspectives: those of program director,

classroom teacher, and professional researcher. The program director will present theprocess by which researchers and project staff jointly developed and implemented a

rubric for measuring the quality of student projects exhibited at annual multimedia fairs

sponsored by participating teams of schools. Two teachers will describe classroom

assessment strategies they developed in partnership with the Institute for Research onLearning (IRL), and a researcher from IRL will discuss the impact of these assessment

practices on teaching and learning. Two researchers from SRI International will discuss

the development of a performance assessment and scoring rubric used to measure theimpact of Multimedia Project participation on classroom learning.

Significance of the Problem

For many years, educators have noted the deep learning that can happen when

students do long projects together.Learning through projects was an important part of

Progressive Education in the early part of the twentieth century. The Project Method, as itwas sometimes called (Kirkpatrick, 1918), was seen as a tool for engaging students in

systematic, student directed inquiry. Dewey (1997) in particular believed that various

crafts, including the graphic arts, afford great opportunity for training in self-reliant and


2/13

efficient social service (p. 168) and provide students with opportunities to encounter

problems that require students to reflect and experiment to solve them.

As the popularity of student-centered approaches to teaching and learning hasgrown in recent years, projects have again become the focus of attention of educators and

researchers. Todays examples of project-based learning are similar to earlier practices in

that they require students to focus over an extended period of time on the resolution of areal-world problem (Blumenfeldet al., 1991). Typically, through the course of

completing a project, learners use multiple sources, collaborate with others, and apply

cognitive tools to plan, conduct and evaluate possible solutions to the problem at hand. In

this way, projects provide an opportunity for students to develop deep understanding ofsubject matter as they acquire new information and concepts and apply this new

knowledge to a production task.

Multimedia technology is a potentially powerful aid for structuring project-basedwork to engage student learning. Multimedia products create artifacts for teachers and

students that can become the basis for ongoing reflection and critique, helping students to

develop higher standards for their work over time. For example, Allen and Pea (1992)

traced the joint construction of a set of expectations for student learning by teachers andstudents over the course of an extended student project involving the construction of

multimedia presentations. They documented the ways that teachers and students

negotiated a balance between a focus on design and content, and came to see the twoelements of constructing a presentation as interdependent. In a similar study of project-

based learning using multimedia, Erickson and Lehrer (1998) researched the evolution of

critical standards for judging the quality of student work. Over the course of two yearsinvolvement in projects, students and teachers came to develop shared representations of

what constitutes a good project.

In the two case studies cited above, multimedia technology alone was notsufficient to support student learning. In both these projects, teachers played a critical

role by fostering and guiding student reflection about the quality of student work, both as

it progressed and as students developed final multimedia products. Teachers usedongoing assessment to help drive student collaboration and learning. In our analyses of

student learning in the Challenge 2000 Multimedia Project, we provide additional

examples of assessment tools that can support students' own developing understanding of

what constitutes a quality multimedia project. We also go beyond previous case studiesof project-based learning using multimedia by considering the ways that formative

classroom assessment practices can be coordinated with program design and evaluation,

so that the program design can be continuously improved and knowledge of effectivepractices can be distributed widely to researchers and policy makers.

Symposium Presentations

Michael Simkins, Joint Venture: Silicon Valley Network, will describe the

overall program design and present the tool and method developed to judge the quality of

student multimedia projects exhibited at annual, end-of-year fairs. These fairs were acritical component of the program design and were intended to motivate completion of

projects, reward student and teacher effort, and foster community awareness of the ways

in which multimedia technology can be applied in the classroom. Working together,


3/13

teachers, evaluators, and project staff created a scoring rubric based on the model of

project-based learning espoused by the Challenge 2000 Multimedia Project. The rubric

allowed for projects to be rated on a five-point scale in each of three dimensions: content,collaboration, and multimedia. Scoring was done by panels of college-educated lay

judges following 30-minute student interviews. Project scores were used as one source of

data for measuring progress of the Challenge 2000 Multimedia Project toward its statedgoal of infusing the schools of Silicon Valley with an exemplary model of project-based

learning supported by multimedia. In the two years the scoring process has been used,

the proportion of projects rated "exemplary" has increased dramatically, from 22% to

50%.

Gayle Britt, Central Middle School in San Carlos, CA, will describe assessment

strategies she developed in partnership with the Institute for Research on Learning aimedat helping students judge the quality of HyperStudio stacks they created as part of a

project focused on Chinese dynasties. Initially, Britt helped students create a checklist to

judge the quality of their work and found that when students revised their work using this

checklist, their comments focused almost exclusively on design issues and not on content.Britt worked with her class again to develop a set of content-focused criteria for judging

the quality of their stacks. With the new student-generated checklist, students realized

they knew more than they had incorporated into their stacks and recognized the need toincorporate more content into their stacks. In a second assessment laboratory, Britt

worked with SRI researchers who were conducting a performance assessment in select

Multimedia Project classrooms. Students were asked to develop criteria for assessing thequality of brochures they and other students had created as part of the assessment task.

Students watched a video of domain content and design experts from the field before

generating a first set of criteria. After scoring one brochure, students revised theirchecklist, reviewed the original instructions for the task, and scored a second brochure.

Britt reports that through this process, students learned to become more specific in

generating their checklists and to develop an appreciation of how to use the standards in arubric or checklist, rather than their first impressions, as the basis for judging the quality

of brochures.

Otak Jump, Ohlone Elementary School in Palo Alto, CA, will describe a processhe developed to help students reflect on their skill in collaboration as they participate in

computer-supported project-based learning activities. The process involves a group

discussion about the types of behaviors that will help reach the particular learning goalsfor the project. Students then begin work, and while they are working the teacher records

short video segments showing students doing whatever they are doing. Either at the end

of the session or at the beginning of the next class period, the group watches five minutesof video, and Jump points out specific behaviors that he wants students to learn from and

solicits student comments about the behaviors they have seen. During this debrief, the

question of behavior is framed as to which behaviors "helped" the group achieve their

learning goals and which behaviors "hindered." The discussion focuses on describing thebehaviors, not the people involved in doing the behaviors. Jump employed this process

throughout the student multimedia projects this year and worked with the Institute on


4/13

Learning to document the effects of these strategies on the collaborative behavior of

students.

Karen A. Cole, Institute for Research on Learning, will outline the results of her

study of the effects of the assessment strategies in Britts and Jumps classrooms. Cole

found that ongoing, future-oriented formative assessment was effective improvingstudent learning and extending teacher practice. Assessment was particularly effective in

helping to solve problems particular to student-designed multimedia projects. These

problems include learning enough academic subject matter, developing language for

assessing quality in the new media, and helping students take advantage of thetechnology (rather than producing an on-line term paper). Video analysis of student work

sessions and assessment events, combined with analysis of multiple revisions of students'

multimedia projects, allowed us to trace how assessment events affected student workand assessment practices. Student projects became more content-rich as assessment

progressed, and project design elements became more content-appropriate. Students

developed language for talking about the quality of projects, and changed they way they

evaluated a project's quality. As assessment progressed, they became more focused onparticular qualities of a project and less focused on numerical scores. Teachers changed

their view of assessment's function from measurement to supporting and enhancing

learning.

William R. Penuel andBarbara Means, SRI International, will describe the

design, implementation, and results from a performance assessment aimed at measuringstudent problem solving skill in constructing products like the ones created by students in

the Challenge 2000 Multimedia Project. As part of their evaluation of the Multimedia

Project, Means and Penuel set out to design a task that would be able to compare theperformance of middle-school students in classrooms with experienced Multimedia

Project teachers with middle-school students in classrooms where teachers used little

technology or project-based learning methods. Two key constraints were addressed inthe design: comparison classrooms lack of access to technology, and the need to provide

content to classrooms that represented a number of core subject areas. SRI researchers

worked in partnership with classroom teachers and their students, in particular Gayle

Britt, to develop the outlines of a rubric that was later refined by researchers for scoring.Results showed that Multimedia Project students evidenced greater mastery of content,

more sensitivity to their audience, and better design skills. The researchers will discuss

the implications of the findings and plans for sharing knowledge of the programseffectiveness beyond the projects teachers.

Symposium Format

The chair will provide a brief overview of the session, outlining the theoretical

and practical significance of the session. Each speaker will be given 10-15 minutes per

presentation. The discussant will take 10-15 minutes to provide comments on each of thepapers, and the remaining time for the session will be devoted to questions from the

audience.


5/13

References

Allen, C., & Pea, R. (1992). The social construction of genre in multimedia

learning environments. Menlo Park, CA: Institute for Research on Learning.

Blumenfeld, P.C., Soloway, E., Marx, R.W., Krajcik, J.S., Guzdial, M., &Palincsar, A. (1991). Motivating project-based learning: Sustaining the doing, supporting

the learning.Educational Psychologist, 26, 369-398.

Dewey, J. (1997). How we think. Mineola, NY: Dover.

Erickson, J., & Lehrer, R. (1998). The evolution of critical standards as students

design hypermedia documents. Journal of the Learning Sciences, 7, 351-386.

Kirkpatrick, W. H. (1918). The Project Method. Teachers College Record. Also

reprinted in Schultz, F.(Ed.) Sources: Notable Selections in Education. Guilford, CT: The

Dushkin Publishing Group.,Inc. pp.26 - 33.


6/13


Learning

Symposium ProposalAERA 2000

Creating Knowledge in the 21st

Century: Insights from Multiple Perspectives

Symposium Participants

Objective of the Symposium

In this symposium we will present diverse approaches to addressing the problem

of how to measure the quality of learning in classrooms where students design

multimedia products in the context of project-based learning. We share studiesconducted as part of the evaluation of the Challenge 2000 Multimedia Project, a

federally-funded Technology Innovation Challenge Grant now in its fourth year of

implementation in California's Silicon Valley region. . Through our symposium, we aim

to provide promising examples of assessment and evaluation practices that can both helpteachers improve student learning and provide researchers and policy makers with

knowledge of how the effective use of multimedia technology can impact student

achievement.Our symposium draws upon three perspectives: those of program director,

classroom teacher, and professional researcher. The program director will present the

process by which researchers and project staff jointly developed and implemented arubric for measuring the quality of student projects exhibited at annual multimedia fairs

sponsored by participating teams of schools. Two teachers will describe classroom

assessment strategies they developed in partnership with a national research institute, anda researcher from this institute will discuss the impact of these assessment practices on

teaching and learning. Two researchers from another national research institute will

discuss the development of a performance assessment and scoring rubric used to measurethe impact of Multimedia Project participation on classroom learning.

Significance of the Problem

For many years, educators have noted the deep learning that can happen when

students do long projects together.Learning through projects was an important part of

Progressive Education in the early part of the twentieth century. The Project Method, as itwas sometimes called (Kirkpatrick, 1918), was seen as a tool for engaging students in

systematic, student directed inquiry. Dewey (1997) in particular believed that various

crafts, including the graphic arts, afford great opportunity for training in self-reliant andefficient social service (p. 168) and provide students with opportunities to encounter

problems that require students to reflect and experiment to solve them.

As the popularity of student-centered approaches to teaching and learning has

grown in recent years, projects have again become the focus of attention of educators andresearchers. Todays examples of project-based learning are similar to earlier practices in

that they require students to focus over an extended period of time on the resolution of a

real-world problem (Blumenfeldet al., 1991). Typically, through the course of


7/13

completing a project, learners use multiple sources, collaborate with others, and apply

cognitive tools to plan, conduct and evaluate possible solutions to the problem at hand. In

this way, projects provide an opportunity for students to develop deep understanding ofsubject matter as they acquire new information and concepts and apply this new

knowledge to a production task.

Multimedia technology is a potentially powerful aid for structuring project-basedwork to engage student learning. Multimedia products create artifacts for teachers and

students that can become the basis for ongoing reflection and critique, helping students to

develop higher standards for their work over time. For example, Allen and Pea (1992)

traced the joint construction of a set of expectations for student learning by teachers andstudents over the course of an extended student project involving the construction of

multimedia presentations. They documented the ways that teachers and students

negotiated a balance between a focus on design and content, and came to see the twoelements of constructing a presentation as interdependent. In a similar study of project-

based learning using multimedia, Erickson and Lehrer (1998) researched the evolution of

critical standards for judging the quality of student work. Over the course of two years

involvement in projects, students and teachers came to develop shared representations ofwhat constitutes a good project.

In the two case studies cited above, multimedia technology alone was not

sufficient to support student learning. In both these projects, teachers played a criticalrole by fostering and guiding student reflection about the quality of student work, both as

it progressed and as students developed final multimedia products. Teachers used

ongoing assessment to help drive student collaboration and learning. In our analyses ofstudent learning in the Challenge 2000 Multimedia Project, we provide additional

examples of assessment tools that can support students' own developing understanding of

what constitutes a quality multimedia project. We also go beyond previous case studiesof project-based learning using multimedia by considering the ways that formative

classroom assessment practices can be coordinated with program design and evaluation,

so that the program design can be continuously improved and knowledge of effectivepractices can be distributed widely to researchers and policy makers.

Symposium Presentations

The first presenter will describe the overall program design and present the tool

and method developed to judge the quality of student multimedia projects exhibited at

annual, end-of-year fairs. These fairs were a critical component of the program designand were intended to motivate completion of projects, reward student and teacher effort,

and foster community awareness of the ways in which multimedia technology can be

applied in the classroom. Working together, teachers, evaluators, and project staffcreated a scoring rubric based on the model of project-based learning espoused by the

Challenge 2000 Multimedia Project. The rubric allowed for projects to be rated on a

five-point scale in each of three dimensions: content, collaboration, and multimedia.

Scoring was done by panels of college-educated lay judges following 30-minute studentinterviews. Project scores were used as one source of data for measuring progress of the

Challenge 2000 Multimedia Project toward its stated goal of infusing the schools of

Silicon Valley with an exemplary model of project-based learning supported by


8/13

multimedia. In the two years the scoring process has been used, the proportion of

projects rated "exemplary" has increased dramatically, from 22% to 50%.

The second presenter will describe assessment strategies she developed in

partnership with the first research institute aimed at helping students judge the quality of

HyperStudio stacks they created as part of a project focused on Chinese dynasties.Initially, this presenter helped students create a checklist to judge the quality of their

work and found that when students revised their work using this checklist, their

comments focused almost exclusively on design issues and not on content. The presenter

worked with her class again to develop a set of content-focused criteria for judging thequality of their stacks. With the new student-generated checklist, students realized they

knew more than they had incorporated into their stacks and recognized the need to

incorporate more content into their stacks. In a second assessment laboratory, thepresenter worked with researchers who were conducting a performance assessment in

select Multimedia Project classrooms. Students were asked to develop criteria for

assessing the quality of brochures they and other students had created as part of the

assessment task. Students watched a video of domain content and design experts fromthe field before generating a first set of criteria. After scoring one brochure, students

revised their checklist, reviewed the original instructions for the task, and scored a second

brochure. The presenter reports that through this process, students learned to becomemore specific in generating their checklists and to develop an appreciation of how to use

the standards in a rubric or checklist, rather than their first impressions, as the basis for

judging the quality of brochures.

The third presenterwill describe a process he developed to help students reflect

on their skill in collaboration as they participate in computer-supported project-basedlearning activities. The process involves a group discussion about the types of behaviors

that will help reach the particular learning goals for the project. Students then begin

work, and while they are working the teacher records short video segments showingstudents doing whatever they are doing. Either at the end of the session or at the

beginning of the next class period, the group watches five minutes of video, and the

teacher points out specific behaviors that he wants students to learn from and solicits

student comments about the behaviors they have seen. During this debrief, the questionof behavior is framed as to which behaviors "helped" the group achieve their learning

goals and which behaviors "hindered." The discussion focuses on describing the

behaviors, not the people involved in doing the behaviors. The presenter employed thisprocess throughout the student multimedia projects this year and worked with the

Institute on Learning to document the effects of these strategies on the collaborative

behavior of students.

The fourth presenter will outline the results of her study of the effects of the

assessment strategies in the second and third presenters classrooms. The researcher

found that ongoing, future-oriented formative assessment was effective improvingstudent learning and extending teacher practice. Assessment was particularly effective in

helping to solve problems particular to student-designed multimedia projects. These

problems include learning enough academic subject matter, developing language for


9/13

assessing quality in the new media, and helping students take advantage of the

technology (rather than producing an on-line term paper). Video analysis of student work

sessions and assessment events, combined with analysis of multiple revisions of students'multimedia projects, allowed us to trace how assessment events affected student work

and assessment practices. Student projects became more content-rich as assessment

progressed, and project design elements became more content-appropriate. Studentsdeveloped language for talking about the quality of projects, and changed they way they

evaluated a project's quality. As assessment progressed, they became more focused on

particular qualities of a project and less focused on numerical scores. Teachers changed

their view of assessment's function from measurement to supporting and enhancinglearning.

The fifth and sixth presenters will together describe the design, implementation,and results from a performance assessment aimed at measuring student problem solving

skill in constructing products like the ones created by students in the Challenge 2000

Multimedia Project. As part of their evaluation of the Multimedia Project, the presenters

set out to design a task that would be able to compare the performance of middle-schoolstudents in classrooms with experienced Multimedia Project teachers with middle-school

students in classrooms where teachers used little technology or project-based learning

methods. Two key constraints were addressed in the design: comparison classroomslack of access to technology, and the need to provide content to classrooms that

represented a number of core subject areas. Researchers worked in partnership with

classroom teachers and their students, in particular the second presenter, to develop theoutlines of a rubric that was later refined by researchers for scoring. Results showed that

Multimedia Project students evidenced greater mastery of content, more sensitivity to

their audience, and better design skills. The researchers will discuss the implications ofthe findings and plans for sharing knowledge of the programs effectiveness beyond the

projects teachers.

Symposium Format

The chair will provide a brief overview of the session, outlining the theoretical

and practical significance of the session. Each speaker will be given 10-15 minutes perpresentation. The discussant will take 10-15 minutes to provide comments on each of the

papers, and the remaining time for the session will be devoted to questions from the

audience.

References

Allen, C., & Pea, R. (1992). The social construction of genre in multimedia

learning environments. Menlo Park, CA: Institute for Research on Learning.

Blumenfeld, P.C., Soloway, E., Marx, R.W., Krajcik, J.S., Guzdial, M., &

Palincsar, A. (1991). Motivating project-based learning: Sustaining the doing, supporting

the learning.Educational Psychologist, 26, 369-398.


10/13

Dewey, J. (1997). How we think. Mineola, NY: Dover.

Erickson, J., & Lehrer, R. (1998). The evolution of critical standards as students

design hypermedia documents. Journal of the Learning Sciences, 7, 351-386.

Kirkpatrick, W. H. (1918). The Project Method. Teachers College Record. Also

reprinted in Schultz, F.(Ed.) Sources: Notable Selections in Education. Guilford, CT: The

Dushkin Publishing Group.,Inc. pp.26 - 33.


11/13

List of Participants

Means, Barbara, SRI International, 333 Ravenswood, Menlo Park, CA 94025

Simkins, Michael, JVSV, 99 Almaden Blvd #700, San Jose, CA 95113Gail B, Central MS, 828 Chestnut Street,San Carlos, CA 94070Otak C, Ohlone ES, 950 Amarillo, Palo Alto, CA 94303Karen A. D, IRL, 66 Willow Pl, Menlo Park, CA 94025

Penuel, William R., SRI Intl, 333 Ravenswood, Menlo Park, CA 94025

Richard Lehrer, WCER, 1025 W. Johnson St., Madison, WI, 53705


12/13

List of Participants

Means, Barbara, SRI International, 333 Ravenswood, Menlo Park, CA 94025

Simkins, Michael, JVSV, 99 Almaden Blvd #700, San Jose, CA 95113

Gail B, Central MS, 828 Chestnut Street,San Carlos, CA 94070Otak C, Ohlone ES, 950 Amarillo, Palo Alto, CA 94303

Karen A. D, IRL, 66 Willow Pl, Menlo Park, CA 94025

Penuel, William R., SRI Intl, 333 Ravenswood, Menlo Park, CA 94025

Richard Lehrer, WCER, 1025 W. Johnson St., Madison, WI, 53705


13/13

Information for Index card:


Learning

Organizer: Penuel, William R.SRI International

333 Ravenswood, Mailstop BS116

Menlo Park, CA 94025

(650) 859-5001

Chair: Means, Barbara

SRI International333 Ravenswood

Menlo Park, CA 94025

(650) 859-4004

assessing quality of student

Documents