feedback and presentation in the microsoft distributed classroom project richard anderson university...

Feedback and Presentation in the Microsoft Distributed Classroom

ProjectRichard Anderson

University of Washington and Microsoft Research

Richard Anderson UW & MSR 2

t

Overview

• Presentation system– Develop system for lecture delivery that

increases instructor flexibility

• Feedback system– Develop system that supports increased in

class communication

• Design experiment– Experiment with system in actual classroom

Richard Anderson UW & MSR 3

t

Educational Technology

…in the winter of 1813-1814 … I attended a mathematical school kept in Boston…on entering his room, we were struck at the appearance of an ample Blackboard suspended on the wall, with lumps of chalk on a ledge below, and cloths hanging at either side. I had never heard of such a thing before. [Mr. May, 1866]

Richard Anderson UW & MSR 4

t

Classroom Technology

• Black board

• White board

• Overhead projector

• Transparencies– Persuasion, PowerPoint, LaTeX

• PC with data projector

Richard Anderson UW & MSR 5

t

Research goal

• Develop presentation system for PC/Data Projector– Increase flexibility in lecture delivery– Support greater communication

• Design for instructional use at the university level

• Evaluate based on pedagogical impact

Richard Anderson UW & MSR 6

t

Scenarios

• Distributed classroom– Multi site video conferenced classes– Data projectors in each room– Possible desktop access

• Lecture hall– Public projected display– Wireless enabled class– Possible student devices

• Laptop, tablet, handheld

Richard Anderson UW & MSR 7

t

Teaching styles

• Wide range of teaching styles and opinions• Instructors will use technology in different ways• Instructors will use technology in unexpected

ways• We are not comparing between different

technologies– PC delivery already established– Some situations require PC delivery

Richard Anderson UW & MSR 8

t

DISC Project

• Internet based video conferenced classes

• Multicast technology to support multiple sites

• Deploy system on a PC

• Simple startup (the big red button)

Richard Anderson UW & MSR 9

t

Preliminary Study

• UW Professional Master’s Program– Video conferenced class– Netmeeting for PowerPoint distribution– Smartboard ™ Electronic whiteboard

• Instructor interviews

• Student interviews

• Classroom observations

• Review of video archive (5 years of data)

Richard Anderson UW & MSR 10

t

Study conclusions

• Need to support handwriting and slides• Current electronic whiteboards not

adequate• Important to integrate whiteboard and

slides– Every observed use of whiteboard was in the

context of a slide

• Use pen based device as presenter console

Richard Anderson UW & MSR 11

t

Electronic whiteboard comments

• "The whiteboard would be better if: both it and the slides could be seen at the same time the handwriting was captured more accurately; if I write in anything other than slowly-drawn block letters, it looks like a heap of pick-up sticks."

• "The problem with the whiteboard is that, unless you draw very carefully, half of it won't come out, and similarly when you erase. This is enough of a hassle that I wound up not using it much."

Richard Anderson UW & MSR 12

t

PowerPoint comments

• "I could do some of what I wanted with PowerPoint animations, but these took a huge amount of time to prepare. Previously I had used slide overlays (transparencies) a lot, frequently presenting a slide that was partially filled in (some background and a question) and then using class discussion to fill it in."

• "PowerPoint is also a pain for the same reason it's a pain in non-distance classes - the slides impose a rigid structure on the lecture, and make it more difficult to adjust to the interactions that occur during it. "

Richard Anderson UW & MSR 13

t

Presentation system

• Key idea – integration of ink and slides

• Lecture from tablet pc– Mouse input adequate for gesturing, but not

for writing

• Write on PowerPoint

• Synchronized display of slides

• Real time transmission of ink

Richard Anderson UW & MSR 14

t

Implementation

• Export slides as images, broadcast images– Advantages: easy and it worked. Viewer does

not need to have PowerPoint. Easier support of a range of devices.

– Disadvantages• Loss of fidelity• Modifying intermediate form• Loss of animations, transitions

Richard Anderson UW & MSR 15

t

Technical Issue: Slide distribution

• Current approach– Broadcast slide deck using multicast– Interpacket delay introduced to avoid network

saturation– Rebroadcast on request

• Issues– Dealing with clients joining sessions– Large images (instructor’s baby pictures)

• Recognized as a challenge for distributed presentations

Richard Anderson UW & MSR 16

t

Whiteboard features

• Scrollable overlays (virtual mylar)– Simulate overhead projector

• Minimize or hide slide for whiteboard use

• Ink and transparent ink

• Region highlight

• Remote pointer

Richard Anderson UW & MSR 17

t

Classroom use

• Fujitsu tablet

• UW Distance master's class (Professional Master's Program, PMP)– Database– UW Lecturer, MSR classroom– 3 hour evening lecture for 10 weeks

• Several sessions in introductory computer programming

Richard Anderson UW & MSR 18

t

Observation results

• Used for both highlighting and examples

• Lecturers used "dense PowerPoint"– Presentations not designed for ink annotation

• Novelty wore off quickly– Instructors soon became comfortable

• Used from podium

• Some difficulty writing on Fujitsu tablet

• User interface issues to be resolved

Richard Anderson UW & MSR 19

t

Unexpected use

• Alon Halevy: "Could I get volunteers to come up and write their sample queries on the tablet"

• Martin Dickey. TV Talk show style. Took the tablet to the audience, and had them write on it. Wireless tablet.

• Used the device to allow new interaction. Technology to enhance classroom collaboration.

Richard Anderson UW & MSR 20

t

Current experiment

• Use presentation tool in Introductory Computer Programming, Summer 2002

• Evaluate tool in sustained, large lecture use

• Deeper questions– Does the presentation tool enable a better

educational experience?– Does the presentation tool alter the

pedagogy?

Richard Anderson UW & MSR 21

t

Research questions

• Gestures for lecturing from tablet– Specific technology very important– Limited experience with tablet pc– Specific gestures for presentation– Constraints imposed by “performance

environment”

• Opportunities for manipulation of slide content

Richard Anderson UW & MSR 22

t

Feedback System

• Motivation and challenge– Increase level of interaction in large lecture

classes– Give instructor information on level of

understanding of lecture

• Real time feedback– Students annotation of materials on personal

devices– Real time summarization for instructor

Richard Anderson UW & MSR 23

t

Prior work

• Telep (and others)

• Active class (UCSD)

• Class talk (UMass)

• Classroom 2000

• University of Maryland

• Results mixed

Richard Anderson UW & MSR 24

t

Challenges to Interaction in Large Classes

• Student apprehension– Concerns about interrupting class for

“unimportant” questions

• Comment verbalization– Difficulty in expressing comments

• Feedback lag– Questions not formed until topic has been

changed

• Single speaker paradigm

Richard Anderson UW & MSR 25

t

Design Principles

• Non-verbal communication

• Anonymity

• Shared context for feedback

• Rapid, automatic synthesis of feedback

• Simple interfaces

• Value to all users

Richard Anderson UW & MSR 26

t

Important questions

• Will instructors be willing to use system?

• Will instructors be able to take advantage of feedback during lecture?

• Will students be willing to use system?

• Will students benefits to students outweigh the costs of using the system?

Richard Anderson UW & MSR 27

t

Feedback System

• Viewers select annotations from context menus

• Annotations associated with slide geometry

• Real time summary of annotations provided to viewer

• Potential to deploy on a wide range of devices– Take advantage of existing student devices

Richard Anderson UW & MSR 30

t

Feedback categories

• Low impact to make comments• Allow presenter to solicit particular

feedback– State of mind (I’m confused, I understand)– Suggested action (Explain, Elaborate,

Example, Define)– Pacing (Slower, Faster, Just Right)

• Presenter may set categories in a preprocessing phase

Richard Anderson UW & MSR 31

t

Richard Anderson UW & MSR 32

t

Summarization of feedback

• Presenter needs summarization if there is significant feedback

• Summarization done on "semantic regions"– Identify slide geometry - rectangles– Summarize comments in rectangles

• Alternate summarization methods– Intensity or Histogram

• Comment menus also linked to regions

Richard Anderson UW & MSR 33

t

CSE 100 Experiment

• Design experiment• Done in actual class• Understand use and effect in real context• Engineering a learning environment by

introducing an intervention• Study environment to guide design• Conduct pen and paper feedback • Use laptops in class for feedback

Richard Anderson UW & MSR 34

t

Study

• CSE 100: Fluency in information technology• Large, diverse introductory class

– 150 students, large lecture format

• Observations– Very limited student participation– Most students surveyed reported speaking in class

less than once per week– Instructor reported few questions, so she could not

gauge student understanding

Richard Anderson UW & MSR 35

t

Pencil and paper study

• Group of students given paper copies of the slides

• Students asked to annotate slides from predefined categories– Slow Down, Explain, Question, Cool topic

• Simulate student experience

• Evaluate type of feedback delivered

Richard Anderson UW & MSR 36

t

Classroom study

• Limitations:– One class period– Seven laptops (8 of 100+ students

participated in study)– Novelty of system and participation in a study

likely influenced how students used system– Instructor had a TA mediate feedback

(intended use of the system is for instructor to get feedback directly)

Richard Anderson UW & MSR 37

t

Study data

• Logging of system actions

• Classroom observations

• Student questionnaires

• Instructor interviews

• Draw qualitative conclusions from multiple sources of data

Richard Anderson UW & MSR 38

t

System usage

• Students using laptops gave between 1 and 9 annotations

• Students using paper gave between 7 and 15 annotations

Category type

Laptop Paper

Slow Down

2 7

Question 2 21

Explain 12 19

Cool topic 16 13

Richard Anderson UW & MSR 39

t

Instructor response

• Change in presentation style• Altering pace of lecture in response to slow

down comments• Providing additional explanation in response to

question or explain• Examples of terms annotated

– Tracking number – forward reference, clarified later– Batch processing – old term which students weren’t

familiar with

Richard Anderson UW & MSR 40

t

Reactions

• Positive instructor interaction– “[The students] can actually say something

right away, and if the teacher has the wherewithal or time, they can actually make small improvement or small changes that will treat those anxieties or concerns doing what they’re doing that the students like”

Richard Anderson UW & MSR 41

t

Student reactions

• Students completing survey said that the instructor responded to their comments

• “Further explanations were given when asked to”

• “The computer guy [the mediator] told [the instructor] that there was a question and she did respond to it”

Richard Anderson UW & MSR 42

t

Effect on classroom

• Student participation on day of experiment was consistent with previous classes– 6 student initiated interactions on day of

experiment, 11 teacher initiated interactions– Average of 6 student initiated interactions in

earlier classes, 6 teacher initiated interactions

• New style of interaction– Teacher response to feedback creating

discussion

Richard Anderson UW & MSR 43

t

Conclusions

• Positive evidence from design experiment– Students used system– Instructor was able to take advantage of

information– Linking feedback to content was important– Selected categories appropriate– Positive response from instructor and

students– Level of interaction did not drop

Richard Anderson UW & MSR 44

t

Future work

• System development

• Larger field tests– Introductory computing, summer 2002– Direct instructor use of system– Sustained use of feedback system

• Feedback infrastructure as platform for “structured interaction”

Richard Anderson UW & MSR 45

t

Acknowledgments

• UW– Tammy Vandergrift, Steve Wolfman, Ken

Yasuhara

• MSR DISC Project– portal.learningwebservices.com/disc