Visualizing Gaze Information fromMultiple Students to Support RemoteInstruction

Nancy YaoNorthwestern UniversityEvanston, IL 60208, USAnancyyao2019@u.northwestern.edu

Jeff BrewerNorthwestern UniversityEvanston, IL 60208, USAjeffreybrewer2019@u.northwestern.edu

Sarah D’AngeloNorthwestern UniversityEvanston, IL 60208, USAsdangelo@u.northwestern.edu

Michael HornDarren GergleNorthwestern UniversityEvanston, IL 60208, USAmichael-horn@northwestern.edudgergle@northwestern.edu

Permission to make digital or hard copies of part or all of this work for personal orclassroom use is granted without fee provided that copies are not made or distributedfor profit or commercial advantage and that copies bear this notice and the full citationon the first page. Copyrights for third-party components of this work must be honored.For all other uses, contact the owner/author(s).

Copyright held by the owner/author(s).CHI’18 Extended Abstracts, April 21–26, 2018, Montreal, QC, CanadaACM 978-1-4503-5621-3/18/04.https://doi.org/10.1145/3170427.3188453

AbstractTechnologically-mediated learning environments are be-coming increasingly popular, however remote learning stilllacks many of the important interpersonal features that areleveraged in effective co-located learning. Recent work hasstarted to build in non-verbal cues to support remote col-laboration, such as showing pairs where their partner islooking on the screen. This method of displaying gaze vi-sualizations has been shown to support coordination andlearning in remote collaborative tasks. However, we haveyet to explore how this technique scales to support multiplestudents with one teacher in a technology-mediated learn-ing environment. In this study, we design and evaluate asystem for displaying real time gaze information from multi-ple students to a single teacher’s display during a computerscience studio session. Our results suggest that multiplegaze visualizations can improve the teaching experience inremote settings. Further, we provide design recommenda-tions for future systems based on our preliminary results.

Author KeywordsCollaboration; eye-tracking; learning; gaze visualizations

ACM Classification KeywordsH.5.m [Information interfaces and presentation (e.g., HCI)]:Miscellaneous

IntroductionMillions of students are enrolled in technologically-mediatedlearning programs1 and the ability to learn remotely is in-creasing information accessibility to a broad and diverseaudience. A major drawback of remote learning environ-ments is that the teacher-student relationship becomes im-personal, and teachers are unable to rely on non-verbalcues that normally support communication in a physicalclassroom [1]. These interpersonal cues, such as gazepatterns, play a key role in facilitating communication andrecent evidence suggests they can be integrated in remotesettings [6]. For example, students have been shown to usevisual representations of the teacher’s gaze to help themunderstand content in MOOC style video lectures [10]. Ad-ditionally, gaze information from students can be used tohelp identify if they are falling behind [11]. However, thesestudies investigate displaying gaze information in an asyn-chronous context. Evidence from dual eye tracking studiessuggests that displaying gaze information is useful for sup-porting communication and establishing joint attention inreal time remote work [2, 6, 9, 13]. However, these stud-ies only display gaze information from a single user and wehave yet to explore displaying gaze information from multi-ple students in real time.

Figure 1: Gaze Visualization

Figure 2: Study Setup

Displaying where students are looking in a shared visualspace can provide interpersonal cues to instructors engag-ing in distance education. Expanding on traditional dualeye tracking methods, we developed a system to displaygaze information from multiple students to a single displayfor a teacher in real time. This technique has yet to be in-vestigated, therefore we present preliminary results thatreveal potential benefits of this system in the context oftechnologically-mediated learning. For example, when ateacher is communicating to multiple students at the same

1http://digitallearningcompass.org/

time, displaying each student’s individual gaze positiongives teachers the ability to monitor the students attention.In order to design applications that can effectively incorpo-rate real time gaze information from multiple students, wemust first understand the impact of multiple gaze visualiza-tions on remote instruction.

Related WorkRecent advances in eye tracking and the growth of com-puter based learning allow for an opportunity to integrategaze-based interventions into remote environments. Gazeinformation can provide us with insights into the learner’sattention in computer based activities. For example, fixa-tions tell us what information a student is attending to; thiscan be used to understand if students are following alongin distance learning environments [11, 8]. Thus, visualizingthis information for teachers could help improve instruction.

In educational contexts there has been a growing interest inusing eye tracking technology to improve computer scienceeducation [4]. Most of the work in this space has focusedon helping novices learn from experts by displaying gazeinformation from experts to novices [12]. However, gaze in-formation has also been used to understand how novicesparse code. For example, an eye-tracking study revealedthat novice programmers read code more linearly than ex-pert programmers [3].

In real time studies, displaying gaze information from re-mote pair programmers helped them communicate aboutlocations in the code more effectively [5]. Additionally, dis-playing gaze information between remote students helpedthem understand complex problems [9]. Therefore, display-ing real time gaze information from students to teacherscould help teachers provide targeted instruction in distanceenvironments.

Current StudyIn this study we designed a system for visualizing gaze in-formation from multiple users to a single display and con-ducted a pilot study to understand its potential applicationsfor technology-mediated learning. We evaluated the systemin the context of a remote learning task where a teacherguides multiple students through a coding exercise (seeFigure 3) while receiving gaze information from the stu-dents, visualized using circles illustrating each student’sfixation location in a unique color. We evaluated our designbased on interviews with teachers and feedback from stu-dents. Our results suggest that displaying gaze informationfrom multiple students helps teachers confirm that studentsare following along and monitor the entire class without dis-tracting from instruction. We discuss implications for thedesign of future systems.

Design of Gaze VisualizationsEach student’s gaze is visualized by a circle with a 70 pixelradius (see Figure 1) and is assigned a unique color. Thecolors and student ID are displayed in a key for the teacher.The radius was chosen to compensate for the accuracy ofthe eye trackers while still displaying meaningful locationinformation. Additionally, we reduce shakiness from smallshifts in gaze or microsaccades by only moving the visual-ization when students look to a new location outside of the70px radius. The gaze design also includes a drop shadowvisible when the student’s gaze moves, which is a subtleindicator of the direction of gaze movement (see Figure 1).

Figure 3: Multiple studentvisualization.

Figure 4: Scroll bar visualization.

Additionally, because the coding exercise extends beyonda single screen, participants must scroll to see all of it. Toenable teachers to easily locate where the students arelooking when the object takes up more than just the cur-rent screen, the scroll bar along the side of the screen hascolored indicators indicating the real-time position of each

student, similar to visualization techniques explored by Hillet al. [7] and by D’Angelo and Begel [5]. This provides aglobal view of where all students are looking and allowsteachers to easily click to jump to the location of each stu-dent to see more detailed information (see Figure 4).

MethodsParticipants12 students at a Midwestern university participated in thestudy (2 teachers, 10 students). A "teacher" is a partici-pant with prior knowledge of the material and the solutionsto the exercise. Each teacher was paired with two groupsof two to three students each. A "student" is a participantwho has no prior knowledge of the task but is currently en-rolled in an introductory computer science course. Consentwas obtained prior to the study; students received $20 andteachers received $50 compensation.

ApparatusThe study setup includes four laptops and three Tobii EyeTracker 4C devices (see Figure 2). The three "student"computers are set up with the eye trackers, and the "teacher"computer displays the gaze coordinates received from alleye trackers. The computers are locally networked to re-duce delay in displaying gaze coordinates.

ProcedurePrior to the study, the teachers were given the coding exer-cise and solution key. The teacher was instructed to answerstudents’ questions and provide guidance without reveal-ing the answers. Students were participants with someprior knowledge related to the material (enrolled in an in-troductory C++ course), but no knowledge of the specifictask. Groups consisting of one teacher and two to threestudents engaged in a remote learning task in a pseudo-remote setup. All participants were in the same room, al-

lowing them to communicate verbally. However, to simu-late a remote setting they were not allowed to move andwere seated at different sides of the table to prevent partic-ipants from seeing each others’ screens (see Figure 2). Allscreens showed the same task, a code debugging exercise.

The coding exercise is divided into two parts. The first is10 minutes of instruction, where the teacher explains thecode and related concepts. The concepts, including classinheritance, virtual functions, and inherited member access,are covered in an introductory C++ class that was a pre-requisite for the study. The second part is the 15 minuteexercise, in which students find five bugs in the code. Oncea student locates a bug, they notify the teacher and discusstheir answers with the class before moving on to the nextbug as a group.

One teacher led three sessions. For a baseline, the teacherconducted one session without the gaze visualizations. Inthe second and third sessions, the gaze visualizations werevisible. The second teacher led one session and the gazevisualizations were visible. Note that in all setups, studentscould communicate freely with the teacher and ask ques-tions about the task. Following the session, teachers andstudents participated in a semi-structured interview with theresearch team about their experience.

Figure 5: Example task problemwith solution.

Figure 6: Example of a studentlooking in a different section fromthe group.

Learning ExerciseThe exercise consists of C++ code including class defini-tions for the base class "Animal" and its derived classes, amain function, and expected and actual output (see Figure5). The goal is for the student to find the errors in the codethat would cause the code to produce the actual (incorrect)output when run, rather than the expected. During the ex-ercise, students may ask questions to the teacher, who hasbeen given the solutions in advance.

AnalysisAll sessions were recorded and analyzed by the researchteam to identify common themes. Our analysis focusedon how teachers used the gaze visualization to supportinstruction, how students felt about having gaze visualized,and feedback on the design on the gaze visualizations.

Findings and DiscussionInterviews with teachers revealed that displaying gaze infor-mation presents advantages for assisting teachers duringboth the instruction and individual work phases.

Confirmation/Feedback : Teachers used the gaze visual-izations when giving instructions to confirm that studentswere following along. This was a form of immediate feed-back that allowed teachers to confirm that students were re-sponding to their instructions. For example, when a teacherdirected the students’ attention to a specific portion of thecode, the aggregation of visualizations to that area con-firmed that students were looking in the correct location andthe teacher could begin discussing it, while a wandering vi-sualization could signal a misunderstanding in real time andallow the teacher to clarify immediately.

Monitoring: While students were working independently,teachers used the gaze visualizations to monitor the class.Displaying multiple visualizations allowed them to easilyunderstand the status of the classroom (see Figure 6).Since teachers knew the structure of the task they couldinfer what students were working on and pick up on theirthought processes, as well as confusion or misunderstand-ing. For example, if students were looking in the wrong spotor their gaze was wandering back and forth or incoherently,the teachers knew to speak up and nudge them in the rightdirection. Teachers also commented that it was easier to

monitor the group of students with the visualizations com-pared to walking around to check on each individual.

Not Distracting: In contrast to prior work that suggestsdisplaying real time gaze visualizations is distracting [6],teachers did not find this to be the case, despite receivingvisual feedback from multiple students at once. This maybe because of the nature of the teachers’ role; being al-ready familiar with the task, the teachers only needed tokeep track of students’ progress. They were not engaged incompleting or understanding the task, or coordinating withstudents, especially given that their own gaze was not dis-played. Instead, they devoted their attention to using thevisualizations to confirm and monitor class status, so thevisualizations became a useful tool rather than somethingthey needed to ignore. Thus, teachers found it relativelyeasy to acclimate to the gaze visualizations and were ableto use them effectively to support their teaching role.

Implications for DesignBased on our preliminary results, there appears to be po-tential for visualizing students’ gaze information. There areseveral important implications for designing systems thatdisplay multiple gaze points from students.

Figure 7: Example of gazeclustering.

Figure 8: Example of teacherusing gaze visualizations to givetargeted instruction.

Gaze Clustering: Having students work together allowedfor natural clustering of gaze to relevant areas of code. Thisbehavior was especially prevalent when teachers explaineda piece of code to the group. The clustering made the mul-tiple visualizations manageable and provided a cue if a stu-dent deviated from the group, either lagging behind or readyto move on (see Figure 7). If students were working indi-vidually or on separate problems, it would likely be difficultfor teachers to attend to each student. It could also becomedifficult to keep up with students as their number increases.

Privacy : Students reported that they did not feel uncom-fortable with teachers being able to see where they werelooking. This is promising for similar situations where stu-dents are solving a problem or working through a dense setof code in a group. However, in contexts where studentsaren’t actively working, such as lectures, if students arelooking at personal content they may be less comfortablewith teachers having access to their gaze information.

Scroll Bar Visualization: Teachers reported that the scrollbar visualization was useful when students were attendingto different locations in the code because the full assign-ment spanned much farther than the height of the displayscreens. It provided a global view of all students and a cuefor teachers to monitor the entire class while focusing onindividuals. In particular, teachers noted this as a benefitwhen compared to monitoring a physical classroom, be-cause they could oversee all students’ activities withouthaving to move around to observe them separately, andbased on the path of each students’ gaze, they could infertheir progress on the assignment.

Conclusion & Future WorkEye tracking technology is becoming more affordable, al-lowing for opportunities to collect gaze information frommany users. Understanding how to visualize that informa-tion in meaningful ways is vital. This work presents prelim-inary findings that suggest that visualizing the gaze of mul-tiple students in a technology-mediated learning scenariohelps teachers effectively guide students through instruc-tion and discussion of relevant tasks. For example, teacherscan easily monitor students and detect when extra help isneeded (see Figure 8), despite not having access to thenuanced physical cues of a real classroom, such as facialexpression and body language. Gaze information providesuseful feedback to the teacher without disrupting students.

These encouraging preliminary results should motivate fur-ther research in visualizing gaze information from multipleusers. There are interesting unanswered questions regard-ing how gaze visualizations impact instruction and under-standing of the perspective of the students. Finally, we sus-pect there is a limit to the number of gaze visualizations thatcan be used effectively, after which they become distracting;however, the scroll bar visualization is promising for scalingthis method up to support many students.

REFERENCES1. Kristen Betts. 2009. Lost in translation: Importance of

effective communication in online education. OnlineJournal of Distance Learning Administration 12, 2(2009).

2. Susan E Brennan, Xin Chen, Christopher A Dickinson,Mark B Neider, and Gregory J Zelinsky. 2008.Coordinating cognition: The costs and benefits ofshared gaze during collaborative search. Cognition106, 3 (2008), 1465–1477.

3. Teresa Busjahn, Roman Bednarik, Andrew Begel,Martha Crosby, James H Paterson, Carsten Schulte,Bonita Sharif, and Sascha Tamm. 2015. Eyemovements in code reading: Relaxing the linear order.In Proceedings of the 2015 IEEE 23rd InternationalConference on Program Comprehension. IEEE Press,255–265.

4. Teresa Busjahn, Carsten Schulte, Bonita Sharif,Andrew Begel, and others. 2014. Eye tracking incomputing education. In Proceedings of the tenthannual conference on International computingeducation research. ACM, 3–10.

5. Sarah D’Angelo and Andrew Begel. 2017. ImprovingCommunication Between Pair Programmers UsingShared Gaze Awareness. In Proceedings of the CHIConference. ACM, 6245–6290.

6. Sarah D’Angelo and Darren Gergle. 2016. Gazed andConfused: Understanding and Designing Shared Gazefor Remote Collaboration. In Proceedings of the CHIconfernece. ACM, 2492–2496.

7. William C Hill, James D Hollan, Dave Wroblewski, andTim McCandless. 1992. Edit wear and read wear. InProceedings of the CHI conference. ACM, 3–9.

8. Kari-Jouko Räihä, Oleg Špakov, Howell Istance, andDiederick C Niehorster. 2017. Gaze-assisted remotecommunication between teacher and students. InEuropean Conference on Eye Movements 2017.123–123.

9. Bertrand Schneider and Roy Pea. 2013. Real-timemutual gaze perception enhances collaborativelearning and collaboration quality. International Journalof Computer-Supported Collaborative Learning 8, 4(2013), 375–397.

10. Kshitij Sharma, Sarah D’Angelo, Darren Gergle, andPierre Dillenbourg. 2016. Visual Augmentation ofDeictic Gestures in MOOC videos. In ICLS’16. ACM.

11. Kshitij Sharma, Patrick Jermann, and PierreDillenbourg. 2014. “With-me-ness”: A gaze-measurefor students’ attention in MOOCs. In ICLS’14.

12. Randy Stein and Susan E Brennan. 2004. Anotherperson’s eye gaze as a cue in solving programmingproblems. In Proceedings of the 6th internationalconference on Multimodal interfaces. ACM, 9–15.

13. Roel Vertegaal. 1999. The GAZE groupware system:mediating joint attention in multiparty communicationand collaboration. In Proceedings of the CHIconference. ACM, 294–301.