V A N E S S A D E N N E N

F L O R I D A S T A T E U N I V E R S I T Y V D E N N E N @ F S U . E D U

@ V D E N N E N

A E C T – O C T O B E R 2 0 1 6 – L A S V E G A S , N V

Using electronic diaries for data collection:

An online learning case study

Problem Overview:

Collecting learner habit data

�  Collecting data about online learner habits may prove challenging

�  Discussion board posts are a performance, do not illuminate "backstage online" activities (Gibson, 2014)

�  Analytic data can be useful, but cannot answer all questions

�  Self-report data can fill in gaps �  Common criticism: Self-report

data are unreliable

Proposed solution: Electronic diaries for experience sampling

�  Why not surveys? ¡  Inability to accurately recall earlier

events ¡  Blunt or overly general/summative

�  Why diaries? ¡  Track subjective elements of the user

experience at multiple intervals, as it occurs in the user's natural setting (Scollon, Kim-Prieto & Diener, 2013; Zirkel, Garcia & Murphy, 2015)

Research Questions

�  How do students approach the task of completing electronic diaries?

�  Do students accurately represent their course activities in self-report diaries?

Topical Study Context

�  An examination of student log-in behaviors and weekly activities in an online course

Method

�  Participants (69 consented of approximately 120 in course): ¡  Voluntary ¡  Learners in one intact online class ¡  Undergraduates

�  Data: ¡  Diaries ¡  End-of-course survey ¡  LMS data ¡  Survey log data

Diary Procedures

�  Diaries conducted twice, during Week 5 & Week 10

�  Day 1: Each participant received: ¡  Unique link to diary (editable online survey) ¡  PDF of diary (offline record keeping)

�  Day 7: Reminder �  Day 9: Reminder �  Day 10: Diary closed

Findings: Participation Trends

�  Diary 1 ¡  Started by 64 ¡  Completed by 47

�  Diary 2 ¡  Started by 44 ¡  Completed by 44

�  Diary completion/response rate: 63%

Findings: Diary Use/Typicality

Findings: Reliability Check

�  18 (20%) diaries randomly selected for a check against LMS data ¡  All items that could be verified were: days logged in; days

posted; times posted ¡  Data matching was 98% ¡  Discrepancies occurred in time

Findings: Individual Cases

�  Triangulated data necessary for verification �  Some diaries that might have been discarded based on

log data were accurate �  Example 1: Diary completed/submitted on Day 2 in 3

minutes ¡  Closer look showed valid data ¡  Student completed full diary ¡  Student said he was going to ignore this class and spend the week

working on other classes �  Example 2: Diary completed/submitted on Day 1 in 6

minutes ¡  Closer look showed valid data ¡  Student indicated that she followed a weekly schedule for doing

coursework

Discussion

�  Two main approaches to diary: ¡  Start on Day 1 or 2 and complete over time ¡  Start at end of week and complete all at once

÷ Presumably keeping records throughout week

�  No discrepancies in accuracy, regardless of approach �  Student comments when asked if weeks were typical

were extremely frank, supporting the notion of honest responses

Limitations

�  Participant bias (non-response bias) ¡  Individuals who are more motivated, conscientious or

agreeable may be more likely to complete a diary (Scollon, Kim-Prieto & Diener, 2003)

�  Participants in this study may have been the more reliable and active participants in the course ¡  More difficult to capture data from less active students /

students with poor study habits ¡  Those who do not participate in discussion may be similarly

unlikely to participate in a voluntary study

Conclusion

�  University students are able and willing to provide accurate data about course habits using electronic diaries

�  Diaries can be triangulated with log data to check reliability

�  Other data points may be needed to identify perspectives not included among the diary paticipants

T H A N K Y O U F O R A T T E N D I N G V D E N N E N @ F S U . E D U

@ V D E N N E N S L I D E S A V A I L A B L E A T S L I D E S H A R E . N E T /

V A N E S S A D E N N E N

Questions?

R E F E R E N C E S G I L M O R E , D . M . ( 2 0 1 4 ) . G O F F M A N ' S F R O N T S TA G E A N D B A C K S TA G E B E H AV I O R S I N O N L I N E E D U C AT I O N . J O U R N A L O F L E A R N I N G A N A LY T I C S , 1 ( 3 ) , 1 8 7 - 1 9 0 . S C O L L O N , C . N . , K I M - P R I E T O , C . , & D I E N E R , E . ( 2 0 0 3 ) . E X P E R I E N C E S A M P L I N G : P R O M I S E S A N D P I T F A L L S , S T R E N G T H S A N D W E A K N E S S E S . J O U R N A L O F H A P P I N E S S S T U D I E S , 4 ( 1 ) , 5 - 3 4 . Z I R K E L , S . , G A R C I A , J . A . , & M U R P H Y, M . C . ( 2 0 1 5 ) . E X P E R I E N C E - S A M P L I N G R E S E A R C H M E T H O D S A N D T H E I R P O T E N T I A L F O R E D U C AT I O N R E S E A R C H . E D U C AT I O N A L R E S E A R C H E R . D O I : 1 0 3 1 0 2 / 0 0 1 3 1 8 9 X 1 4 5 6 6 8 7 9 .