A Quasi-Experimental Study of the Impact of A Quasi-Experimental Study of the Impact of Digital Audio Recording of Classroom Lectures Digital Audio Recording of Classroom Lectures in a College Technology Classroomin a College Technology Classroom

Zenia C. Bahorski Ph. D.Department of Computer ScienceEastern Michigan University – United StatesJune 29, 2010

Discussion Topics - Discussion Topics - AgendaAgenda

Statement of the Problem Significance of the Problem Research Questions Protection of Human Subjects Pre-Pilot / Pilot Resources Research Design Participants and Sampling Methods Addressing Validity Data Collection / Analysis Hypotheses - Results Implications Suggested Research

2

Statement of the Statement of the ProblemProblem

Recording of Classroom Lectures has become an Accepted Practice: ◦ Faculty recording F2F classroom lectures (audio and/or visual) & putting them online Urged by Universities and Programs (such as UM Dental School)

Urged by Educational Organizations (such as International Society for Technology in Education - ISTE)

Urged by Corporations (such as Apple’s iTunes U) Urged by Faculty themselves!

Are there negative impacts on student learning as a consequence of this accepted practice?

3

Research QuestionsResearch Questions Does the presence of the digital audio recorder have an impact on students’ perceived level of classroom participation (Does it differ from presence of peers or instructor)?

If students’ perceived level of classroom participation is affected, does it change over time?

Will students perceive that the digital audio recordings are useful?

Will students perceive that digital audio recordings are easy to access and use?

Will age or gender affect perceived usefulness or use of the recordings?

Would students make use of the digital audio recordings?

4

Research QuestionsResearch Questions

5

Expansion of the Technology Acceptance Model (TAM) developed by Davis, 1989; Adams, 1992; Hendrickson, 1993; Alshare, 2004

Research QuestionsResearch Questions Hypotheses: H1: There is no difference between perceived student

interaction levels during traditional classroom sessions where a digital recorder is recording the classroom sessions and in those traditional classroom sessions where a recorder is not present.

H2: In classes where the sessions are recorded, students' perceived participation levels will not change over time.

H3: Age will not have an effect on perceived usefulness of the recorded sessions.

H4: Age will not have an effect on perceived ease of use of the recorded sessions.

H5: There is no significant difference between men and women as to the perceived usefulness of recorded sessions.

H6: There is no significant difference between men and women as to the perceived ease of use of the recorded sessions

H7: Students who perceive the recorded classroom sessions as useful are no more likely to use the digital recordings than students who do not perceive the recorded sessions as useful.

H8: Students who perceive the digitally recorded classroom sessions as easy to use are no more likely to use the digital recordings than those who do not perceive the digitally recording as easy to use.

6

Protection of Human Protection of Human SubjectsSubjects

Institutional Research Board Approval:◦Past Practice:

Faculty and Students already recording lectures

No new treatments being performed

◦Participation: Completely voluntary and confidential Students could withdraw from study at any time

◦Informed Consent Letter

7

Pre-Pilot / PilotPre-Pilot / Pilot

Preparing for the Pilot: (Pre-pilot: Su08)◦Survey Instrument Development

Individual Interviews and Small Groups: Given time to complete the survey Each question discussed individually

Did interviewee understand the question? Did the question reflect what was intended?

Adjustments Made (repeat) Also determined time necessary to complete the surveys

In all, 11 Students, 5 Faculty, 2 Administrators (plus IRB)

8

Pre-Pilot / PilotPre-Pilot / PilotPreparing for the Research Study: (Pilot - Fa08)

◦ Selecting appropriate digital audio recording devices Several options: iPod (external mic.); Computer + Audacity (using internal and external mic.); different models of handheld digital recorders (using internal and external mic) – Each was tested

◦ Training faculty in the use of the digital recording device

◦ Acquiring space on the institution’s “secure” podcasting site for each instructor that was recording classes (pilot and for study)

◦ Training faculty in the creation and maintenance of a podcasting site (Created Handout)

◦ Tested when to distribute the survey instruments◦ Tested the time it took to complete the survey instruments

◦ Collecting classroom observational data (wrote everything)

◦ Organizing the data into the SPSS application◦ “Training” students? (Found that students needed a handout) 9

ResourcesResourcesRecording Devices:◦3 Sony ICD - UX80◦2 Sony ECM-DS70P microphones ◦1 Sony ECM-CS10 lapel microphone◦Batteries+Batteries+Batteries!

Incentives:◦6 iPod Shuffles (1 for each class)

Copy Costs◦Surveys◦Observation Charts

10

Participants & Participants & Sampling MethodsSampling Methods

Study took place at:◦Regionally Accredited University of 22,000 students

◦Winter 2009Participants:◦Students enrolled in F2F introductory Computer Science or Technology courses

◦Assume no prior computer experience◦Come from any major or minor

Sampling Method: Purposive◦Selected possible classes all times of day and evening (2 morning, 2 afternoon, 2 evening)

◦Approached Faculty (based on courses, time of day, and willingness) All 6: Tenured Faculty with at least 10 years teaching experience

11

Research DesignResearch Design

Quasi-experimental Design◦2 Groups (107)

3 Classes: Experimental Group (69) Lectures recorded and made available online Received two surveys (one pre and one post)

3 Classes: Control Group (38) Lectures not recorded Received only initial survey (same instrument as experimental group)

◦Pretest - Posttest:

Experimental Group: O X O

Control Group: O - -

12

Addressing ValidityAddressing Validity Face Validity: Two faculty, expert in creating surveys, recruited to provide structure and design feedback

History: Classes selected from all times of the day to reduce likelihood that internal validity would be threatened

Maturation: Study performed during a short time period (maturation changes should not have occurred)

Testing: Selecting of six classes (3 in each group) should have alleviated the threat of the posttest altering results of study

Instrumentation: 11 students, five faculty, 2 administrators to determine clarity and that responses were in relation to constructs

Statistical Regression: Students were not selected based on test score. Control group saw questions once. Results should not have been affected by progressive series of repeated questions

13

Addressing ValidityAddressing Validity Differential Selection of Participants: This was a purposive study (no control over which students enrolled in which classes). Study could not be randomized. No control over who would teach which class.

Mortality: To maintain motivation, incentives were included. One student from each class was randomly selected to win a prize.

Selection-Maturation Interaction: Both groups were observed and surveyed during the same time period. Training faculty in recording class sessions took place at the beginning of the semester. Each faculty member worked with the same set of procedures.

14

Data CollectionData CollectionSurvey 1: (Both Groups-Experimental and Control)◦ Survey 1 distributed during first or second lecture

◦ Response Rate: All who were in attendance decided to participate in the study.

Instrument Inquiry:◦ Demographic Data◦ Previous Computer Knowledge◦ Perceived Level of Participation◦ Perceived Usefulness* (From Technology Acceptance Model)

◦ Perceived Use* (From Technology Acceptance Model)

15

Data CollectionData CollectionDemographic Data:

Age, Gender, Class Standing, ownership of an mp3 player

Previous Computer Knowledge [SA, A, NA/D, D, SD, NE (No Exp.)]:

1. I can use a word processing program on a computer with no trouble.

2. I can use an email program to compose and read email with no trouble.

3. I can send attachments using email with no trouble.4. I can surf the Web with no trouble.5. I can refine a search on the Web with no trouble.6. I can download music to a computer from a CD with no

trouble.7. I can download music to a computer from the Web with no

trouble.8. I can load music onto an MP3 player with no trouble.9. I can connect to eLearning environments such as

eCompanion, or WebCT, with no trouble.10.I can upload files to eLearning environments such as

eCompanion, or WebCT, with no trouble. 16

Data CollectionData CollectionPerceived Level of Participation [SA, A, NA/D,

D, SD]:

11. In general, I feel comfortable speaking up in class when my peers are present.

12. In general, I feel comfortable speaking up in class when the instructor is present.

13. I feel comfortable sharing my research ideas with the class.

14. I feel comfortable speaking in class when a recorder is present, recording the class discussion, knowing that the recording will not be publicized or broadcast in any way.

15. I feel comfortable speaking in class when a recorder is present, recording the class discussion, knowing that the recording will be made available over the Internet.

16. I feel comfortable sharing my research ideas with the class when a recorder is present, knowing that it will be made available over the Internet.

17. I feel that what I say in class would be affected more if a recorder were present than if a recorder were not present.

17

Data CollectionData CollectionPerceived Usefulness and Perceived Use* (From Technology Acceptance Model modified to reflect the usefulness and use of digital recordings of classroom lectures) [SA, A, NA/D, D, SD]:

18. Having access to recorded classroom sessions will allow me to learn more quickly.

19. Having access to recorded classroom sessions will improve my grade performance.

20. Having access to recorded classroom sessions will improve my effectiveness in learning the presented classroom session content.

21. Having access to recorded classroom sessions will make the learning of the material presented easier.

22. Having access to recorded classroom sessions will be useful in my studies.

23. The students were asked to rank their perceived use of recorded classroom lectures on a scale from listening to all of the recordings to none of the recordings. 18

Data Analysis - Survey Data Analysis - Survey 11

Purpose of Demographic Data: ◦ To determine whether non-randomly assigned students were equivalent

◦ Testing Age, Gender, Academic Status, Ownership of mp3 player Age:

No significant difference between the groups Gender:

No significant difference between the groups Academic Status:

No significant difference between the groups Ownership of mp3 player:

No significant difference between the groups

19

Data Analysis - Survey Data Analysis - Survey 11

Testing Word Processing/Email/Web-factored, Music-factored (to/from Computer & MP3 player), eLearning Environment-factored◦ Significant differences that were NOT found Between Groups: Word Processing/Email/Web eLearning Environment

◦ Significant differences WERE found between Groups and MP3 component Separated Music knowledge Responses (testing age groups, gender, academic status, ownership of MP3 player)

Downloading music to a computer from a CD Downloading music to a computer from the Web Load music onto an mp3 player

Differences in Gender and each of Music Knowledge questions Females less comfortable in this area than males (*Alshare - Computer Literacy) 20

Data Analysis - Survey Data Analysis - Survey 11

Attitudes Toward Speaking up in Class with and without recorder present◦Two factors extracted:

A. Speaking up in class B. Speaking up in class with digital recorder present

◦Significant differences were NOT found: Groups Age Groups Gender (for factor A)

◦Significant differences that WERE found: Between Gender and Speaking up in class with recorder present: Females were less comfortable speaking up in class when recorder is present

Compared Separately: A (84.9%) and B (71.7%) Number of students participating DECREASES when recorder present 21

Data Analysis - Survey Data Analysis - Survey 11

Perceived Usefulness of digital audio recordings:

◦ No Significant differences that were found: Group Age Group Gender

◦ 81.3% responded that they Agree or Strongly Agree that having access to recorded classroom sessions would be USEFUL to their studies.

22

Data Analysis - Survey Data Analysis - Survey 11

Relationship between Perceived Usefulness and Use: ◦ Pearson Correlation used to find the relationship between Usefulness and Use: (2-tailed, = .01) Usefulness: M = .87, SD = .814

Use: M = 2.41, SD = 1.165◦ r(105) = -.600, p < .000◦ The more useful the students found the audio digital recordings to be, the more likely they perceived that they would use them

23

Unlikely Will listen to all

St. Ag

St. Di s

Data CollectionData Collection

Survey 2: (Experimental Group ONLY)◦Purpose:

To determine if perceptions change over time To determine use (and intended use) of the recordings

◦Distributed after third week of recording classroom lectures

24

Data Analysis - Survey Data Analysis - Survey 22

Did perceptions change?◦Paired Sample t tests were performed

Perceived classroom participation levels Perceived Usefulness of recorded lecture sessions

No significant differences were found between the answers to the questions from the first survey to the questions on the second survey

25

Data Analysis - Survey Data Analysis - Survey 22

Perceived Usefulness◦81.3% - Most students perceived the recorded lecture sessions as useful No significant differences reported (age groups and gender)

Perceived Ease of Use◦90.6% - Most students perceived that the podcast site would be easy to use Males did perceived podcast site would be easier to use than females

26

Data Analysis - Survey Data Analysis - Survey 22

Use Question:◦ I have listened to at least one of the lecture recordings (18.9%)

◦ I have not listened to any of the recordings, but intend to (62.3%)

◦ I have not listened to any of the recordings and do not intend to listen (18.9%)

Analyses:◦ Age Group: No significant differences◦ Gender: No significant differences

81.2% have or intended to listen.

27

Data Analysis - Survey Data Analysis - Survey 22Comparing Perceived Usefulness on Use and Perceived Ease of Use on Use:◦ Perceived Usefulness had a stronger negative relationship to Use than did Perceived Ease of Use:

Perceived Usefulness on Use*: Perceived Ease of Use on Use:

r(51) = -.657, p < .000 r(51) = -.379, p < .000

28

Strongly Agree

UsedWill not Use

Strongly Disagree

UsedWill not Use

Strongly Agree

Strongly Disagree

Hypotheses - ResultsHypotheses - Results H1: There is no difference between perceived student interaction levels during traditional classroom sessions where a digital recorder is recording the classroom sessions and in those traditional classroom sessions where a recorder is not present.◦ Not Supported: The study found that students feel less comfortable speaking up in class if a digital audio recorder were present. In addition, when testing gender differences, females in this study were less comfortable speaking up in class with a recorder present than were the males.

H2: In classes where the sessions are recorded, students' perceived participation levels will not change over time.◦ Supported.

H3: Age will not have an effect on perceived usefulness of the recorded sessions.◦ Supported.

H4: Age will not have an effect on perceived ease of use of the recorded sessions.◦ Supported.

29

Hypotheses - ResultsHypotheses - Results H5: There is no significant difference between men and women as to the perceived usefulness of recorded sessions. ◦ Supported.

H6: There is no significant difference between men and women as to the perceived ease of use of the recorded sessions◦ Not Supported: Males responded that the hosting site would be easier to use than did the females.

H7: Students who perceive the recorded classroom sessions as useful are no more likely to use the digital recordings than students who do not perceive the recorded sessions as useful.◦ Not Supported: The more the students felt that the recorded sessions were useful, the more likely they were to use the recordings. The less the students felt they were useful, the less likely they were to use the recordings.

H8: Students who perceive the digitally recorded classroom sessions as easy to use are no more likely to use the digital recordings than those who do not perceive the digitally recording as easy to use.◦ Not Supported: A weak relationship showed that the easier the interface was perceived to be to access the digital audio files, the more likely the students were to use them.30

ImplicationsImplications <<< STRESS: Purposive Study / Cannot be generalized to all introductory computer science and technology classrooms at all colleges and universities! >>>

According to this Purposive Study:Classroom discussions may be affected

◦ Most students felt comfortable speaking up in class 85.1% felt comfortable speaking up in class without recorder present

◦ Fewer students felt comfortable speaking up when a digital recorder was present than when the recording was put online (this difference was significant) 71.9% felt comfortable speaking up in class with recorder present (not put online)

62.6% felt comfortable speaking up in class with recorder present (put online)

◦ Fewer students felt comfortable discussing research ideas with recorder present (This difference was significant) 71.9% would share without recorder present 57.9% would share with recorder present

31

ImplicationsImplicationsFemales were less likely to feel comfortable speaking in class when a digital recorder is present:

◦Faculty should be made aware of this difference: More females may be excluded from classroom discussion than men.

For those females, less discussion may result in a lower understanding of course material (Weaver, 2005, p. 594)

32

Suggested ResearchSuggested ResearchReplication of study◦No significant differences between pretest and posttest results.

◦Recommend: True Experimental Design with random sample

Expansion of Study◦Include video (in addition to digital audio recording of classroom sessions)

33

Contact InformationContact InformationZenia C. Bahorski Ph. D.◦Department of Computer Science◦Eastern Michigan University◦Ypsilanti, MI 48197◦United States

Email: zbahorski@emich.eduWeb: people.emich.edu/zbahorski

34