video lecture capture initiative - fall 2009 initiative report
DESCRIPTION
Dr. Naomi Hall Behavioral Sciences and Social Work July 2010TRANSCRIPT
Submitted by Dr. Naomi HallBehavioral Sciences and Social Work
July 2010
Video Lecture Capture Initiative
Fall 2009 Initiative Report
Winston-Salem State University
EXECUTIVE SUMMARY
Winston-Salem State University (WSSU) has experienced a steady decline in student retention and graduation rates over the past five years. Additionally, the number of students who are dropping, failing, or withdrawing (DFW) from classes each semester has been increasing. The recently released WSSU Strategic Plan (2010-2015), acknowledges a number of contributing factors to deteriorating student outcomes. Poor student preparation has been identified as an outcome that demands systematic and timely address. One of the techniques, video lecture-capture, is being piloted by the Center for Excellence in Teaching and Learning (CETL) on the campus of WSSU. Since the core of students’ academic success centers around what happens in the classroom, CETL believes that the lecture-capture approach will be a good supplemental tool for enhancing student performance. Video lecture-capture (VLC) is a mechanism that allows faculty to either pre-record lectures or tape live lectures and upload for students to review at their leisure. The VLC has great potential to enhance teaching and learning outcomes—as indicated by surveys completed by both students and faculty. Unfortunately, there were some challenges with some of the data so the discussion of the extent to which this project enhanced teaching and learning is not included. VLC usage was significantly correlated with aggregated assessment scores and increased from video to video—generally declining after the third video. Nonetheless, overwhelmingly, students thought the VLC was helpful, beneficial, and valuable to their learning experience in the participating classes. This report identifies a number of interesting findings and trends; however, six notable findings/trends are as follows:
The vast majority of students thought the VLC was helpful, beneficial, and valuable to their learning experience in the participating classes;
The students who accessed the VLC system most frequently were those who had GPAs either in the very high or low range;
Students who worked more than 11 hours per week were more likely to utilize the videos for instruction and review;
The availability of pre-recorded and live recorded videos did not negatively impact student attendance in classes;
Data indicate that class structure should be should be taken into account when deciding whether or not to implement VLC in a particular course; and
VLC usage did not have a statistically significant impact on assessment scores in individual classes; however the impact of usage on assessment scores was seen when data was aggregated
2
Introduction
The lecture-capture initiative was started to address the increasing
problem at Winston-Salem State University (WSSU) of students who are
dropping, failing, or withdrawing (DFW) from classes each semester.
Institutional research studies have shown a constant decline in retention
each semester as a result of this problem. According to the WSSU Strategic
Plan (2010-2015), there are a number of contributing factors to
deteriorating student outcomes, but one of the main ones is poor student
preparation (WSSU, 2010). This can be interpreted as poor pre-college
preparation, and/or poor classroom preparation. A number of different
approaches have been taken across the academy to address the issue of
high DFWs. Some of these approaches included modifying tutoring services,
increasing mentorship initiatives, following up with at-risk students at
shorter intervals in the semester in order to monitor their progress,
providing remediation, among other efforts. It is important to develop,
enhance, and supplement instructional support programs so that students
3
are successful inside and outside of the classroom. Since the core of
students’ academic success centers around what happens in the classroom,
the lecture-capture approach seemed to be a good solution for enhancing
student performance. Earlier work with lecture-capture has shown a
positive improvement in the learning and performance of students who
review the videos (Chandra, 2007).
Lectures captured using digital video and made available using streamed
video allows students to review at their own pace the information the
instructor explained. They can also view any associated visual aids and
search for concepts and resources relating to those aids. Lectures captured
in this way may help students overcome weaknesses in areas such as taking
notes, paying attention over extended periods of time, or understanding
language barriers. Another very important aspect is the ability to make
remediation sessions available so that students who may lack certain skills
can remediate without having to spend class time doing so. With this in
mind, the Center for Excellence in Teaching and Learning (CETL) explored
alternative video lecture capturing solutions, which in essence, places at the
student’s disposal the core of what affects their academic success, which is
the professor’s perspective on the course material. An initial pilot (N = 72)
took place during the 2009 summer session II. A brief summary of the
findings are as follows (Hall, 2009):
4
Students who utilized Panopto overwhelmingly (85%) thought it was
helpful, beneficial and valuable to the class in which they were
enrolled.
A student’s grade point average (GPA) was correlated with whether or
not the videos were viewed, and for how long. The students who
reviewed the videos more frequently, and for longer periods of time
appear to be those students whose GPAs are hovering around the 3.1-
3.5 range.
It appears that two of the main factors that helped students decide
whether or not to access the videos were class requirements, and
time. Of those students who did not review any videos, over half said
they did not because it was not required for the course.
o The summer sessions are very fast and intense, and most of the
students who were attending classes (per Assessment data) also
worked at least 20 hours per week. Students were honest in
saying they had to prioritize what they could and could not do in
this shorten course. This speaks to making Panopto a
requirement so that students will not have to make those types
of study and review choices in the future.
Summer faculty did not see the benefit of the videos to the students.
The videos were not seen as convenient, helpful in student
preparation for quizzes and exams, or in discussions.
5
o This may be a direct reflection of the fact that all videos during
the pilot were recorded during an actual class session. Pre-
recording has the potential to have faculty teach a
concept/theory/phenomenon in a virtual one-on-one session with
PowerPoint slides, instead of moving around the classroom.
None of the faculty agreed to pre-record, so this is something
that will be explored in the fall 2009.
With the Fall 2009 Initiative, CETL investigated:
Whether this type of system enhanced the teaching and learning
processes
If it can enhance it, to what extent?
If the availability of the videos had any significant impact on
student assessment scores
If students found the video lecture-capture software helpful and/or
valuable to their learning experience
Methodology
Participants
Students participating in this project were from eight courses that
have a high percentage of DFWs at Winston-Salem State University. There
were no specific inclusion and/or exclusion criteria for students. All students
in the eight selected classes had the opportunity to participate. Additionally,
the faculty who taught those courses participated in the project as well.
Measures
6
Students were asked to complete three measures: 1) Initial Student
Assessment, 2) Individual Video Lecture Capture (VLC) Assessment, and the
3) Final Student Assessment. The Initial Student Assessment was given to
all students (N=264) at the beginning of each course and asked primarily
for demographic information such as gender, student classification, age,
major, current work status, and instructor. The Individual VLC Assessment
was an ongoing survey that asked students to provide information on the
video(s) they viewed, along with their attitudes and experience using the
software. Finally, students were asked to complete a Final Student
Assessment to ascertain their attitudes and experiences viewing the video,
using the software, and how it may have affected their performance in the
class. This assessment also asked students to provide any suggestions for
improving the videos and software, and implementing the initiative in other
classes. Faculty who participated in the initiative were asked to complete a
Faculty Assessment Form to gather feedback about their perceptions of
using the software, how they think the performance of students in class was
impacted, and any of their suggestions for future use.
Procedures
The videos were captured and uploaded using Panopto. Panopto is a
lecture-capture system software which makes portability quite easy. It
allowed faculty to record their lectures from anywhere, at anytime, using
video and audio, and captured information from PowerPoint slides, or other
text sources. From the metadata that is created, students had the
7
opportunity to search using any of the text, or PowerPoint thumbnails.
During a search, all data (video, text, audio) is synchronized at the point of
delivering the search results. Students were all instructed to sign up for a
Panopto account so that they could access the videos.
After a few trials to be sure that Panopto would not pose any problems
with its integration with Blackboard, the software was fully implemented for
use by any faculty member. Eight faculty members, mainly in the College of
Arts and Sciences and School of Health Sciences, either pre-recorded or
recorded their lectures live, and made the contents available for viewing by
their students. Additionally, faculty were provided with the measures from
which the following data were collected.
Students and faculty completed all assessment forms via SurveyMonkey
© by assessing the provided link. Faculty provided the link to the three
assessment tools to all students via email. Students clicked on the link and
were taken to the appropriate survey. Faculty were also given a link to use
for completion of the Faculty Assessment Form.
Results
Initial Student Assessment
A total of 309 students provided information for the initial survey.
However, only 264 students indicated that they consented to participate in
the study—the remaining 45 students skipped this initial question. The
following data represents only the students who indicated their informed
consent for participation. The participant pool (N=264) for this survey was
8
overwhelmingly populated by women (78.2%). When students were asked
“in which range does your age fall?” we saw that 69.1% of the students
indicated they were less than 20 years of age, 18.5% between 21 and 24
years, and 12.4% over the age of 25. The student classification was diverse
with the sample consisting of 27.7% Freshman, 32.7% Sophomore, 22.3%
Juniors, 11.9% Seniors, and 5.4% pursuing a second degree. There were 32
different major courses of study represented in this sample. The top five
most frequently indicated majors were: 1) Nursing (29.6%); 2) Psychology
(8.8%); 3) Business Administration (8.1%); 4) Mass Communication (7.3%);
and 5) Information Technology (6.9%). The other 27 majors represented
39.3%, collectively, of the pool.
Of the 264 students who completed the survey, 57 (21.6%) were
repeating the course. When asked why students were retaking the course,
43.4% indicated they had failed the course. Approximately 32% received a
‘D’ and 24.5% either dropped or withdrew from the course. The trend
seemed to be that those students who were taking the course again, did so
with a different instructor (88.9%). Forty percent (41.6%) of the students
indicated that they were currently working, and 39.5% worked 11-20 hours
per week, and 35.4% worked over 21 hours per week. An interesting finding
was discovered when examining the relationship between those students
who work, and those who had to retake, χ2 (1, N = 263) = 4.30, p = .038.
Students who worked were significantly more likely to be the ones who
were re-taking the class.
9
Although there were eight instructors participating in the study, the
vast majority of the students came from the classrooms of Drs. Nelson
Adams (PSY 2301 General Psychology-33%) and David Kump (BIO 1301
Biological Concepts-24.2%). The other instructors (see Figure 1), and the
percentage of students participating are as follows: Manjunatha Bhat (BIO
2311 Anatomy and Physiology-11.4%), Johanna Porter-Kelley (BIO 1331
General Microbiology-10.5%), Christopher Stanley (PSY 2326 Statistics for
Social/Behavioral Sciences-8.7%), Jan Jasper (FIN 2356 Personal Finance-
7.6%), Jeffrey Overholt (BIO 3311 Fundamentals of Anatomy and
Physiology-4.2%), and Sathasivam Mathiyalakan (MIS 3380 File
Structure/Database Operations-.4%).
Adams (PSY)33%
Bhat (BIO)11%
Jasper (FIN)8%
Kump (BIO)24%
Mathiyalakan (MIS)0%
Overholt (BIO)4%
Porter-Kelley (BIO)10%
Stanley (PSY)9%
Figure 1: Percentage of Participating Students
Individual VLC Assessment
There were a total of 149 survey responses for the Individual VLC
Assessment. The Individual VLC Assessment asked students to provide
10
information on the video(s) they viewed, along with their attitudes and
experience using the software. Even though there were eight faculty
members participating in the study, only six had students who completed
the surveys. The six faculty members whose class data is used for this
section are Drs. Mathiyalakan, Stanley, Kump, Adams, Porter-Kelley, and
Overholt. Students were asked to indicate which videos they reviewed
during the course. The following highlights these results:
1. Dr. Mathiyalakan—it was difficult to ascertain a pattern of video
viewing. His students represented 53.5% of those who viewed
the videos; however, there was no consistency in how students
reported the name of the video. Some students indicated an
assignment number, while others listed a chapter number. Even
though it was difficult to identify the most frequently viewed
video, a large number of students said they “viewed all of the
available videos.”
2. Dr. Stanley—his class made up 24.3% of the students who
viewed videos, and the top three videos viewed were Chi-
Square, Hypothesis testing, and Means and Variances.
3. Dr. Kump—his class made up 9.1% of the students who viewed
videos, and the top video viewed was the one related to Chapter
7 (Carbon Cycle).
4. Dr. Adams—his class made up 7.6% of the students who viewed
videos, and the top video viewed was on the Brain.
11
5. Dr. Porter-Kelley—her class made up 4.2% of the students who
viewed videos, and the top video viewed was on Chapter 7 (DNA
Replication).
6. Dr. Overholt—his class made up 1.4% of the students who
viewed videos, and the top video viewed was on the Endocrine
System.
It was important to gather information on the reasons students viewed
the videos, and how the video(s) may or may not have contributed to their
success in class. The overall tone of the students was that it was good for
review and clarification. When students were asked to indicate why they
viewed the video, overwhelmingly the response was focused on “reviewing
concepts.” When examining qualitative responses, three responses stood
out. One student said “to make sure I had an understanding of what was
going on so I am prepared for my test.” Another student indicated that
he/she had “missed half of this lecture and I wanted to see and hear it in my
own space so that I could understand it. I reviewed it totally twice and
viewed it a third time in sections that I needed to in order for me to
understand fully.” Finally, a non-traditional student said “I am returning to
school and don’t catch on as fast as the younger students. The videos
allowed me to pay attention and grasp the information at my pace.”
Students were asked if the video(s) were helpful in providing the
information they needed for the class, and 96.6% indicated it was.
12
Different instructors used different recording techniques with their
class—some pre-recorded and others recorded live. Students were asked if
they acquired any new information by watching the videos. Slightly over
half (54.7%) said they acquired new information, and one student said that
he or she was able to “reaffirm my knowledge of the subject.”
The overall viewing pattern for students appeared to be that they
viewed the entire video (93.2%) by themselves (97.3%). For those students
who did not view the entire video, interestingly, there was not much variety
in the amount of time spent viewing. Of the students who did not view the
entire video, approximately 33.3% viewed it for less than 5 minutes, 33.3%
viewed it between 6-10 minutes, and 33.3% viewed it for over 10 minutes.
We asked students if the video(s) were helpful to them, and 84.7% said
“yes.” On a scale from 0 to 10 (10 being the highest), students were asked
how beneficial was the lecture-capture video you just watched to your
understanding of concepts relating to this class? The mean rating was 8.48
(range 0-10, sd = 1.90). So, all in all, most students felt these videos were
beneficial to the course.
We wanted to see the grade point average (GPA) of those who viewed
the videos to see if those who had higher GPAs were more or less likely to
utilize the software. Of the students who used the software to view one or
more video, 35.4% had a GPA of 3.0 or better. The video(s) were viewed
most by those who had a GPA between 3.1-3.5. Of those who viewed at least
one video, 46.9% worked, in addition to being a fulltime student, and most
13
(55.2%) worked more than 20 hours per week. This initiative is still in the
preliminary phase of implementation, so we wanted to make sure to ask the
students to rate the technical quality of the Panopto software.
Overwhelmingly (89.9%), students felt the quality was either “good” or
excellent.”
We looked at the relationship between GPA and how beneficial
students thought the videos were. An analysis of variance (ANOVA) was run
on the two variables and found a significant relationship between a
student’s GPA and their ratings on the benefit of viewing the videos, F (3,
131) = 4.05, p = .004. Interestingly, students with the highest GPAs (over
3.5) and lowest GPAs (less than 2.0) found the videos most beneficial (9.35
and 9.40, respectively). A post-hoc Tukey’s test was conducted to examine
any possible differences between the five GPA groups. Two significant
differences were found between how beneficial the videos were for those
students whose GPAs were between 2.1-2.5 (8.81) and 3.1-3.5 (7.48), p
= .024; and those whose GPAs were between 3.1-3.5 and Over 3.5 (9.35), p
= .007. This finding is different than that of the initial pilot data (summer
2009). Pilot data revealed that students who were more likely to watch
these videos were those who were in the 3.1-3.5 GPA range, not those above
or below. However, the fall semester students, whose GPA is in the same
range, actually rated these videos to be the least beneficial.
Although there were no significant difference between employment
status and benefit of the videos, there were significant differences between
14
the number of hours a student works and their beneficial rating of the
videos, F (2, 63) = 10.58, p <.001. Those students who work more than 21
hours per week rated the videos as much more beneficial (9.62) than those
who worked between 11-20 hours per week (7.59), p <.001. Students who
worked less than 10 hours per week rated the benefit of the videos lower
than both groups (7.0).
Final Student Assessment
Two hundred and sixty four students completed the initial survey, but
only 150 completed the final survey (57%). As with the initial student
survey, the majority of the students who completed the final survey were
from Drs. Adams (26.2%) and Kump’s (20.1%) classes. Most of the faculty
had better initial survey rates than final survey rates. However, two
professors (Drs. Overholt and Mathiyalakan) had better final survey rates
(18.1% and 12.1%, respectively).
The students were asked to indicate the number of videos watched
over the course of the semester, and one in three students (35.6%) viewed
five or more videos. Only 5.4% of the students did not view any videos, and
the remaining 59.1% viewed between one and four of the videos available.
Of those 5.4% who did not view the video, 38.5% said they did not because
“it was not required for the course.” Unfortunately another 38.5% did not
view any of the videos because they indicated they had technical difficulties
assessing Panopto; 15.4% said they “didn’t see the benefit;” and 7.7% did
not know about the videos. Students had the opportunity to choose multiple
15
options, so the total does not equal 100%. Unlike the summer session pilot
project, most of the videos were pre-recorded (61.2%) versus taped during a
live class session (38.8%). Students were asked which style of video they
preferred, and two in five (41.6%) prefer the pre-recorded videos, while
30.2% indicated “no preference.” When asked about video viewing habits,
students provided the following information:
47.9% indicated they viewed the videos once, but went back to certain
parts of the video, while 18.8% viewed the video only one time.
26% of the students who viewed at least one video watched between
6-10 minutes of each video, and 25.3% watched at least 10 minutes of
each video.
Keeping in line with the Individual VLC Assessment, students
overwhelmingly viewed the videos alone (78.4%).
Students were asked how beneficial the videos viewed were in
understanding the concepts related to the course, and how valuable the
videos viewed were to the course. Both of these questions were on a 0-10
scale, with ‘0’ being the lowest (‘not at all’) point on the scale. More than
half of the students (58.3%) rated the benefit of the videos between an 8
and 10, 34.9% between a 4 and 7, and 6.8% between a 0 and 3. The mean
beneficial rating was 7.46 (range 0-10, sd = 2.37). Overwhelmingly,
students (65.4%) rated the value t of the videos between an 8 and 10, 29.2%
between a 4 and 7, and 5.4% between a 0 and 3. The mean value rating was
7.73 (range 0-10, sd = 2.33). Although there were some students who rated
16
the benefit and value fairly low, 99% said the videos were “very beneficial”
or “somewhat beneficial” for student learning in the course. We were
interested in any impact the initiative may have on a student’s attendance.
Once of the concerns voiced was that students may not show up to class
because the videos are available to them. This was unfounded in this study,
as 85% of the students said the video(s) had “no effect” on their attendance.
We asked students a series of eight questions designed to get their
attitudes, opinions, and experiences with using the software and videos. The
results of the eight questions are in Figure 2.
0
20
40
60
80
100
120
Figure 2: Student Perceptions/Experiences
Strongly Agree Agree Neutral Disagree Strongly Disagree
The last two questions asked of students were to provide any
suggestions to other students on how to use the videos to their benefit, and
provide any suggestions on how to improve using the videos in the future. A
17
number of the suggestions to students were similar and, therefore,
aggregated for the purposes of this report. Sixty five students (43.3%)
responded to the question, and the trend focused on two main categories: 1)
Review of material; and 2) “How to get the most from the video.” The vast
majority of comments were about using the videos to review concepts,
theories, and techniques prior to quizzes and exams. Since quizzes and
exams were the main method of assessment in the participating classes, this
is interpreted as a reinforcement of the benefit and importance of having
these videos available. Many students took this opportunity to provide
advice to other students on how to get the most from the videos. Some of
the suggestions were to “make sure you have your notes with you when you
watch the video,” “watch the whole video—sometimes the meat is at the
end,” and “pay attention because teachers say things in the videos that they
don’t in class.”
Of the 51 (34%) students who provided suggestions for improving the
use of the videos, 42% addressed the length of the videos. Students thought
the videos should be “short and focused on one topic,” and “integrated into
the class discussion more.” Suggestions for improvement also addressed the
quality and ease of access to the videos via the Blackboard system.
However, a few of the suggestions focused on the Panopto system itself.
Finally, there were eight (16%) students who thought that the student
should be able to see the Professor point out certain things or draw certain
elements when the video is combined with a PowerPoint presentation.
18
Student Assessment Information (Individual Classes)
Complete statistical information on student assessment outcomes was
submitted by two professors, Drs. Kump and Adams. Dr. Kump found that
VLC usage did not have a statistically significant effect on the quiz or exam
scores of students. While both quiz and exam scores were higher for those
students who watched the videos versus those who did not, the p-values
were not below 0.05. Dr. Kump compared the mean GPA of students (video
watchers and non-watchers), with the score received at the end of the
course. As expected, there was a strong correlation (r2 =0.6) between the
students’ GPA and the grade received at the end of the course (p < .001).
Additionally, the mean GPA (2.7) was significantly higher (p <.001) for
those who reported watching at least one video than those who did not
watch any. Dr. Adams had different results for the quizzes and exams in his
class. Those students who viewed at least one video averaged significantly
higher scores than those who did view any videos (83.9% vs. 70%,
respectively) t(43) = 4.02, p <.01. Additionally, Dr. Adams reports
significant improvement in the final exam scores of students (to which
videos were available) than the first three exams.
Panopto statistics were obtained on four professors: Bhat, Stanley,
Overholt and Porter-Kelley. After examining viewing data, it was
determined that VLC usage did not have a statistically significant effect on
the assessment scores of students in the individual classes. T-tests (students
who viewed at least one video vs. students who did not view any videos)
19
reveal differences in exam and quiz scores for students in these classes;
however, none of the differences were significant at p ≤ .05.
Looking at the aggregated VLC usage and assessment data, we see
that most students did not view the first video in any of the classes.
However, the number of students viewing videos increased significantly
from the first video to the second video, t(69) = 1.97, p =.05. It is unclear
why the increase in number of student viewing occurred, but student ‘word-
of-mouth’ and faculty encouragement may have contributed to the increase.
Following along with the increased number of students viewing the second
video, a statistically significant trend was found with number of minutes
viewed (Video 2) and corresponding assessment score (Assessment 2), F (1,
69) = 5.28, p <.05. This significant finding was not observed with any other
assessments or videos. The number of students viewing continued to
increase from the second to third videos—although not significant. The
student numbers began to decrease around each professor’s fourth video.
All participating faculty had four or five videos except Dr. Porter-Kelley,
who had eight videos for students to view.
Faculty Perceptions
Faculty who participated in the initiative were asked to complete a
Faculty Assessment Form to gather feedback about their perceptions of
using the software, how they think the performance of students in class was
impacted, and any of their suggestions for future use. Many of these
questions were the same in the Final Student Assessment—just re-worded.
20
Seven of the faculty members completed the assessment—although only six
completed most of the questions. All of the faculty (100%) recorded five or
more lectures. Most of the faculty (42.9%) indicated they spent more than
30 minutes on each lecture, while 28.6% spent between 21-30 minutes, and
the remaining 28.6% spent less than 20 minutes. All faculty (100%) said
they informed all of their students about the videos, and showed all of the
students how to access the videos. We encouraged all faculty members to
provide incentives for their students, and most provided this in the form of
points. Three faculty members provided extra credit, one gave participation
points, and one gave assignment points. The other faculty provided no
points and/or incentives for viewing the videos. One faculty member
elaborated on his/her decision to not provide any extra motivation for using
the videos, “As I stated many times before, if one of our outcomes is
increased grades we cannot inflate the grades of the experimental group by
giving points or incentives. The incentive has to be their want to do better.”
Instructors were asked how beneficial they thought the videos viewed
were in students’ understanding the concepts related to the course, and
how valuable the videos viewed were to the students. Both of these
questions were on a 0-10 scale, with ‘0’ being the lowest (‘not at all’) point
on the scale. The majority of the faculty rated the benefit and value very
high; 85% indicated the benefit between 8-10, and the 66.6% rated the
value between 8-10. This is very different than the summer session faculty
ratings. The faculty members, during the summer session, all rated the
21
benefit and value of the videos less than 5. All but one faculty member
(85.7%) felt that the videos had “no effect” on student attendance, but one
professor felt that having the videos available “reduced student
attendance.”
We also asked the faculty members a similar series of eight questions
designed to get their attitudes, opinions, and experiences with using the
software and videos. The results of the eight questions are in Figure 3.
Videos were a convenient way for students to access material
Videos helped students to prepare for quizzes
Videos helped students to prepare for exams
Videos helped me prepare for class discussions
Videos helped me in my overall effort to present course information to students
Videos helped to clarify concepts discussed in class for students
Videos were easily accessible for students
The video technical quality was good
0 20 40 60 80 100 120
Figure 3: Faculty Perceptions/Experiences
Strongly Agree Agree Disagree Strongly Disagree No Opinion
Faculty were asked, “In your opinion were there any apparent learning/
performance differences between students who viewed certain topics versus
those who did not?”Although the majority of the faculty did not analyze
their student outcome data, two out of three (66.7%) said “yes.” A
qualitative response from a faculty was as follows: “Students who used the
22
videos regularly remarked that this should be made available in every
class.” Although this in anecdotal information, and not statistically
generated, the faculty perceptions of the learning and performance
differences must be taken into consideration. The entire faculty
participating in this study recommended the expanded, and continued, use
of the VLC.
Finally, faculty were asked to “please provide any additional comments
you would like us to consider relating to the use of Panopto at WSSU.” The
open-ended responses were summed up by the following four comments:
“I think it is very useful, especially for the students who used it. I
was disappointed in the number of students who used it. The video
portion is rather useless, especially if you move around in class.
That portion needs to upgraded.”
“It would be quite valuable in conducting distance-learning or on-
line courses or for students who miss class due to an excused
absence. As a tool for enhancing learning in traditional courses, in
my opinion it does not appear to be effective, at least in the courses
I have taught. Students do not appear to want additional lecture,
even though the videos might not technically be a lecture. Other
methods of enhancing learning may be far more valuable. In the
courses I teach, this includes studying the textbook and reviewing
the notes.”
23
“I wish the system was a little more user friendly—is there a way
the students can log on through Blackboard and not an outside
link”
Continue to use an "experimental approach" where there are some
systematic comparisons. In class video versus supplemental video,
or enrichment type presentations versus repetition (of same
material from class), etc.”
Discussion
Based on the information provided by the three student surveys,
faculty assessment data, and faculty assessment forms—it appears that the
VLC has great potential to enhance teaching and learning outcomes. We
know that those students who split their time between working and being a
fulltime student were more likely to have taken the course again because of
a DFW. Additionally, we know that these students were also more likely to
utilize the videos for instruction and review. With the large number of
undergraduate students (traditional and non-traditional) working more than
11 hours per week, this may prove to be an essential learning tool for
students. Chandra (2007) indicated that one of the primary concerns of
faculty and administrators was that students would just listen/watch to the
videos and not attend the class sessions. We found that this was not a factor
with our students—this tool did not have a negative impact on student
attendance.
24
Unfortunately, we don’t have enough statistical data to discuss the
extent to which this project enhanced teaching and learning. One of the
major concerns is the way that faculty collected student assessment data.
There was a lack of consistency in collection and reporting of final
assessment data. It was very difficult to measure whether or not the
accessibility and usage of the videos were significantly related to the
assessment scores in each class. We have descriptive and qualitative
information on all classes; however, we don’t have final, correlated,
assessment data on all students. Dr. Kump found that the videos did not
significantly increase student assessment scores, while Dr. Adams found
that it did. One of the suggestions moving forward is to have a standardized
data collection format for all faculty and classes.
Data indicated that VLC usage did not have a statistically significant
effect on the assessment scores of students in the individual classes.
However, examination of the aggregated VLC usage and assessment data
reveal that a statistically significant trend was found with number of
minutes viewed and corresponding assessment score of the second video. It
appears that most students did not view the first video in any of the classes
but viewing increased with each subsequent video—plateauing and then
declining at approximately the fourth video. Speculation is made that
perhaps it took students some time to become comfortable with the Panopto
system and/or word of mouth began to spark student curiosity. Another
25
hypothesis is that continued faculty encouragement may have prompted
students to utilize the resource.
Similar to the results of the 2009 summer session II pilot,
overwhelmingly, students thought the VLC was helpful, beneficial, and
valuable to their learning experience in the participating classes. The
majority of the responses by students, quantitative and qualitative, were
positive and in favor of keeping the VLC system in classes. In fact, several
students indicated that it should be implemented in their other classes. We
found that students who had GPAs either in the very high or low range were
more likely to watch the videos, and rate them as more beneficial and
valuable. This may speak to the fact that those students who are doing very
well or very poorly are more conscientious about their performance.
Additionally, motivation may play a major factor in how students perceive
and utilize this tool. If you are at the top of your class, you may want to
engage in additional activities to stay there. If you are not doing very well,
you may be motivated to seek additional assistance to ‘pull’ your grades up.
What still remains unknown is how to motivate and encourage those
students in the ‘middle’ of the class to take advantage of the VLC.
Other interesting findings have to do with the length, type, and
content of the videos. Students seem to enjoy the shorter, concept-focused
videos over the longer, class-related videos. Additionally, two in five
students indicated they preferred the pre-recorded videos versus the ‘live’
videos. Based on some of the feedback from students and faculty, a
26
combination of both may be the best practice. For activities such as concept
explanation and instructional techniques, shorter videos may be preferred.
However, the longer ‘live’ recorded videos may be better for exam review
and content questions. It was noted by two faculty members that the class
structure should be taken into account in deciding whether or not to
implement VLC in the course. CETL will look at the structure and course
content of the classes with the instructors to see if VLC is a viable
supplemental tool.
Finally, with any technological tool, there are some challenges with
quality and usability. Students and faculty thought the quality of the videos
were pretty good. However, there were a few comments that focused on the
accessibility of the videos through an external link. If there is a way to
access the videos through Blackboard, instead of directly through Panopto,
this may increase the usage. One comment from a faculty member was that
some students are not willing to try and access the Panopto system multiple
times.
Limitations
No study, especially a quasi-experimental one, is without limitations.
The first limitation is the lack of consistency with type and amount of data
collected. There were some professors who were able to conduct descriptive
and inferential analyses, while others were only able to produce descriptive
data. Even though the data was aggregated, two professors had the
majority of students using the software. It is unclear if accessing/using the
27
software is positively correlated with the incentive given by that professor.
Secondly, students were self-reporting and may over exaggerate their
software usage because of the desire to obtain an incentive offered by their
professor. Conversely, students may have not participated because there
was not an incentive offered by their professor. A large portion of the
sample were nursing majors and this may skew the data. Some students are
theoretically more motivated to do well in this particular class, because not
doing so would mean denied entry into a specialized program.
References
Chandra, S. (2007). Lecture video capture for the masses. ITiCSE, June 23-
27.
Hall, N. M. (2009). Video lecture capture initiative: summer pilot final
report. Winston-Salem State
University: Center for Excellence in Teaching and Learning.
Winston-Salem State University (2010). Achieving academic distinction: The
plan for student
success. Winston-Salem, NC: Office of the Chancellor
28