Results of Application of “Lessons Learned” Flipped Classroom to an Introductory Level

Marketing Class

George B. Krueger

Assistant Professor—Marketing

Christine Storlie

Business Administration Distance Learning Program Coordinator

School of Business

University of Wisconsin-Platteville

Background

A study was conducted by Krueger & Storlie in the fall of 2013 on to evaluate the flipped

classroom methodology in an introductory level marketing course. The professor has taught the

course for several years in a conventional lecture format so a baseline of learning outcomes and

student evaluations data was available.

The study was focused on two specific flipped strategies. The first was online quizzes

that the students took ahead of the actual class lecture/discussion. The purpose of the online

quizzes was for the students to be familiar with the subject matter for each week when they came

to class. The required quizzes forced the students to engage with the subject matter prior to

discussions in class. The second was expanded classroom discussions and applications focused

on specific group projects where the students prepared for the presentations and discussions

outside of class.

The flipped classroom methodology utilizing the key tools of online quizzes taken before

classroom coverage of the subject matter, student preparation for presentations outside of class,

student presentations each week and a much higher level of group work in class yielded no

statistical differences in 7 of 8 (only Exam 3 was statistically different in favor of the flipped

class) student outcomes as measured by grades when compared the control class taught in

conventional format.

The students did not perceive a significant level of difference in the way the flipped and

conventional classes were taught from their previous experiences when surveyed at mid-point

and at the end of the semester. They did not perceive that the additional effort required on their

part to participate in a flipped class to be an issue.

Additionally, it appeared that the flipped class was not as knowledgeable of the subject

matter as they were expected to be. The professor perceived less understanding of key concepts

in the individual project and the final exam in the flipped class as compared to the conventional

class, although the results were not statistically significant.

In the fall of 2013, the flipped classroom created a higher level of satisfaction for the

students and appeared to engage the students more actively as measured by statistically

significant improved student evaluation scores in the flipped classroom as compared to the

control class and previously taught classes in conventional format. Because of these results, the

professor modified his flipped approach to teaching Introduction to Marketing for the spring of

2014. The rest of this paper focuses on this modified flipped approach used in the spring 2014

class, which is referred to as the Lessons Learned class.

The Current Study: Lessons Learned

As a part of an ongoing process to improve course outcomes the professor implemented

two major lessons learned from the previous study. The purpose of this paper is to assess the

how effective the professor was when implementing an approach to teaching the introduction to

marketing class, wherein the professor implemented these lessons learned. These lessons were:

1. To take the best of both worlds and utilize highly focused lectures on key

marketing concepts during the Tuesday class session and then have the students apply the

concepts in a weekly discussion format during the Thursday class session.

2. Utilize the student’s projects throughout the semester in the context of

lecture and application of marketing concepts.

The professor utilized these approaches to teaching a spring 2014 Introduction to Marketing

class. The results of quizzes, exams, and the final marketing plan project from this spring 2014

Lessons Learned class will be compared with the fall 2013 Flipped class averages to determine if

there is a significant difference.

Spring 2014 Lessons Learned Class Approach

Online Quizzes

Online quizzes were administered through the Desire To Learn course management

system. The quizzes were multiple choice and generated randomly for each student based on the

assigned chapters. The quizzes had a deadline of 11:59 pm on Monday of each week and the

students had 60 minutes to complete each quiz. Students were allowed to use notes and the

textbook for reference. There were 10 quizzes with 20 questions at 1 point each for a total of 20

points per quiz and 200 total points. There was a bonus 11th

quiz that counted for extra points.

In the 2013 Flipped class the quizzes were valued at 50 points as compared to the conventional

class at 20 points. The purpose of the increased points was to increase the importance in the

overall grading rubric and to stimulate more emphasis on the quizzes by the students since they

counted for a higher percentage of the grade in the flipped class. Since there was no statistical

difference in the quiz scores in the fall of 2013 the quizzes were returned to 20 points each for

the Lessons Learned class.

Group Projects

The Lessons Learned class was split into 6 groups of 5-7 students. Each group then

selected a company to follow and analyze.

The focus companies/products were Gatorade, NorthFace, Signal Snowboards, Apple I-

Phone and Samsung Galaxy Watch.

The groups were assigned learning outcomes and marketing concepts each week to apply

to their project. Each group had to prepare outside of class but were also given limited in-class

time for their work. At the end of the semester each group briefly presented their learning

outcomes to the class for their company project. Throughout the semester, discussion of each

group’s work on the weekly learning outcomes was interwoven into the lecture and discussed

during class.

Exams

One of the minor issues identified by students from the previous semester was that the

exams were too long and covered too much material. In response to this issue the course was

modified to include an additional exam. Each exam covered less total course material than

previous semester. As a result of this change, there were four exams of 100 points each as

compared to three exams of 100 points in the previous semesters.

Final Exam

The final exam was a comprehensive exam covering the entire semester. The exams

were similar. The final exam was 40% multiple choice and 60% application of terms related to a

case on Subway Restaurant’s marketing. The case subject was given to the students along with a

study guide before the exam.

Final Individual Marketing Plan Project

A final individual marketing plan project was assigned to the students. The assignment

was for the student to select a product or service to market and write a marketing plan for their

selection. An outline and grading rubric was provided to the students. This project is intended

to be a capstone application of the marketing concepts.

Table 1. Summary of Grading

Lessons Learned

Class Spring 2014

Flipped Class Fall

2013

Online Quizzes 200 500

Group Projects 100 100

Exams 400 300

Final Exam 100 100

Final Individual Project—

Marketing Plan

100 100

Class Participation 100 100

Total Points 1000 1200

Results

Four areas were measured from the grading outcomes to directly analyze the learning

outcome results of the Lessons Learned strategy. The four areas were:

Average total score of the online quizzes,

Average total score of the exams,

Average final exam score, and

Average score on the individual marketing plan project.

The null hypothesis that there is not a significant difference in the scores for the 2013

Flipped class and the Lessons Learned class was tested using an independent samples t-test for

each of the four areas.

Online Quizzes

The Lessons Learned class took the quizzes on Tuesday by 11:59 pm after the lecture but

before the Thursday discussion day. There was not a significant difference between the average

quiz scores for the Lessons Learned class and the 2013 Flipped class (Table 2).

Exams

There were 3 exams in the 2013 Flipped classover the subject matter during the semester,

while there were 4 exams over the same subject matter in the Lessons Learned class. The

results for average of the total of the exam scores given throughout the semester were not

statistically significant using a standard t-test (Table 2). The t-test conducted on the average

Final Exam scores did show significantly higher average scores in the Lessons Learned class at

the .01 level (Table 2), such that the Lessons Learned class average was 77% while the 2013

Flipped class average was 68%.

Table 2. Average scores for the fall 2013 Flipped class and spring 2014 Lessons Learned class.

Fall 2013 Flipped

n = 33

Lessons Learned

n = 31

t-test results

Mean Std Dev Mean Std Dev t p

Online Quizzes 77% 8.6 78% 16.5 0.344 .37

Total Exams 76% 9.2 76% 9.8 0.035 .49

Final Exam 68% 9.5 77% 16.1 2.62 .005**

Marketing Plan Project 81% 13.6 88% 13.5 2.02 .024*

*Significant at the .05 level, **Significant at the .01 level

Final Individual Marketing Plan Project

The 2013 Flipped class scored 81% on the individual marketing project while the

Lessons Learned class scored 88%. The t-test conducted on the average Marketing Plan Project

scores did show significantly higher average scores in the Lessons Learned class at the .05 level

(Table 2).

Results Of Semester-End Student Evaluations

The student evaluation scores for the Lessons Learned class were compared to the 2013

Flipped class and previous classes. Questions 3, 4, 7, 9, 10 and 15 are featured because they are

directly related to the student perceptions on the flipped methodology (Appendix 1). For the

Lessons Learned class, on all the questions the student’s evaluations appeared to revert to the

historic mean as compared to the fall 2013 Flipped class (Table 3).

There was a numerical decline of the student evaluation scores of the course in terms of

satisfaction from the 2013 Flipped as compared to the Lessons Learned. However, the average

scores for the final exam and final project statistically improved. It should be noted that spring

Lessons Learned class evaluation scores mostly reverted to the mean of the previous semesters.

This score indicates a general satisfaction with the class. There appears to be an issue with the

“Explains Course Material Clearly” which is surprising since there was more lecture in the

Lessons Learned class than the 2013 Flipped course. It appears that while students appeared to

be slightly less satisfied (although a 4.39 is considered “Outstanding”) their learning outcomes

were statistically higher as measured by the final exam and final project in the Lessons Learned

class.

Table 3. Student evaluations from fall 2011to spring 2014.

Question Lessons Learned Sp 2014

Flipped Fall 13

Control F 13

Sp 13 Fall 12 Fall 12 Fall 11 Fall 11

3. Uses examples and illustrations that clarify the material.

4.45 4.857 4.524 4.367 4.308 4.519 4.548 4.31

4. Explains course material clearly.

3.69 4.5 4.05 3.967 3.8 3.926 4.097 4.179

7. Incorporates current material into the course.

4.61 4.821 4.619 4.533 4.692 4.519 4.452 4.621

9. Encourages classroom participation from students.

4.66 4.857 4.476 4.4 4.28 4.481 4.677 4.483

10. Presentations/ discussions enhance the text.

4.03 4.571 4.238 4.367 4.038 4.296 4.258 4.138

15. Uses class time for student discussions, presentations or questions.

4.64 4.821 4.238 4.3 4.292 4.296 4.613 4.448

Mean (all responses) 4.39 4.738 4.394 4.361 4.278 4.51 4.498 4.42

Responses 29 28 21 30 26 27 31 29

Student Comments

The qualitative student comments were unaided answers to the following questions:

1. Identify specific things your instructor has done especially well in

teaching this course.

2. Identify specific things you believe could be done to improve the teaching

of this course.

3. Other constructive comments.

The comments were coded into categories, and the results are shown in table 4.

Table 4. Categorical assessment of student comments including number of responses (n).

Done Well

n 2013 Flipped n Lessons Learned

14 Entertaining/enthusiasm/engaging 20 Entertaining/enthusiasm/engaging

6 Good knowledge 9 Good knowledge/examples

6 Liked group presentations/discussions 4 Liked group presentations/discussions

Improvements

n 2013 Flipped n Lessons Learned

9 More help on

projects/expectations/assignments

8 More help on projects/expectations/

assignments

2 Dislike online quizzes 5 Dislike online quizzes

4 Go through study guides & reviews 4 Go through study guides & reviews

1 Less PowerPoint by professor

1 Some student groups were not good

presenters

Generally the comments were very positive for the class with a high degree of

satisfaction for the involvement of current events and live demonstrations of content during class

time. The professor observed a high degree of involvement in the class and student attendance

was overall at an excellent level.

The online quizzes continue to be a source of student angst. Reducing the level of

importance from 500 to 200 points actually increased the negative comments. Even though it

was emphasized throughout the semester that the quizzes are out of the textbook and the

textbook is required reading the students did not engage with this evaluation tool on a positive

basis.

Professor’s Lessons Learned Observations

One of the major issues of the 2013 Flipped classroom was the disengagement of students

from the class when their group wasn’t involved and/or when the student presentations were not

engaging. The strategy of engaging the groups in discussions rather than student presentations

worked extremely well in this regard. If the professor observed a disengaged individual or a lack

of group involvement attention was then placed on the individual or group and they were then

engaged in the discussion. Classroom management was more positive with the discussion

strategy.

Each week all groups worked on the learning outcomes. This led to more consistent

engagement of the class rather than the “it’s not my week” mentality. It also led a positive

classroom environment because the professor engaged with the entire class rather than individual

groups.

Another issue with the 2013 Flipped classroom strategy is the lack of basic knowledge of

the subject matter which can lead to frustration of the students. This issue was addressed with a

conventional, highly focused lecture on Tuesday and the online quiz due on Tuesday night. The

students then had a basic knowledge base to prepare for Thursday’s discussion and focused

flipped classroom.

The professor observed a higher level of insightful and applied analysis of basic

marketing principles throughout the semester from the students. Also, since the formal student

presentations were changed to discussions the level of mistakes observed in student classroom

discussions were much less in the Lessons Learned class than the previous semester in the

flipped class. Finally, the discussion format of the Lessons Learned class created a more positive

classroom environment than the previous flipped strategy of student presentations. The

professor was more involved in the Lesson Learned class student discussions each week and

could more readily guide the subject matter discussions as compared to the flipped class where

the students presented subject matter. This dynamic was a major improvement.

Student discussions in class also led to more compare and contrast exercises between the

different groups. For example, the Samsung and Apple groups would discuss current marketing

campaigns in the context of the weekly subject matter. They would compare and contrast the

different marketing strategies that the companies were implementing and evaluate the perceived

effectiveness of the marketing strategies. This led to deeper understanding of the key marketing

concepts as indicated in the individual marketing project and final exam scores.

Conclusions

The Lessons Learned class showed a marked improvement in measured learning

outcomes as measured by the final exam and individual marketing project scores. These two key

measures are an indication of the success of achieving the learning outcomes because they are

capstone learning outcomes where the student has to show their comprehension and application

of the subject matter.

The Lessons Learned class was not as enthusiastic as the 2013 Flipped class format about

the way the class was taught and reverted to the mean of the professor’s student evaluations.

However, in the qualitative part of the student surveys the positive comments were more

numerous than previous surveys for the professor.

The improvement in the final semester evaluations of the final exam and the individual

marketing plan project showed that even with reduced student satisfaction scores statistically

significant improvements in key learning outcomes can be a positive result of a judiciously

implemented flipped classroom strategy.

Reference

Krueger, G., & Storlie, C., (2014). Flipping an Introduction to Marketing Course: A Case Study.

2014 Hawaii International Conference on Business, (Conference Proceedings).

Appendix 1—Student Evaluation DEPARTMENT OF BUSINESS AND ACCOUNTING Course:

___________________________

Evaluation of Instructor (v1) Instructor:

_________________________

As a University, we consider student evaluations of instruction as an important element. Your input is valuable.

Please evaluate this course. This evaluation is confidential. Instructors do not have access to student evaluations of

teaching until final course grades have been turned in.

PLEASE USE PENCIL • DO NOT WRINKLE OR FOLD THIS FORM

Fill in the letter that best describes your perception of the instruction of this course.

A - I strongly agree with this statement.

Str

on

gly

Ag

ree

Ag

ree

Neu

tral

Dis

agre

e

Str

on

gly

Dis

agre

e B - I agree with this statement

C - I have a neutral opinion with this statement

D - I disagree with this statement.

E - I strongly disagree with this statement.

1. Uses speech that is understandable.

2. Is well prepared for class.

3. Uses examples and illustrations that clarify the material.

4. Explains course material clearly.

5. Gives exams, quizzes or other assessments that are related to assigned material.

6. Shows enthusiasm for subject material.

7. Incorporates current material into the course.

8. Communicates effectively with students.

9. Encourages interaction from students.

10. Presentations/discussions enhance the text.

11. Sets high standards for students.

12. Explains evaluation criteria.

13. Graded work is returned within two weeks.

14. Assigns homework, projects, casework or papers.

15. Uses class time for student discussions, presentations or questions.

16. Presents material in an organized manner.

17. Starts on time for class.

18. Covers the material stated in the syllabus.

PLEASE RESPOND TO THESE THREE STATEMENTS

A. Identify specific things your instructor has done especially well in teaching this course.

B. Identify specific things you believe could be done to improve the teaching of this course.

C. Other constructive comments: