caphia teaching and learning forum 18-19 september 2014, perth

CAPHIA TEACHING AND LEARNING FORUM 18-19 September 2014, Perth

Jane TaylorPublic Health Program

University of the Sunshine Coast

Evaluating flipping: Development of the Flipped Classroom Student Engagement

Questionnaire (FCSEQ)

Flipped team

Jane Taylor

Public Health Course

Coordinator & Evaluator

Rachel Cole


Coordinator & Evaluator

Mary Kynn

Statistician & Movie Maker

Julie-Anne Foster

Paramedic ScienceCourse

Coordinator

Bec Tretheway

Project Research Assistant

Kara Lilly


Coordinator

18/09/2014 Jane Taylor, University of the Sunshine Coast

Exploratory Learning and Teaching Grant, 2013Centre for Support and Advancement of Learning and Teaching, USC

FCSEQ Movie

http://youtu.be/oSbLlS28OwY

Overview

• A bit about flipping

• Research aim and process

• Flipped Classroom Students Engagement Questionnaire (FCSEQ)

• Some very preliminary (limited) findings

• Comments & questions18/09/2014 Jane Taylor, University of the Sunshine Coast

What is a flipped classroom?• Pre-class online learning activities • Face-to-face workshops • Post class learning activities


http://ctl.utexas.edu/teaching/flipping_a_class/what_is_flipped

Design foundations

Bloom’s taxonomy

SOLO taxonomy

Introduction of concepts

KnowledgeComprehension Application

Pre-structural, Uni-structural, Multi-structural

Clarification and exploration of concepts

ApplicationAnalysisSynthesisEvaluation

Multi-structural,

Relational

Consolidation of concepts through assessment

ApplicationAnalysisSynthesisEvaluation

Relational,

Extended abstract


Why flip a classroom?Constructivist pedagogy• Student centred• Active and problem based learning strategies• Emphasis on student responsibility for learning• Facilitator role of teacher

Students• Learn more deeply• Greater participation• Increase in interaction and learning from one another• Greater level of instructor feedback• More time on higher levels of learning e.g., analysis, synthesis and

evaluation18/09/2014 Jane Taylor, University of the Sunshine Coast

Jane Taylor, University of the Sunshine Coast

Flipped project overview

2013, S1: Academic team foci

2013, S2: Test run in 3 classes, process feedback

2014, S1&2: Pilot in 10 classes (about 300 students) evaluation

Evaluation: Informal in class feedback , survey end course, discussion groups end course, SETAC

18/09/2014

Why this research?

• Limited literature on assessing and monitoring impacts of flipped classroom approach on student engagement and learning

• Engagement = involvement in purposeful / meaningful academic activities (Kuh, 2001)

• Aim: To develop and pilot an impact evaluation questionnaire called the Flipped Classroom Student Engagement Questionnaire, to evaluate the impact of the flipped classroom on student engagement in learning18/09/2014 Jane Taylor, University of the Sunshine Coast

Flipped Classroom Student Engagement Questionnaire

Aim• To assess students’ overall level of engagement at the course

level, and in pre, in class and post class flipped classroom components

Development1. Item generation – literature review, existing instruments and

items, peer-review, pilot test

2. Validation (currently here)

3. Finalisation


1. Item generation• Literature review

– Engagement measures at course level– Handelsman et al., 2005; Delialioglu, 2012– National Survey of Student Engagement (NSSE), 2011

• Student engagement constructs & indicators– Skills, Emotional, Participation and interaction, Performance

(Handelsman et al)– Active and collaborative learning (Delialioglu, 2012; NSSE)– Time on task (Delialioglu, 2012)– Overall level of engagement (Handelsman, et al., 2005)– Confounding variables


Jane Taylor, University of the Sunshine Coast

Construct / Indicators

Students … Examples

Skills engagement Study behaviours Keeping up to date with readings, taking notes in class, and coming to each scheduled class

Participation/interaction engagement

Participation in class and interact ion with the lecturer/tutor and fellow students

Ask questions if they don’t understand the lecturer/tutor; participate in small group discussions and overall class discussions and assisted fellow students

Emotional engagement

Connection with the course Choosing to make the course material relevant to their lives; applying the course material to their lives; finding ways to make the course interesting; thinking about the course between classes; and really desiring to learn the material.

Performance engagement

Levels of performance towards getting a good grade

Doing well in assessment tasks and asking questions, confidence that they can learn and do well in the course

Global engagement Levels of engagement between courses

Flipped compared to traditional

Active and collaborative learning

Frequency of particular learning strategies in and out of the classroom

Asking questions, completing pre-class activities, assisted other students, asked for assistance from other students, and worked with other students on course material

Total time on task Sum of time in hours doing specific learning activities

Pre-class activities, watching e-lectures, studying, reading and writing

Survey examples

Skills engagement

Participation/interaction engagement

Emotional engagement

Performance

Active and collaborative learning


Pre-testing instrument

• Peer-review– Steering Committee– Two additional academics

• Pilot test– OT class using flipped– Interpretability, usability, timeliness…


1. Item generation• Confounders (Handelsman et al)

– Entity theory of learning• Learning can be extended and is not a fixed entity (Handelsman et al)

– Incremental theory of learning• Certain amount of intelligence and you can’t do much to change it

(Handelsman et al)

– Socio-demographic characteristics • Age, sex, employment, level of education striving for; level of parent

education; international/domestic student; living arrangements, access to internet at home, confidence in using technology (NSSE)

• Assessment of flipped approach compared with usual/traditional• Perception of impact of flipped approach on academic

performance, non-academic skills, working in groups, preferences


FCSEQ structure

• Four sections1. Overall engagement2. Participation in course3. Achievement4. Socio-demographic

• 19 closed-answer questions• 2 open-ended questions• 10 minutes• Administration hard copy or electronic18/09/2014 Jane Taylor, University of the Sunshine Coast

2. Validation

• Administration: four public health courses, one paramedic course

• Electronic and hard copy administration

• June-August 2014


2. Validation

• Factor analysis may collapse one or more constructs – in progress

• How do these constructs then align with other student attributes and measures of performance?


3. Finalisation & next steps

• Finalise instrument by end October 2014

• Recruitment of more flipped classrooms by end January 2015

• Full implementation 2015

• Snippets from public health courses Sem 1 2014


Overall engagement in course across semester?

n=51


Most engaged

Overall flipped or standard classroom?

n=32


Most prefer flipped

Overall level of engagement?

n=51


Most more engaged

Overall academic performance?

n=50


Some felt it do, others not sure, some not

Pre classDoing pre class activities?

(Active & collaborative learning)

n=52


Characteristic of most to do something

Pre classHours per week preparing for workshops

(Total time on task)

n=51


Majority 1-5 hours

In-classWorkshop attendance?

(Skills engagement)

n=52


Most characteristic to attend

In-classWorkshops without pre-class activities?


n=49


Some do, some don’t

In-classHaving fun in workshops?

(Participation / interaction engagement)

n=52


Majority having fun

In-classActive participation on small groups discussion?

(Participation /interaction engagement)

n=52


Majority small group discussions characteristic

In-classExplaining materials to others?


n=50


Majority sometime/ often explained course materials to others

Post classMaking course material relevant to life?


n=52n=52


Majority found ways to make course material relevant to life

Post classRead additional materials(Active & collaborative learning)


Majority read additional material related to course content

FCSEQ qualitative feedbackThink more engaged? Interaction in class (n=9) Having a supportive class learning environment (n=7) Pre-class activities assisting in preparing for class (n=4)

Supported engagement? Getting to know teachers (n=4) More in-depth learning and expanding on content (n=3) Ensuring on-track or same page (n=3) Getting to know/ learning from fellow students (n=3) Being responsible for/being in control of own learning (n=2) × Not like pre-class activities (n=2)× 3 hour workshop too long (n=1)


Comments & Questions


Key References• Australian Council for Educational Research. Australasian survey of student

engagement: Student Engagement Questionnaire. Australia, 2011.

• Handelsman M, Briggs W, Sullivan N & Towler A. A measure of college student course engagement. The Journal of Educational Research 2005;98(3): 184-192.

• Delialioglu O. Student Engagement in Blended Learning Environments with Lecture-Based and Problem-Based Instructional Approaches. Journal of Educational Technology & Society 2012;15(3): 310.

• Biggs J. & Collis K. Evaluating the quality of learning: The SOLO Taxonomy. New York: Academic Press, 1982.

• Krathwohl D. A revision of Bloom's taxonomy: An overview. Theory into Practice. 2002;41(4): 212.


caphia teaching and learning forum 18-19 september 2014, perth

Documents

learning engagement

learning strategiesemphasis

advancement of learning

learning forum

student responsibility

sunshine coast5why

sunshine coastwhy

sunshine coast3what