data analytics slides - university of north carolina at

Using Student Data to Advance the University's

Teaching Mission

10:05- 10:55

Comments/Questions: pollev.com/carolina

KEVIN GUSKIEWICZDean of the College of Arts and Sciences

KELLY HOGANAssociate Dean of Instructional InnovationTeaching Associate Professor of Biology

Using data to track student outcomes: a case study in transitioning to high structure active learning

STEM classes

4 DEPARTMENTS (BIOLOGY, CHEMISTRY, PHYSICS and MATH)

12 GATEWAY COURSES TRANSFORMED

25FACULTY APPRENTICES IN SIX SEMESTERS

>6,000 STUDENTS PER YEAR

Improved Learning…

45FACULTY IN LEARNING COMMUNITIES

Presenter

Presentation Notes

KEVIN. Start with the punchline

New tools needed

PRACTICE AND FEEDBACKLISTENING TO CONTENT

Presenter

Presentation Notes

KEVIN. An instructor-centered classroom like this is how education has been done since ancient Greek times. In this student-centered model, the students work alone and together to practice doing science, while the professor interacts and guides the learning, as only the expert in the room can. The student centered model improves outcomes for all students, but especially for underrepresented students. For a professor to go from the lecture-only style of pedagogy on the left, to the student-centered model on the right, a professor needs an entirely new set of tools, often around inclusive facilitation, instructional design, technology, and learning science.

Use a new innovation

“WHAT BARRIERS EXIST?”

TIME TO DEVELOP THE COURSE.

KNOWLEDGE ABOUT PEDAGOGY

SUPPORT FROM A GROUP OF COLLEAGUES

Learn about innovation

Convinced of value of innovation

Decide to try the innovation

Stages of Innovation Process:

Presenter

Presentation Notes

KEVIN. For a faculty member to adopt a new innovation, they would first need to learn about it, be convinced of its value, decide to try it, and then use it. Literature SUGGESTS that 1/3 who try will stop using it. And another 1/3 won’t use it very effectively. Years ago, we knew we had STEM faculty willing to TRY the student-centered methods. But how to avoid losing them after trying it? Our team thought more about the barriers for faculty, including lack of time, knowledge, and support. Rogers’ stages of the Innovation Decision Process. https://journals.aps.org/prper/abstract/10.1103/PhysRevPhysEducRes.12.010110

1. MENTOR-APPRENTICE RELATIONSHIPS

2. FACULTY LEARNING COMMUNITIES

TIME (COURSE RELEASE)KNOWLEDGE/PRACTICE WITH PEDAGOGY

SUPPORT FROM COLLEAGUES DURING CULTURAL SHIFT

INTERVENTION:

Presenter

Presentation Notes

KEVIN. Funding from our project grant and the College allowed us to “buy out” out a professor’s teaching time for one semester. In this semester, they would work with a mentor in an actual classroom. The mentor had already developed all the course material, saving the apprentice TIME in the future– no need to develop more materials. The mentor provided the KNOWLEDGE and feedback needed for learning. A special piece of magic in what we did– egos were put aside– faculty learned from each other, regardless of rank and title. For example, one of our first apprentices was a distinguished professor and senior associate dean– who had been out of the classroom for many years. A lecturer, recently hired, was his mentor. So that covers the need to help professors with time and lack of knowledge. What about the support from colleagues during this cultural shift? Our faculty met monthly in faculty learning communities to talk about what was working/not working. They did what our students did– reflected on their learning and formed support networks within and across departments.

Before class

During class

After class

Readings, videos, online homework, online discussions

problem solving, individual work, peer instruction, polling/clickers, designing experiments, etc

problem sets, online quizzes

DESIRED OUTCOME FOR PEDAGOGY:High Structure Active Learning

Presenter

Presentation Notes

KELLY

Organic Chemistry I. Nearly identical final exams given.

CHEMISTRY: EVIDENCE THAT INCREASING STRUCTURE IMPROVES PERFORMANCE FOR ALL

M.T. Crimmins and B. Mdkiff J. Chem. Educ., 2017, 94 (4), pp 429–438

Additionally, D/F rates dropped from 18% to 9.5% for all students

PHYSICS: Increasing structure improves learning of specific concepts

Traditional structure

Traditional structure +

Some Active Learning

Lecture/Studio with Life Science Focus

D.P. Smith et al. American Journal of Physics 86, 862 (2018)

Presenter

Presentation Notes

https://aapt.scitation.org/doi/10.1119/1.5058685 Force Concept Inventory. PHYS 104 was the pre-transformed version of the first semester IPLS course at UNC-Chapel Hill; PHYS 114 is the transformed version. Solid grey bars are courses taught traditionally, orange bars with upward-slanting hash marks are courses taught in traditional structure and content but with some active engagement in lecture, and blue bars with downward-slanting hash marks are courses taught in Lecture/Studio mode.. Error bars represent standard error.

Preliminary findings across project:

13% increase in normalized learning gains with high structure, active learning compared to instructor centered design

(data from over 14,000 student measurements)

PHOTO CREDIT: KRISTEN CHAVEZ

Have the pedagogical/curricular changes impacted different student groups?

PHOTO CREDIT: KRISTEN CHAVEZ

Biology: Some student groups disproportionately benefit from increased structure

3.2 - 3.7 % increase

6.1 % increase

6.3 % increase

All s

tude

nts

Firs

t gen

erat

ion

stud

ents

Blac

k st

uden

ts

Eddy, S. L.; Hogan, K. A. CBE Life Sci. Ed. 2014, 13, 453-468

Failure rates dropped by 40%.

Those results required obtaining data from

institutional data requests…

What if we all had a way to see data for our own classes routinely?

Change in Grade Distribution for First Generation, Pell-Eligible Students in

One Redesigned Course (Biol 202)

0102030405060

Fall 2012 Fall 2016

A B C D F W77% successful

82 % successful

n=22 n=27

perc

ent o

f stu

dent

s

Data retrieved from MCAD

My Course Analytics Data (MCAD)

Data are available from Fall 2010 to the most recent semester during which the course was taught and completed.

Example data from the dashboard show the percent of students in different demographic categories. Hovering over the bars will give the percentages and total number in each subgroup.

Learn more: https://cfe.unc.edu/mcad/

https://cfe.unc.edu/mcad/

My Course Analytics Data (MCAD)Example data. Dashboard users customize their queries to examine individual courses in specific semesters or to combine multiple semesters.

A

A

B

B C

C D

D

Pell recipient

Non-Pell recipient

A PROFESSOR’S PERSPECTIVE

MARC COHENASSISTANT PROFESSOR

ENGLISH AND COMPARATIVE LITERATURE

Underrepresented minority

Non-Underrepresented minority

A B C

A B

Female

Male A B C

B

First generation

Non-First generation A B

A B C

MCAD IN THE CLASSROOM – AND IN THE CHAIR’S OFFICE

PATRICK CONWAY

CFE WORKSHOP

2 NOVEMBER 2018

PEDAGOGICAL PRINCIPLE

Students come to the classroom with preconceptions about how the world works.

If their initial understanding is not engaged, they may fail to grasp the new concepts and information that are taught.

National Research Council: How People Learn. Washington, DC: National Academy Press, 2000, pp 14-15

MCAD’S ROLE

• In large classes, difficult to identify those individual preconceptions.

• One partial solution: • Look to the success of various groups in previous years, and

• Adjust the class content to reach those with less success in those years.

EXAMPLE: WOMEN IN ECONOMICS

• A national trend: fewer women majoring in Economics

• 2013: look “by hand” at grades I gave in Econ 101. Women seemed to have somewhat lower grades on average.

• Response:• Changing examples and applications – more music/design/advertising.

• Identifying/highlighting successful women in economics.

• Greater emphasis on engaging women in the classroom.

• By 2015: closer to parity.

• Today: with MCAD, such queries are done in seconds.

PROBLEMS

• Loose link between pedagogical goal and available categories.

• Only available for completed courses. “Formative assessment” would be very helpful.

FROM THE CHAIR’S PERSPECTIVE

• NB: Chairs don’t have access to other faculty’s data at present.

• In courses where we have challenges – differing success rates –we can quantify these differences.

• Over time, we can experiment with teaching techniques as a department to bring success rates in line with our high expectations for our students.

• Example: Eddy, Sarah L., and Kelly A. Hogan. 2014. “Getting Under the Hood: How and for Whom Does Increasing Course Structure Work?” CBE-Life Sciences Education 13 (3): 453–68.

Discussion “rules”.

Raise your hand. (Please limit your comments to 60 seconds or less, so that many people have an opportunity to join the conversation.)Write on a notecard and signal to the moderator

for pick up.Use pollev.com/carolina for anonymous

questions/comments.

We invite you to join into the conversation in multiple ways:

data analytics slides - university of north carolina at

Documents