using a smartphone-based integrated data … senyo douglas gloag shigeru asano takao hashizume koji...
Post on 31-Mar-2019
220 Views
Preview:
TRANSCRIPT
Katsumi Senyo Douglas GloagShigeru Asano Takao HashizumeKoji Fujiwara Jun-ichiro Yasuda
October 22, 2018
2018 Assessment Institute
Using a Smartphone-based Integrated Data Collection System to Measure Student Learning Gains
Go to Google and search “Yamagata OIRE”
Or type https://ir.yamagata-u.ac.jp/AI2018.pdfOr scan with QR Code Reader
1
Slides are available!
Here!
2
Agenda
1. About Us2. Key Characteristics of
Higher Education in Japan3. Data Collection via smartphone4. CBT Assessment for General Ed –
Development & Analysis5. FYE Program / Big 5 personalities6. Results (Grades / Attendance)7. Conclusions
Douglas
Douglas
Jun-ichiro
Takao
3
Where is Yamagata, Japan?
Direct Distance from Tokyo: 190 miles• 3 hours by bullet train• 45 minutes by airplane
4
Yamagata University (YU)
■Type: National University• multi-campus (4 campuses)
■Students: 8,800 • 7,500 undergraduates
• 1,700 new incoming/year• 1,300 graduates
■Full-time Faculty: 900
■Six Distinct Colleges:• Agriculture• Literature and Social Science• Education, Arts and Science• Sciences• Medicine• Engineering
• Yamagata City – Tsuruoka City: 60 miles• Yamagata City – Yonezawa City: 30 miles
5
Yamagata University (YU)
Degree Programs Undergraduate: 22 Master: 19 Doctorate: 10
2017-2018 FT Enrollment & FT Faculty/Staff Degree-seeking UG: 7,546
• Male: 62% Degree-seeking, first-time freshmen: 1,731
• In-state: 23% Degree-seeking Grad: 1,394
• Male: 75% Faculty & Staff: 800+ & 300+, respectively
Same Tuition$4,800/year
6
Key Characteristic of H.E. in Japan
The academic year begins in April and ends the following March.
Students must decide on their major when they apply for admission.• Can’t change the major after enrollment• No double or more majors• No pre- program (e.g., Pre-Med, Pre-Vet, etc.)• Almost impossible to
Transfer to another university Have flexible study plan
High expectation to graduate in four years
7
Data Collection via Smartphone
8
Assessment via Smartphone
FACT: 99+% of YU students own a smartphone with Bluetooth4.0.
We would like to…a. Collect assessment data more effectively.b. Provide the personalized info. more efficiently.
• At YU, the data/info. is not centralized.c. Track classroom attendance more cost-effectively.
• Previously:NFC-based reader & IC chip-embedded student ID card
• Ongoing:Bluetooth4.0(BLE) based beacon & students’ smartphone
Much info. collected & provided
9
Attendance Tracking at YU
Current MethodNFC-based reader
IC chip-embedded student ID card
✚
NFC = Near-field Communication
10
Smartphone Application: YU Portal
Important Info. is now centralized!!
11
Main Features
Test Survey Personalized Class Schedule Class Attendance Record Personalized Notification
• e.g., Class Cancellations
ClassSchedule
Attendance Record
Semester Course Schedule
Test
12
Our TEST: CBT Assessment for General Education
- Development and Analysis -
13
TEST Objective
• Assess learning outcome directly & effectiveness of education program Senior
Sophomore, Junior (Specialized Ed)
Freshman (General Ed)
Assessment 1
Assessment 2
Assessment 3
14
Schedule
Planning, Pre-survey
Improvement, Pre-test
TEST
to Freshman
TEST to Freshman, Sophomore
2015 2018Apr
2017Apr2016
• 1,700 students for each grade
15
TEST Development
Subjects Statistics, Mathematics, Physics, Chemistry,
Biology Created 30~45 questions for each subject
Purpose To assess not only memorization but also
understanding of scientific concepts
16
Resources
Our TEST
Past exam
Our original questions
Diagnostic
test
Physics Force Concept
Inventory
Chemistry Concept test
Biology Bio- literacy
etc.
17
Example of Question (Physics)
Q. Which is the correct force direction in a parabolic motion?
18
TEST Design
1. Assessment via smartphone application
2. Use of Computerized Adaptive Testing (CAT) Questions are selected based on a response
Computerized Adaptive Testing
Q.1
Q.2(Difficult)
Q.2(Easy)
correct
incorrect ・・・
・・・
19
TEST Design
1. Assessment via smartphone application
2. Use of Computerized Adaptive Testing (CAT) Questions are selected based on a response
3. Students answer 5 questions for each subject
20
Computerized Adaptive Testing
With 5 questions, students’ abilities can be graded in 32 levels
21
TEST Design
1. Assessment via smartphone application
2. Use of Computerized Adaptive Testing (CAT) Questions are selected based on a response
3. Students answer 5 questions for each subject4. Time limit is 3 minutes for each question Test time is about 30 minutes for 5 subjects
22
Item Response Theory
1. Estimate the difficulty parameters of each question in a pre-test
23
Schedule
Planning, Pre-survey
Improvement, Pre-test
TEST
to Freshman
TEST to Freshman, Sophomore
2015 2018Apr
2017Apr2016
• 1,700 students for each grade
• To ~ 60 students (various majors Freshmen in December)
24
Item Response Theory
1. Estimate the difficulty parameters of each question in a pre-test
2. Estimate the ability parameters (θ) of examinees in our TEST
3. We used 1 parameter (Rasch) model
25
TEST Examination Rate
Statistics Math Physics Chemistry Biology
Examinee All Only science course students
Class of 20171st time
99.3% 99.0% 99.0% 99.0% 99.3%
Class of 20181st time
99.2% 99.6% 99.6% 99.6% 99.5%
Class of 20172nd time
89.2% 90.5% 90.2% 90.2% 90.6%
26
Class of 2017: 1st -time vs 2nd ; Chemistryability θ 1st
ability θ 2nd
ability θ diff. p d
all -0.19 0.09 0.28 0.00 0.33A -0.14 0.31 0.45 0.00 0.50
B 0.38 0.44 0.06 0.55 0.08
C -0.48 -0.56 -0.09 0.39 -0.13
D -0.22 0.50 0.72 0.00 0.81
E -0.05 0.32 0.37 0.00 0.46
F -0.39 -0.43 -0.04 0.54 -0.06
G -0.16 0.18 0.34 0.00 0.40
H -0.36 -0.11 0.26 0.43 0.33
I -0.45 0.16 0.62 0.00 0.72J -0.45 -0.37 0.08 0.22 0.13
For Freshman in Dec.2016,θ ave. = 0θ sd. = 1
p < 0.05
Cohen’s d + -
small
middlelarge
27
ability θ 1st
ability θ 2nd
ability θ diff. p d
all -0.08 0.03 0.11 0.00 0.12A -0.13 0.12 0.25 0.00 0.27
B 0.42 0.71 0.29 0.00 0.32
C 0.61 1.00 0.39 0.00 0.42
D 0.09 0.21 0.12 0.24 0.17
E -0.40 -0.38 0.02 0.79 0.02
F -0.54 -0.43 0.11 0.59 0.09
G 0.00 0.01 0.01 0.91 0.01
H -0.56 -0.60 -0.04 0.65 -0.06
I -0.54 -0.52 0.02 0.94 0.01J -0.39 -0.19 0.20 0.36 0.34
Class of 2017: 1st -time vs 2nd ; Biology
For Freshman in Dec.2016,θ ave. = 0θ sd. = 1
p < 0.05
Cohen’s d + -
small
middlelarge
28
Class of 2017: 1st -time vs 2nd ; Mathability θ 1st
ability θ 2nd
ability θ diff. p d
all -0.10 -0.14 -0.05 0.12 -0.05A 0.02 0.03 0.01 0.94 0.01
B 0.48 0.06 -0.42 0.00 -0.57
C -0.96 -0.94 0.02 0.85 0.02
D -0.07 -0.13 -0.06 0.57 -0.07
E -0.25 -0.06 0.19 0.02 0.24
F 0.08 0.11 0.03 0.70 0.04
G -0.01 0.14 0.15 0.04 0.19
H -0.07 -0.27 -0.20 0.49 -0.23
I 0.03 -0.10 -0.13 0.48 -0.17J -0.48 -0.75 -0.27 0.00 -0.32
For Freshman in Dec.2016,θ ave. = 0θ sd. = 1
p < 0.05
Cohen’s d + -
small
middlelarge
29
Class of 2017: 1st -time vs 2nd ; Physicsability θ 1st
ability θ 2nd
ability θ diff. p d
all 0.13 0.19 0.07 0.03 0.07A 0.16 0.33 0.17 0.03 0.20
B 0.34 0.20 -0.14 0.23 -0.14
C -0.52 -0.55 -0.03 0.78 -0.04
D 0.12 0.28 0.16 0.07 0.19
E 0.04 0.11 0.08 0.32 0.09
F 0.43 0.39 -0.04 0.68 -0.05
G 0.53 0.68 0.15 0.11 0.19
H 0.43 -0.06 -0.49 0.05 -0.56
I 0.14 0.34 0.19 0.31 0.21J -0.46 -0.37 0.09 0.23 0.12
For Freshman in Dec.2016,θ ave. = 0θ sd. = 1
p < 0.05
Cohen’s d + -
small
middlelarge
30
Class of 2017: 1st -time vs 2nd ; Statisticsability θ 1st
ability θ 2nd
ability θ diff. p d
all -0.04 0.00 0.04 0.18 0.04A 0.15 0.07 -0.08 0.33 -0.09B 0.45 0.28 -0.18 0.08 -0.22C -0.30 -0.06 0.24 0.05 0.40D -0.17 0.02 0.19 0.08 0.22E -0.08 0.00 0.08 0.40 0.09F 0.00 0.12 0.12 0.22 0.13G -0.04 0.03 0.07 0.53 0.07H 0.35 0.40 0.05 0.87 0.06I -0.22 -0.17 0.06 0.73 0.06J -0.03 -0.12 -0.08 0.37 -0.09K -0.35 -0.07 0.28 0.05 0.32L -0.41 -0.48 -0.07 0.71 -0.08M -0.10 0.00 0.10 0.32 0.11N -0.07 -0.11 -0.04 0.76 -0.05O -0.27 -0.30 -0.04 0.77 -0.05
For Freshman in Dec.2016,θ ave. = 0θ sd. = 1
p < 0.05
Cohen’s d + -
small
middlelarge
31
Summary
• To assess students’ learning outcome directly,we developed computerized adaptive testsfor five scientific subjects
“Important message”• Students’ abilities are improved if YU’s
teaching and learning system works• Students’ level in a particular subject can not
be maintained if we do not provideopportunities to learn it
32
33
First-Year Experience (FYE) Program&
Big Five Personality Traits
FYE Program at YU
Course Goals1. To promote further development of
student success skills2. To help students gain confidence at
new environment (learn from experience)
Characteristics• 2 semester credits (meet 15 times, 100 min./class)• 25 students per class• Mandatory for ALL new first-year students• Target Course Grade: B or above• (Unofficial) Attendance Requirement: 80%
34
YU Start-up Seminar (SUS)
Improve students’ necessary study skills*
Characteristics• Same syllabus / slides• Same assignments and grading• Class size ~25:
Mix of major and gender• 08:50~10:20 time slot
Changes to Class of 20181. Peer/Mentor program initiated2. Slight reduction in class size
Format is aligned
* Data gathering, Discussion, Presentation, Report writing 35
Big Five Personality Test
Localized version developed by Professors Chieko and Yoshihiro Murakami 70 Yes-No Questions
Big Five Personality Traits• Openness to Experience• Conscientiousness• Extraversion• Agreeableness• Emotional Stability
(Changed from Neuroticism)
Are they related to FYE class grade and/orattendance?
36
37
Results, Comparisonsand Conclusions
Comparing Big 5 Results Class of 2017:1st time (April 2017) vs. 2nd time (April 2018)
Stability Coefficient:1st time vs 2nd time Stability Coefficient(ρ)Value showing exactness of result of the same student taking the same test again and again.
ρ gives a value from 0-1 with a score closer to 1 having a higher stability coefficient.
Big 5 Traits Class of 20171st time* 2nd time**
Mean Std. Dev. Mean Std. Dev.E: Extraversion 46.0 10.5 46.0 10.1A: Agreeableness 52.0 8.7 52.4 9.0C: Conscientiousness 54.5 9.5 54.3 9.7N: Emotional Stability 47.6 9.7 48.4 9.3O: Openness 49.4 9.7 49.8 9.8* Students meeting 2017 entry requirements(N = 1,691)
** N = 1,461
Big 5 Traits Class of 2017(ρ)*E: Extraversion 0.80A: Agreeableness 0.60C: Conscientiousness 0.68N: Emotional Stability 0.67O: Openness 0.65* All stability coefficient values significant(p < 0.0001)
39
Analysis Method
Basic Question• Is there any relation between the Big 5 and
academic achievement?
Student Data used for analysis• Attendance and grades of 1st semester
“Start-up Seminar” (2 credits)• Student GPA scores of 1st semester – Not used due
to lack of input time
Analysis scope1. Big 5 vs. “Start-up Seminar” Grades2. Big 5 vs. “Start-up Seminar” Attendance
40
Results concerning SUS Grades (1)
5 grade evaluation:A, B, C, D or F SUS is designed to give students an ‘B’ or higher if they
meet these conditions:
1. Hand in weekly assignment by deadline2. Meet specified course criteria
→ A or B ≅ Outcome accomplishment
Grade comparison:2017 vs. 2018 Evaluation Bands
Class of 2017 Class of 2018Freq. Percent Freq. Percent
Good(A or B) 1,427 84.4 1,499 87.1Poor(C, D or F) 264 15.6 222 12.9
Total 1,691 100.0 1,721 100.0
41
94%
82%
A/B C/D/F
Class of 2018
93%
82%
A/B C/D/F
Class of 2017
Results concerning SUS Grades(2)
Attendance comparison with grades
***
*** p < 0.0001
***
42
SUS Attendance
Attendance:80% expected
Attendance comparison: 2017 vs. 2018
Attendance Levels Class of 2017 Class of 2018Freq. Percent Freq. Percent
Good Attendance(80%+) 1,515 90.0 1,524 88.6Poor Attendance(<80%) 176 10.0 197 11.4Total 1,691 100.0 1,721 100.0
43
SUS: Attendance vs. Grades
***
***
*** p < 0.0001
91%
54%
9%
46%
GOOD ATTENDANCE(80%+)
POOR ATTENDANCE(<80%)
Class of 2018
A/B C/D/F
88%
53%
12%
47%
GOOD ATTENDANCE(80%+)
POOR ATTENDANCE(<80%)
Class of 2017
A/B C/D/F
44
Big 5 vs. 2017 Student Grades
60%
70%
80%
90%
100%
Extraversion Agreeableness Conscientiousness Emotional
Stability
Openness
Grade scoresof ‘A’/’B’
Class of 2017Start-up Seminar
P = 0.97 P = 0.02 P = 0.000 P = 0.04 P = 0.2
Low High← Big5 Score →
45
Big 5 vs. 2018 Student Grades
60%
70%
80%
90%
100%
Extraversion Agreeableness Conscientiousness Emotional
Stability
Openness
Grade scoresof ‘A’/’B’
Class of 2018Start-up Seminar
P = 0.07 P = 0.12 P = 0.000 P = 0.72 P = 0.003
Low High← Big5 Score →
46
Summary: Big 5 vs. SUS Grades
Grades and related Big 5 traits
Implied ResultsClass of 2017 Class of 2018
Agreeableness N/AConscientiousness Groups with high
scores in this trait get higher grades
Groups with high scores in this trait get higher grades (same trend)
Emotional Stability Groups with high scores in this trait get lower grades
Openness to experience
Groups with high scores in this trait get lower grades
47
Big 5 vs. 2017 Student Attendance
60%
70%
80%
90%
100%
Extraversion Agreeableness Conscientiousness Emotional
Stability
Openness
80%+Attendance Class of 2017
Start-up Seminar
P = 0.72 P = 0.79 P = 0.000 P = 0.59 P = 0.32
Low High← Big5 Score →
48
Big 5 vs. 2018 Student Attendance
80%+Attendance Class of 2018
Start-up Seminar
60%
70%
80%
90%
100%
Extraversion Agreeableness Conscientiousness Emotional
Stability
Openness
P = 0.32 P = 0.08 P = 0.02 P = 0.26 P = 0.000
Low High← Big5 Score →
49
Summary: Big 5 vs. SUS Attendance
Attendance related to Big 5
Implied ResultsClass of 2017 Class of 2018
Conscientiousness Clear difference in attendance with high scores having high attendance and low scoring students having lower attendance
Groups with high scores in this trait have higher attendance (almost same trend)
Openness to experience
Groups with high scores in this trait have lower attendance
50
60%
70%
80%
90%
100%
Extraversion Agreeableness Conscientiousness Emotional
Stability
Openness
Combined Data: Big 5 vs. Grades
P = 0.34 P = 0.01 P = 0.000 P = 0.13 P = 0.001
Grade scoresof ‘A’/’B’
Low High← Big5 Score →
51
60%
70%
80%
90%
100%
Extraversion Agreeableness Conscientiousness Emotional
Stability
Openness
Combined Data: Big 5 vs. Attendance
80%+Attendance
P = 0.42 P = 0.31 P = 0.000 P = 0.39 P = 0.000
Low High← Big5 Score →
52
Summary of Results and Future Plans
Personality traits that may be connected to SUS data on grades and attendance
Future Plans Develop Early-Alert (Early-Warning) System for YU
• Implement effective intervention program• Instruct students as to what is required of them
Class of 2017 Class of 2018 CombinedGrades Agreeableness?
Conscientiousness↑Emotional Stability↓
Conscientiousness↑
Openness↓
Agreeableness?Conscientiousness↑
Openness↓
Attendance Conscientiousness↑ Conscientiousness↑
Openness↓
Conscientiousness↑
Openness↓
53
Conclusions (1)
What we have done Two years data collection Understanding students’ study
preferences
What we have found Students’ abilities are improved if YU’s
teaching and learning system works Students’ level in a particular subject can
not be maintained if we do not provide opportunities to learn it
54
Conclusions (2)
What we will do Widen findings from data analysis Introduce logical reasoning questions Improve FYE class content Get more data from other universities Apply our findings to improve YU’s
education continuously
55
Acknowledgements
Special Thanks (in alphabetical order) to:• Shinobu Araki• Takahiro Abe• Takahiro Iijima• Nobuhiro Matsuzaka• Koto Sato• Tatsuya Suzuki• Eriko Watanabe• Minako Yamamoto
This project issupported by MEXT, Japan.
andAll Students, Faculty, Staff & Administrationat Yamagata University
The smartphone App is developed with
56
top related