analyzing open-ended student responses to the apple...
TRANSCRIPT
![Page 1: Analyzing Open-Ended Student Responses to the APPLE ...depts.washington.edu/celtweb/caee/ASEE2009/micah... · 6/9/2015 · School Advising Co-op Gender/URM Social Curriculum Language](https://reader034.vdocuments.us/reader034/viewer/2022051914/6004fe1919b4c53da835c2bf/html5/thumbnails/1.jpg)
MORE TO SAYAnalyzing Open-Ended Student Responses to the
APPLE (Academic Pathways of People Learning Engineering) Survey
Micah Lande, Sarah Parikh, Sheri Sheppard, George Toye, Helen Chen & Krista Donaldson
Stanford University
15 June 2009
American Society for Engineering Education Conference
![Page 2: Analyzing Open-Ended Student Responses to the APPLE ...depts.washington.edu/celtweb/caee/ASEE2009/micah... · 6/9/2015 · School Advising Co-op Gender/URM Social Curriculum Language](https://reader034.vdocuments.us/reader034/viewer/2022051914/6004fe1919b4c53da835c2bf/html5/thumbnails/2.jpg)
2APPLE (Academic Pathways of People Learning Engineering) Survey
Center for the Advancement of Engineering Education
Academic Pathways Study
![Page 3: Analyzing Open-Ended Student Responses to the APPLE ...depts.washington.edu/celtweb/caee/ASEE2009/micah... · 6/9/2015 · School Advising Co-op Gender/URM Social Curriculum Language](https://reader034.vdocuments.us/reader034/viewer/2022051914/6004fe1919b4c53da835c2bf/html5/thumbnails/3.jpg)
3
10 minute online survey deployed in Spring 2008, paid $4
BROAD NATIONAL SAMPLE
surveyed 4266 undergraduate students at 21 U.S. institutions
www.applesurvey.org
*
*prior to economic downturn
![Page 4: Analyzing Open-Ended Student Responses to the APPLE ...depts.washington.edu/celtweb/caee/ASEE2009/micah... · 6/9/2015 · School Advising Co-op Gender/URM Social Curriculum Language](https://reader034.vdocuments.us/reader034/viewer/2022051914/6004fe1919b4c53da835c2bf/html5/thumbnails/4.jpg)
4
OPEN-ENDED QUESTION AT END OF 50-ITEM SURVEY
Is there anything else you want to tell us that we didn’t already cover?
For inter-rater reliability
between 2 coders,
Cohen’s kappa:
moderate at .532
# of Responses
880
![Page 5: Analyzing Open-Ended Student Responses to the APPLE ...depts.washington.edu/celtweb/caee/ASEE2009/micah... · 6/9/2015 · School Advising Co-op Gender/URM Social Curriculum Language](https://reader034.vdocuments.us/reader034/viewer/2022051914/6004fe1919b4c53da835c2bf/html5/thumbnails/5.jpg)
5
20% sampled for
inter-rater reliability
between 2 coders,
Weighted kappa:
very good at .832
EMERGENT THEMATIC ANALYSIS
School
Advising
Co-op
Gender/URM
Social
Curriculum
Language
Individual Beliefs
Calling
Challenge
Future
Lifestyle
Money
Understanding
CATEGORIES:
SUB-TOPICS:
![Page 6: Analyzing Open-Ended Student Responses to the APPLE ...depts.washington.edu/celtweb/caee/ASEE2009/micah... · 6/9/2015 · School Advising Co-op Gender/URM Social Curriculum Language](https://reader034.vdocuments.us/reader034/viewer/2022051914/6004fe1919b4c53da835c2bf/html5/thumbnails/6.jpg)
6
School
Advising - student advising/mentoring, formal and informal
Co-op - co-operative work arrangements with industry
Gender/URM - gender, under-represented minorities, “other”
Social - social norms of engineering work
Curriculum - teaching pedagogy and curriculum
Language - English language proficiency
![Page 7: Analyzing Open-Ended Student Responses to the APPLE ...depts.washington.edu/celtweb/caee/ASEE2009/micah... · 6/9/2015 · School Advising Co-op Gender/URM Social Curriculum Language](https://reader034.vdocuments.us/reader034/viewer/2022051914/6004fe1919b4c53da835c2bf/html5/thumbnails/7.jpg)
7
Individual Beliefs
Calling - as life’s calling, meant to be or not meant to be
Challenge - engineering is hard, or too hard
Future - career placement, worries about after college
Lifestyle - work/life balance
Money - financial concerns
Understanding - knowing what engineering is, practice of
![Page 8: Analyzing Open-Ended Student Responses to the APPLE ...depts.washington.edu/celtweb/caee/ASEE2009/micah... · 6/9/2015 · School Advising Co-op Gender/URM Social Curriculum Language](https://reader034.vdocuments.us/reader034/viewer/2022051914/6004fe1919b4c53da835c2bf/html5/thumbnails/8.jpg)
8
HOW NEGATIVE / POSITIVE WERE THE RESPONSES?
very
negative
slightly
negative neutral
slightly
positive
very
positive
20% sampled for
inter-rater reliability
between 2 coders,
Weighted kappa:
moderate at .599
1 2 3 4 5
![Page 9: Analyzing Open-Ended Student Responses to the APPLE ...depts.washington.edu/celtweb/caee/ASEE2009/micah... · 6/9/2015 · School Advising Co-op Gender/URM Social Curriculum Language](https://reader034.vdocuments.us/reader034/viewer/2022051914/6004fe1919b4c53da835c2bf/html5/thumbnails/9.jpg)
9
School
Advising
Co-op
Gender/URM
Social
Curriculum
Language
1 2 3 4 5negative positive
![Page 10: Analyzing Open-Ended Student Responses to the APPLE ...depts.washington.edu/celtweb/caee/ASEE2009/micah... · 6/9/2015 · School Advising Co-op Gender/URM Social Curriculum Language](https://reader034.vdocuments.us/reader034/viewer/2022051914/6004fe1919b4c53da835c2bf/html5/thumbnails/10.jpg)
10
Individual Beliefs
Lifestyle
Money
1 2 3 4 5negative positive
Calling
Challenge
Future
Understanding
![Page 11: Analyzing Open-Ended Student Responses to the APPLE ...depts.washington.edu/celtweb/caee/ASEE2009/micah... · 6/9/2015 · School Advising Co-op Gender/URM Social Curriculum Language](https://reader034.vdocuments.us/reader034/viewer/2022051914/6004fe1919b4c53da835c2bf/html5/thumbnails/11.jpg)
11
“[my institution] sucks”
EXAMPLE PARTICIPANT RESPONSES:
“[my institution] rocks”
![Page 12: Analyzing Open-Ended Student Responses to the APPLE ...depts.washington.edu/celtweb/caee/ASEE2009/micah... · 6/9/2015 · School Advising Co-op Gender/URM Social Curriculum Language](https://reader034.vdocuments.us/reader034/viewer/2022051914/6004fe1919b4c53da835c2bf/html5/thumbnails/12.jpg)
12
School : Advising
“… the academic advising from [my institution]’scentral advising has been incorrect, inconsistent,and typically rubbish (at best)”
(2, Advising)
![Page 13: Analyzing Open-Ended Student Responses to the APPLE ...depts.washington.edu/celtweb/caee/ASEE2009/micah... · 6/9/2015 · School Advising Co-op Gender/URM Social Curriculum Language](https://reader034.vdocuments.us/reader034/viewer/2022051914/6004fe1919b4c53da835c2bf/html5/thumbnails/13.jpg)
13
School : Gender/URM
“As a female engineering student, I have noticedthat the engineering courses at [my institution]are generally not composed of more than 5-10%female students. For me, this can beoverwhelming at times.” (2, Gender/URM)
![Page 14: Analyzing Open-Ended Student Responses to the APPLE ...depts.washington.edu/celtweb/caee/ASEE2009/micah... · 6/9/2015 · School Advising Co-op Gender/URM Social Curriculum Language](https://reader034.vdocuments.us/reader034/viewer/2022051914/6004fe1919b4c53da835c2bf/html5/thumbnails/14.jpg)
14
“… Engineers have poor social communicationskills and don’t get girls.”
(2, Social)
School : Social
![Page 15: Analyzing Open-Ended Student Responses to the APPLE ...depts.washington.edu/celtweb/caee/ASEE2009/micah... · 6/9/2015 · School Advising Co-op Gender/URM Social Curriculum Language](https://reader034.vdocuments.us/reader034/viewer/2022051914/6004fe1919b4c53da835c2bf/html5/thumbnails/15.jpg)
15
“Engineering classes should start as soon in thecollege career as possible and should largelyconsist of hands-on and industry-relevantactivities” (3, Curriculum)
School : Curriculum
![Page 16: Analyzing Open-Ended Student Responses to the APPLE ...depts.washington.edu/celtweb/caee/ASEE2009/micah... · 6/9/2015 · School Advising Co-op Gender/URM Social Curriculum Language](https://reader034.vdocuments.us/reader034/viewer/2022051914/6004fe1919b4c53da835c2bf/html5/thumbnails/16.jpg)
16
School : Language
“It is very hard to learn from TAs who cannotspeak English very well and who cannotunderstand our questions.”
(2, Language)
![Page 17: Analyzing Open-Ended Student Responses to the APPLE ...depts.washington.edu/celtweb/caee/ASEE2009/micah... · 6/9/2015 · School Advising Co-op Gender/URM Social Curriculum Language](https://reader034.vdocuments.us/reader034/viewer/2022051914/6004fe1919b4c53da835c2bf/html5/thumbnails/17.jpg)
17
School : Co-op
“I had very little idea of what a job in engineeringconsisted of before I became a co-op student. Ithink these types of programs are crucial tocreating capable engineers.” (4, Co-op)
![Page 18: Analyzing Open-Ended Student Responses to the APPLE ...depts.washington.edu/celtweb/caee/ASEE2009/micah... · 6/9/2015 · School Advising Co-op Gender/URM Social Curriculum Language](https://reader034.vdocuments.us/reader034/viewer/2022051914/6004fe1919b4c53da835c2bf/html5/thumbnails/18.jpg)
18
: Calling
“… For me it was wondering who made cars… onceI got into school I realized I was in the rightprofession… and I love it.” (4, Calling)
Individual Beliefs
![Page 19: Analyzing Open-Ended Student Responses to the APPLE ...depts.washington.edu/celtweb/caee/ASEE2009/micah... · 6/9/2015 · School Advising Co-op Gender/URM Social Curriculum Language](https://reader034.vdocuments.us/reader034/viewer/2022051914/6004fe1919b4c53da835c2bf/html5/thumbnails/19.jpg)
19
“Engineering classes are very hard; however Icannot picture myself studying another major…Work hard or die alive!” (5, Challenge)
: ChallengeIndividual Beliefs
![Page 20: Analyzing Open-Ended Student Responses to the APPLE ...depts.washington.edu/celtweb/caee/ASEE2009/micah... · 6/9/2015 · School Advising Co-op Gender/URM Social Curriculum Language](https://reader034.vdocuments.us/reader034/viewer/2022051914/6004fe1919b4c53da835c2bf/html5/thumbnails/20.jpg)
20
“I studied engineering to go into business. I feel atechnical background with study in business willmake my goals of company ownership a reality…”
(3, Future)
: FutureIndividual Beliefs
![Page 21: Analyzing Open-Ended Student Responses to the APPLE ...depts.washington.edu/celtweb/caee/ASEE2009/micah... · 6/9/2015 · School Advising Co-op Gender/URM Social Curriculum Language](https://reader034.vdocuments.us/reader034/viewer/2022051914/6004fe1919b4c53da835c2bf/html5/thumbnails/21.jpg)
21
“Need to do a better job marketing engineeringprograms to students. Many people who would dowell in engineering do not pursue a degree simplybecause of lack of knowledge and exposure.”
(2, Understanding)
: UnderstandingIndividual Beliefs
![Page 22: Analyzing Open-Ended Student Responses to the APPLE ...depts.washington.edu/celtweb/caee/ASEE2009/micah... · 6/9/2015 · School Advising Co-op Gender/URM Social Curriculum Language](https://reader034.vdocuments.us/reader034/viewer/2022051914/6004fe1919b4c53da835c2bf/html5/thumbnails/22.jpg)
22
: Lifestyle
“… I sometimes feel as if earning an engineeringdegree requires a lot of sacrifice especially in termsof social involvement and a healthy lifestyle.”
(2, Lifestyle)
Individual Beliefs
![Page 23: Analyzing Open-Ended Student Responses to the APPLE ...depts.washington.edu/celtweb/caee/ASEE2009/micah... · 6/9/2015 · School Advising Co-op Gender/URM Social Curriculum Language](https://reader034.vdocuments.us/reader034/viewer/2022051914/6004fe1919b4c53da835c2bf/html5/thumbnails/23.jpg)
23
“… School is too expensive. I would enjoy it muchbetter if I wasn’t financially stressed about it.”
(2, Money)
: MoneyIndividual Beliefs
![Page 24: Analyzing Open-Ended Student Responses to the APPLE ...depts.washington.edu/celtweb/caee/ASEE2009/micah... · 6/9/2015 · School Advising Co-op Gender/URM Social Curriculum Language](https://reader034.vdocuments.us/reader034/viewer/2022051914/6004fe1919b4c53da835c2bf/html5/thumbnails/24.jpg)
24
CONCLUSIONS
School Individual Beliefs<negative neutral< <
![Page 25: Analyzing Open-Ended Student Responses to the APPLE ...depts.washington.edu/celtweb/caee/ASEE2009/micah... · 6/9/2015 · School Advising Co-op Gender/URM Social Curriculum Language](https://reader034.vdocuments.us/reader034/viewer/2022051914/6004fe1919b4c53da835c2bf/html5/thumbnails/25.jpg)
25
+ adding to quantitative findings from survey
+ adding student voices
+ adding issues in that not otherwise captured
+ inform future deployments of the the survey
CONCLUSIONS
![Page 26: Analyzing Open-Ended Student Responses to the APPLE ...depts.washington.edu/celtweb/caee/ASEE2009/micah... · 6/9/2015 · School Advising Co-op Gender/URM Social Curriculum Language](https://reader034.vdocuments.us/reader034/viewer/2022051914/6004fe1919b4c53da835c2bf/html5/thumbnails/26.jpg)
26
CONCLUSIONS
School Individual BeliefsWhat can the change about ?
SchoolIndividual BeliefsWhat can change about the ?
![Page 27: Analyzing Open-Ended Student Responses to the APPLE ...depts.washington.edu/celtweb/caee/ASEE2009/micah... · 6/9/2015 · School Advising Co-op Gender/URM Social Curriculum Language](https://reader034.vdocuments.us/reader034/viewer/2022051914/6004fe1919b4c53da835c2bf/html5/thumbnails/27.jpg)
27
This material is based on work supported by
the National Science Foundation under Grant
No. ESI-0227558, which funds the Center for
the Advancement of Engineering Education.
ACKNOWLEDGEMENTS
RELATED SESSIONS:Sarah Parikh, Stanford University
Monday 12:30-2pm Room 15 #1431
Does Major Matter? A Look at What Motivates
Engineering Students in Different Majors
CAEE Leadership Team
Tuesday 2:15-4pm Room 6B #2530
Findings from the Academic Pathways Study
visit CAEE Exposition Booth #437