snowbird 2018 cra data panel...university of massachusetts amherst have been chair for 3 years just...
TRANSCRIPT
USING CRA DATATO IMPROVE YOUR DEPARTMENTAND INFORM DECISION MAKING
Betsy Bizot and Burçin Tamer, CRAJames Allan, University of MassachusettsTracy Camp, Colorado School of Mines
Thu Nguyen, Rutgers UniversityCal Ribbens, Virginia Tech
2018 CRA Conference at Snowbird
ORGANIZATION OF PANEL
• Intro and CRA Data Overview• Taulbee Survey presentation and discussion• Data Buddies Project presentation and discussion• General Discussion
Betsy Bizot – brief bio
Education
• BS in Computer Science, Virginia Tech, 1978
• MBA, Tulane University
• PhD in Industrial/Organizational Psychology, University of Tulsa
CRA Director of Statistics and Evaluation since 2004
• Taulbee Survey
• Other CRA Data, e.g. the Enrollments Survey that fed the Generation CS report
• Non-CRA Data
• Interests In
– Higher education data, especially computing and STEM
– Diversity in STEM
– Career paths in computing
INTRODUCTION: CRA DATACRA collects data on the field of computing to fill gaps in national sources like IPEDS and NSF surveys
• Department-level (Taulbee)
• Individual-level (Data Buddies)
Cross-cutting data issues for both Taulbee and Data Buddies
• Data collection: balancing what we’d like to know against survey burden
• Reporting: Turning data into usable information
– Overview Reports
– Customized Reports
– Privacy and Confidentiality Concerns
TAULBEE SURVEY
• Taulbee background• One of the first projects of CRA, running close to 40 years now• Surveys doctoral computing departments in North America
• What do we collect with Taulbee?• Student data (all levels, emphasis on doctoral)• Faculty• Financial (salaries, research expenditures, student support)
• What do we do with it?• Annual report• Ad hoc reporting• “Peer group” or benchmarking reporting
TAULBEE BENCHMARKING• The ability to compare Taulbee numbers for your institution against a
self-selected comparison group or peer group• Available to CRA members who responded to the most recent
Taulbee survey• Carefully controlled to protect data privacy• Two types of reports available
• Salary report (since 2015, but little used)• Slide shows (new) will show other variables such as students,
faculty, diversity
See https://cra.org/conference-at-snowbird/reading-lists/
SALARY BENCHMARKING
OTHER VALUES BENCHMARKING: SINGLE YEAR STATS
JAMES ALLAN, UNIVERSITY OF MASSACHUSETTS
Chair, Information and Computer SciencesUniversity of Massachusetts Amherst
Have been chair for 3 yearsJust signed on for another 3 years
Joined the CRA board two and a half weeks agoInterested in collecting and using data for many years
Use of Taulbee data at UMass
• Faculty salary data• Competitive salary offers• Checking need for adjustments at other ranks
• Teaching load• Graduate student stipends• Course enrollment information
Particularly great things
• Data helpfully broken out• Public, rural, large, …
• Can create a “benchmarking” group• Can request data broken out by special groups
• Periodic profiles beyond salaries helpful for sense of how things stand• Feel like our requests to add items we’re interested in are heard
Ideas and challenges
• Q: What are industry starting salaries for graduates?• Q: Sizes of classes being taught (more nuanced teaching load)?• Q: What is range/nature of startup packages being offered?• Difficult to be sure what to ask, but being considered by Taulbee committee
• CHALLENGE: Inconsistent reporting by departments• Some answers seem too out of line with normal
• CHALLENGE: Data can feel out of date, particularly for hiring• IDEA: Dynamic “live” Taulbee, updated every week• (Not really. Well, maybe. Nah, not possible. But what if…)
Cal Ribbens – Brief Bio
Professor in the Department of Computer Science at Virginia Tech
n Joined the faculty at Virginia Tech in 1987n Department Head since 2015n Associate Department Head beginning in 2003
Cal Ribbens
On Using Taulbee Benchmarkingn Peer group definitionn What data is useful? Potentially all of it, including year over year trendsn External audiences: administrators, advisory board, donorsn Internal audiences: faculty, department leadership
¨ setting realistic goals and expectations¨ strategic planning and adjustments¨ motivation
Cal Ribbens
Taulbee Data Discussion
(can continue in general discussion if needed)
CRA Center for Evaluating the Research Pipeline Annual Data Buddies Survey for Students
Burçin ”Boorchin” Tamer – Brief Bio
• Director of Center for Evaluating the Research Pipeline (CERP)
• Social Scientist
• Collect data, analyze data, visualize data with the goal of creating resources for evidence based broadening participation activities
Visit http://cra.org/cerp/data-buddies/ for more information and sign-up for DBS.
CRA Center for Evaluating the Research Pipeline Annual Data Buddies Survey for Students
What is Data Buddies?
• Collects data annually from undergraduate and graduate students• Student satisfaction with degree program• Career aspirations• Perceptions of professional environment and support
structures• More!
Visit http://cra.org/cerp/data-buddies/ for more information and sign-up for DBS.
CRA Center for Evaluating the Research Pipeline Annual Data Buddies Survey for Students
How does it work?
1. Departments sign-up2. Send an email to your students with a survey link 3. CERP does the rest
Visit http://cra.org/cerp/data-buddies/ for more information and sign-up for DBS.
CRA Center for Evaluating the Research Pipeline Annual Data Buddies Survey for Students
What is it good for?
• You will receive a customized report containing your students’ data compared to other similar institutions' students
• Some uses for Data Buddies data• Evaluating programs and changes in department• Learning about undergraduate and graduate students• Used in internal discussions• Included in institutional reporting
Visit http://cra.org/cerp/data-buddies/ for more information and sign-up for DBS.
Tracy Camp - Brief BioProfessor of CS at Colorado School of Mines
● Chair since July 2016 (when CS dept split from EE) ● Former CRA Board member (led Generation CS)
Member of CRA-W Board (CRA-W Co-Chair 2011-14) ● Help launched CERP
Center for Evaluating the Research Pipeline ● CERP deploys annual CRA Data Buddies Survey
Data Buddies Background CRA-W evaluated programs with pre/post surveysCRA-W challenged to improve program evaluation
goal: compare success rates of program participants with non-participants
NSF Grant to CRA-W in Fall 2010
1st round of data collected Spring 2011 46 departments
706 undergraduate students (now 7991) 555 graduate students (now 4290)
CENTER FOR EVALUATING THE RESEARCH PIPELINE
Computing Research Association
# of Survey Respondents
F2015 F2016 F2017Graduate Students 11 16 14Undergraduate Students 54 62 133
# of Survey Respondents
F2015 F2016 F2017Graduate Students 11 16 14Undergraduate Students 54 62 133
Fall 2017women (n=27)
minorities (n=14)
What is the highest degree you plan to attain?
Women Men
Bachelor’s degree (significant) 52% 26%
Plans L
I am confident that I can: (1) Strongly disagree - (5) Strongly agree Women Men
complete my undergraduate degree in computing (significant) 3.15 3.71
Confidence:complete degree L
Within your computing department and/or classes, how often do you feel that: (1) Never - (5) Always
Majority URM
Your ideas or opinions are minimized or ignored. (significant) 1.53 2.07
Perceptions:ideas/opinions ignored L
Shared with Dean (significant)
Question MinesComparison
group
I am confident that I can complete my undergraduate degree in computing
4.89 (0.32) 4.51 (0.85)
Overall, I am satisfied with the computing program at my institution
4.33 (0.85) 3.90 (1.07)
The department is NOT very supportive of its students
1.81 (0.93) 2.43 (1.10)
My department cares about its students 4.37 (0.71) 3.73 (1.00)
Who do you consider to be a mentor? (prof within my department)
61% 40%
I am confident that I can:(1) Strongly disagree - (5) Strongly agree
Mines Comparison group
Become a leader in the field of computing (NOT significant 2015) 3.41 3.16
Become a leader in the field of computing (significant 2016) 3.52 3.12
Become a leader in the field of computing (significant 2017) 3.91 3.42
Confidence:become a leader
Rate how you feel about the environment of the department of your computing program.(1) Strongly disagree - (5) Strongly agree
Mines(2017)
Compare group(2017)
I feel a sense of community in my department. (significant 2015, 2016, 2017)
4.32 3.66
My department cares about its students. (significant 2015, 2016, 2017) 3.95 3.62
Department Environment
Thu Nguyen - Brief BioProfessor of Computer Science at Rutgers University
● Chair since July 2017● Associate Chair July 2012 to June 2017● Chair of Department’s Undergraduate Curriculum Committee● Escalating responsibilities for Rutgers CS education programs● Became involved with soliciting student participation in the CRA Data
Buddies Survey in 2014● Research and applied work for scaling and increasing diversity in CS
education
34
Undergraduate Student Body● Originally, we targeted students in our introductory classes
○ Easy access to large numbers of students■ CS1: several hundreds to a thousand■ CS2: several hundreds
● Too early in CS program for some of the questions● We were also surveying CS1 and CS2 students for research on increasing
diversity and scalability, and in collaboration with NCWIT○ Too many surveys
● Our own surveys for CS1 and CS2, Data Buddies survey for several required sophomore and junior CS classes and declared CS majors○ Correct partition?
35
Gender Gap for Rutgers CS
36
Enrollment in CS1
Underrepresented Racial/Ethnic (URM)
37
Enrollment in CS1
Rutgers CS Undergraduate “at a Glance”
38
Confidence by Gender
39
Confidence by Race/Ethnicity
40
Perception by Gender
41
Perception by Race/Ethnicity
42
DATA BUDDIES DISCUSSION
General discussion