snowbird 2018 cra data panel...university of massachusetts amherst have been chair for 3 years just...

USING CRA DATATO IMPROVE YOUR DEPARTMENTAND INFORM DECISION MAKING

Betsy Bizot and Burçin Tamer, CRAJames Allan, University of MassachusettsTracy Camp, Colorado School of Mines

Thu Nguyen, Rutgers UniversityCal Ribbens, Virginia Tech

2018 CRA Conference at Snowbird

ORGANIZATION OF PANEL

• Intro and CRA Data Overview• Taulbee Survey presentation and discussion• Data Buddies Project presentation and discussion• General Discussion

Betsy Bizot – brief bio

Education

• BS in Computer Science, Virginia Tech, 1978

• MBA, Tulane University

• PhD in Industrial/Organizational Psychology, University of Tulsa

CRA Director of Statistics and Evaluation since 2004

• Taulbee Survey

• Other CRA Data, e.g. the Enrollments Survey that fed the Generation CS report

• Non-CRA Data

• Interests In

– Higher education data, especially computing and STEM

– Diversity in STEM

– Career paths in computing

INTRODUCTION: CRA DATACRA collects data on the field of computing to fill gaps in national sources like IPEDS and NSF surveys

• Department-level (Taulbee)

• Individual-level (Data Buddies)

Cross-cutting data issues for both Taulbee and Data Buddies

• Data collection: balancing what we’d like to know against survey burden

• Reporting: Turning data into usable information

– Overview Reports

– Customized Reports

– Privacy and Confidentiality Concerns

TAULBEE SURVEY

• Taulbee background• One of the first projects of CRA, running close to 40 years now• Surveys doctoral computing departments in North America

• What do we collect with Taulbee?• Student data (all levels, emphasis on doctoral)• Faculty• Financial (salaries, research expenditures, student support)

• What do we do with it?• Annual report• Ad hoc reporting• “Peer group” or benchmarking reporting

TAULBEE BENCHMARKING• The ability to compare Taulbee numbers for your institution against a

self-selected comparison group or peer group• Available to CRA members who responded to the most recent

Taulbee survey• Carefully controlled to protect data privacy• Two types of reports available

• Salary report (since 2015, but little used)• Slide shows (new) will show other variables such as students,

faculty, diversity

See https://cra.org/conference-at-snowbird/reading-lists/

SALARY BENCHMARKING

OTHER VALUES BENCHMARKING: SINGLE YEAR STATS

JAMES ALLAN, UNIVERSITY OF MASSACHUSETTS

Chair, Information and Computer SciencesUniversity of Massachusetts Amherst

Have been chair for 3 yearsJust signed on for another 3 years

Joined the CRA board two and a half weeks agoInterested in collecting and using data for many years

Use of Taulbee data at UMass

• Faculty salary data• Competitive salary offers• Checking need for adjustments at other ranks

• Teaching load• Graduate student stipends• Course enrollment information

Particularly great things

• Data helpfully broken out• Public, rural, large, …

• Can create a “benchmarking” group• Can request data broken out by special groups

• Periodic profiles beyond salaries helpful for sense of how things stand• Feel like our requests to add items we’re interested in are heard

Ideas and challenges

• Q: What are industry starting salaries for graduates?• Q: Sizes of classes being taught (more nuanced teaching load)?• Q: What is range/nature of startup packages being offered?• Difficult to be sure what to ask, but being considered by Taulbee committee

• CHALLENGE: Inconsistent reporting by departments• Some answers seem too out of line with normal

• CHALLENGE: Data can feel out of date, particularly for hiring• IDEA: Dynamic “live” Taulbee, updated every week• (Not really. Well, maybe. Nah, not possible. But what if…)

Cal Ribbens – Brief Bio

Professor in the Department of Computer Science at Virginia Tech

n Joined the faculty at Virginia Tech in 1987n Department Head since 2015n Associate Department Head beginning in 2003

Cal Ribbens

On Using Taulbee Benchmarkingn Peer group definitionn What data is useful? Potentially all of it, including year over year trendsn External audiences: administrators, advisory board, donorsn Internal audiences: faculty, department leadership

¨ setting realistic goals and expectations¨ strategic planning and adjustments¨ motivation

Cal Ribbens

Taulbee Data Discussion

(can continue in general discussion if needed)

CRA Center for Evaluating the Research Pipeline Annual Data Buddies Survey for Students

Burçin ”Boorchin” Tamer – Brief Bio

• Director of Center for Evaluating the Research Pipeline (CERP)

• Social Scientist

• Collect data, analyze data, visualize data with the goal of creating resources for evidence based broadening participation activities

Visit http://cra.org/cerp/data-buddies/ for more information and sign-up for DBS.

CRA Center for Evaluating the Research Pipeline Annual Data Buddies Survey for Students

What is Data Buddies?

• Collects data annually from undergraduate and graduate students• Student satisfaction with degree program• Career aspirations• Perceptions of professional environment and support

structures• More!

Visit http://cra.org/cerp/data-buddies/ for more information and sign-up for DBS.

CRA Center for Evaluating the Research Pipeline Annual Data Buddies Survey for Students

How does it work?

1. Departments sign-up2. Send an email to your students with a survey link 3. CERP does the rest

Visit http://cra.org/cerp/data-buddies/ for more information and sign-up for DBS.

CRA Center for Evaluating the Research Pipeline Annual Data Buddies Survey for Students

What is it good for?

• You will receive a customized report containing your students’ data compared to other similar institutions' students

• Some uses for Data Buddies data• Evaluating programs and changes in department• Learning about undergraduate and graduate students• Used in internal discussions• Included in institutional reporting

Visit http://cra.org/cerp/data-buddies/ for more information and sign-up for DBS.

Tracy Camp - Brief BioProfessor of CS at Colorado School of Mines

● Chair since July 2016 (when CS dept split from EE) ● Former CRA Board member (led Generation CS)

Member of CRA-W Board (CRA-W Co-Chair 2011-14) ● Help launched CERP

Center for Evaluating the Research Pipeline ● CERP deploys annual CRA Data Buddies Survey

Data Buddies Background CRA-W evaluated programs with pre/post surveysCRA-W challenged to improve program evaluation

goal: compare success rates of program participants with non-participants

NSF Grant to CRA-W in Fall 2010

1st round of data collected Spring 2011 46 departments

706 undergraduate students (now 7991) 555 graduate students (now 4290)

CENTER FOR EVALUATING THE RESEARCH PIPELINE

Computing Research Association

# of Survey Respondents

F2015 F2016 F2017Graduate Students 11 16 14Undergraduate Students 54 62 133

# of Survey Respondents

F2015 F2016 F2017Graduate Students 11 16 14Undergraduate Students 54 62 133

Fall 2017women (n=27)

minorities (n=14)

What is the highest degree you plan to attain?

Women Men

Bachelor’s degree (significant) 52% 26%

Plans L

I am confident that I can: (1) Strongly disagree - (5) Strongly agree Women Men

complete my undergraduate degree in computing (significant) 3.15 3.71

Confidence:complete degree L

Within your computing department and/or classes, how often do you feel that: (1) Never - (5) Always

Majority URM

Your ideas or opinions are minimized or ignored. (significant) 1.53 2.07

Perceptions:ideas/opinions ignored L

Shared with Dean (significant)

Question MinesComparison

group

I am confident that I can complete my undergraduate degree in computing

4.89 (0.32) 4.51 (0.85)

Overall, I am satisfied with the computing program at my institution

4.33 (0.85) 3.90 (1.07)

The department is NOT very supportive of its students

1.81 (0.93) 2.43 (1.10)

My department cares about its students 4.37 (0.71) 3.73 (1.00)

Who do you consider to be a mentor? (prof within my department)

61% 40%

I am confident that I can:(1) Strongly disagree - (5) Strongly agree

Mines Comparison group

Become a leader in the field of computing (NOT significant 2015) 3.41 3.16

Become a leader in the field of computing (significant 2016) 3.52 3.12

Become a leader in the field of computing (significant 2017) 3.91 3.42

Confidence:become a leader

Rate how you feel about the environment of the department of your computing program.(1) Strongly disagree - (5) Strongly agree

Mines(2017)

Compare group(2017)

I feel a sense of community in my department. (significant 2015, 2016, 2017)

4.32 3.66

My department cares about its students. (significant 2015, 2016, 2017) 3.95 3.62

Department Environment

Thu Nguyen - Brief BioProfessor of Computer Science at Rutgers University

● Chair since July 2017● Associate Chair July 2012 to June 2017● Chair of Department’s Undergraduate Curriculum Committee● Escalating responsibilities for Rutgers CS education programs● Became involved with soliciting student participation in the CRA Data

Buddies Survey in 2014● Research and applied work for scaling and increasing diversity in CS

education

34

Undergraduate Student Body● Originally, we targeted students in our introductory classes

○ Easy access to large numbers of students■ CS1: several hundreds to a thousand■ CS2: several hundreds

● Too early in CS program for some of the questions● We were also surveying CS1 and CS2 students for research on increasing

diversity and scalability, and in collaboration with NCWIT○ Too many surveys

● Our own surveys for CS1 and CS2, Data Buddies survey for several required sophomore and junior CS classes and declared CS majors○ Correct partition?

35

Gender Gap for Rutgers CS

36

Enrollment in CS1

Underrepresented Racial/Ethnic (URM)

37

Enrollment in CS1

Rutgers CS Undergraduate “at a Glance”

38

Confidence by Gender

39

Confidence by Race/Ethnicity

40

Perception by Gender

41

Perception by Race/Ethnicity

42

DATA BUDDIES DISCUSSION

General discussion