Holistic Review in Graduate Admissions: What We Need to Know

Tracking Student Access to High-Impact PracticesDiversity in the Biomedical Research Workforce Webinar Series

Wednesday, March 8, 1:00-2:00 p.m. ET

If you cannot hear the audio, check the Audio pane on the control panel. You can use your speakers or dial-in using your telephone.You may ask questions at any time using the chat box.Handouts

Welcome

Jennifer C. Danek, MDSenior Director, Urban Universities for HEALTHUSU/APLUSummary of USU/APLU Efforts

Webinar series on Diversity in the Biomedical Research WorkforceShare findings from the USU/APLU/AAMC report (July 2016)Upcoming webinars:www.uuhealth.org/our-work/upcoming-events Overview

Improving Diversity in the Scientific Workforce: Actions for Improving EvidenceCollaborative effort of APLU/USU and AAMC, supported by NIH 70 experts from 28 universities/academic medical centersIdentify research actions for improving evidenceExamined four areas: Diverse Faculty Hiring and AdvancementLeadership, Organizational Change, and Climate;Diverse Student Success;Recruitment and Admissions

5

Tia Brown McNair, Ed.D.Vice President of the Office of Diversity, Equity, and Student Success, Association of American Colleges & UniversitiesReview of the Evidence

Tia McNair, Ed.D.Vice President for Diversity, Equity, and Student Successmcnair@aacu.orgEvidence for high-impact practices (HIPs) and how they are linked with student success

7

Intentionality

8

Critical QuestionsHow can HIPs help students become intentional learners?What do you want to accomplish by focusing on the design and development of HIPs? What are the outcomes?What HIPs currently exist on your campus or in your program? What do you know about who participates? Who has access? Who doesnt?

9

Intentionality of HIPs

10

What Makes a Practice High-Impact?

11

HIPs: Eight Key ElementsPerformance Expectations Set at Appropriately High LevelsSignificant Investment of Time and Effort by Students Over an Extended Period of TimeInteractions with Faculty and Peers about Substantive MattersExperiences with Diversity Frequent, Timely and Constructive FeedbackStructured Opportunities to reflect and Integrate LearningOpportunities to Discover Relevance of Learning Through Real-World ApplicationsPublic Demonstration of CompetenceSource: Kuh, George D., and Ken ODonnell. 2013. Ensuring Quality and Taking High-Impact Practices to Scale. Washington, DC: Association of American Colleges and Universities.

12

HIPs Results of NSSE 2016

60% of first-year students surveyed participated in one HIP, with 7% of them participating in two or more HIPs.

90% of 2016 NSSE Seniors participated in one HIP, with 68% of them participating in two or moreSource: NSSE 2016 High-Impact Practices (institutional report)

Over 1.3 million first-year and senior students from 557 institutions (530 in the US and 27 in Canada) were invited to participate in NSSE 2016. Of this population, 311,086 students responded to the survey. Less than half (45%) of these were first-year students and 55% were seniors. The results presented are from 537 institutions512 in the US and 25 in Canada that participated in NSSE 2016. (http://nsse.indiana.edu/2016_institutional_report/pdf/NSSE_overview_2016.pdf)13

NSSE HIPs 2016 in relation to certain high-impact experiences

The data to the right includes compares the percentage of students from NSSE 2016 who participated in a High-Impact Practice, including the percentage who participated overall (at least one, two or more), with experiences in internships, undergraduate research, capstone courses, and learning communities highlighted.

Source: NSSE 2016 High-Impact Practices (institutional report)

14

Ensuring Quality & Taking High-Impact Practices to ScaleProportionately fewer first-generation students, black and Hispanic students, and transfer students do research with a faculty member, study abroad, do an internship, or have a culminating senior experience. (Kuh & ODonnell, 2013)

15

Source: NSSE 2016 HIPs Institutional Report

This data comes from NSSE 2016 High-Impact Practices Institutional Report. Over 1.3 million first-year and senior students from 557 institutions (530 in the US and 27 in Canada) were invited to participate in NSSE 2016. Of this population, 311,086 students responded to the survey. Less than half (45%) of these were first-year students and 55% were seniors. 16

What are the effects of participation in certain high-impact experiences?

What are the effects of participation in multiple high-impact experiences?

Finley & McNair, Assessing Underserved Students Engagement in High-Impact Practices, 2013.

Finley & McNair, Assessing Underserved Students Engagement in High-Impact Practices, 2013. Data from 38 institutions (CA,OR, WI)NSSE data from 2006-2008 (one year of data from each campus)HIPs examinedLearning CommunitiesService learningStudy AbroadInternshipCapstoneStudent/Fac. Res.0-6 experiences20,000+ studentsFirst Year = 36.5%, SR = 51.1% (included Soph., Jr, Unclassified)Transfer = 33%First-generation = 51%RaceWhite= 58.7%African American = 2.5%Asian American = 11.8%Hispanic = 13.2%(Other = 6.2%, No Response = 7.5%)

17

Outcomes ExaminedDeep Learning = Pursuit of learning beyond memorization to seek underlying meanings & relationshipsGains in General Education = Writing/speaking skills, acquire broad general educ, analyzing quant. probsGains in Practical Competence = Work related knowledge & skills, working effectively w/ others, use of technology, quant. problem-solving, solving complex real-world problems Gains in Personal & Social Development = Developing ethics, understanding diff. bkgrds, understanding self, contributing to community, votingFinley & McNair, Assessing Underserved Students Engagement in High-Impact Practices, 2013.

18

HIP Participation vs. No Participation:Avg. Boost Across All Outcomes

Finley & McNair, Assessing Underserved Students Engagement in High-Impact Practices, 2013.

19

Effect of participation in Multiple HIPs on Outcomes

Standardiuzed Scores (0-100)

20

What is the effect of participation in multiple HIPs relative to students in the same group who do NOT participate?

21

Avg % Increase in Outcomes w/ Participation in Multiple HIPs Vs. No Participation (by First-Generation & Transfer Status)Finley & McNair, Assessing Underserved Students Engagement in High-Impact Practices, 2013.

22

Bettina Huber, Ph.D.Director of the Office of Institutional Research, California State University, NorthridgeUsing NSSE Data To Determine How High Impact Practices Affect Student Success: Some Preliminary Insights From Cal State Northridge

Table 1. Percentage of CSUN Senior Respondents Attempting Selected High Impact Practices (Spring 2007 NSSE Respondents)

Table 2. CSUN Senior NSSE Respondents' Participation in High Impact Practices (Spring 2007 NSSE Responses)

SHORTCOMINGS TO INITIAL APPROACHLimited number of participants drawn from one graduating class.

Little distinction in NSSE items between one-time and multiple HIP participation.

Uncertainly about how student participants interpret NSSE HIP items.

Pam Bowers, Ph.D.Associate Vice President for Planning, Assessment and Innovation, University of South CarolinaBeyond the Classroom Matters

~ 33,000 students~ 26,000 undergraduates

~ 7,000 first-year studentsUniversity of South Carolina

Carnegie Classification:RU/VHCommunity Engagement

USC OVERVIEW - Large, public research universityCarnegie Very high research activityCarnegie Community Engagement

University of South CarolinaColumbia, South Carolina

31

Beyond The Classroom MattersIncorporates records of each students involvement in non-credit, educationally purposeful support and enrichment programs into institutional data to produce a comprehensive student record.

The impact of college is largely determined by individual effort and involvement in the academic, interpersonal, and extracurricular offerings on a campus, Pascarella & Terenzini

Pascarella, E. & Terenzini, P. (2005). How college affects students (Vol. 2): A third decade of research. San Francisco: Jossey-Bass.

32

Academic Advising ProgramsAlcohol & Other Drug ProgramsCampus Activities ProgramsCampus Religious and Spiritual ProgramsCareer ServicesCivic Engagement & Service-Learning ProgramsClinical Health ServicesCommuter and Off-Campus Living ProgramsCounseling ServicesDisability Resources and ServicesEducation Abroad Programs & ServicesFinancial Aid ProgramsFraternity and Sorority Advising ProgramsHealth Promotion ServicesHousing and Residential Life Programs

International Student Programs and ServicesInternship ProgramsLearning Assistance ProgramsLesbian, Gay, Bisexual and Transgender Programs and ServicesMulticultural Student Programs & ServicesOrientation ProgramsParent and Family ProgramsRecreational Sports ProgramsSexual Violence-Related Programs & ServicesStudent Conduct ProgramsStudent Leadership ProgramsTransfer Student Programs and ServicesTRIO & Other Educational Opportunity ProgramsUndergraduate Research ProgramsVeterans and Military Programs & ServicesCAS Categories (Council for the Advancement of Standards in Higher Education)

33

34

High Impact Practices Key ElementsHigh performance expectations, clearly communicatedSignificant investment of student time and effort, over extended periodInteractions with faculty and peers on substantive mattersExperiences with diversity, people and circumstances not familiarFrequent, timely, constructive feedbackPeriodic, structured opportunities for reflection on learningReal-world application of learningPublic demonstration of competence

Kuh, G. & ODonnell, K. (2013) Ensuring Quality & Taking High Impact Practices to Scale, AAC&U.

35

36

Beyond The Classroom MattersComprehensive Student RecordStudent records are - defined in catalog entry- created by sponsoring department collected in Student Records Repository interfaced with Banner recordsmanaged in data warehouse

37

Beyond The Classroom Matters

38

Core Principles of ImprovementCarnegie Foundation Improvement Research Make the work problem-specific and user-centered. Variation in performance is the core problem to address. See the system that produces the current outcomes. We cannot improve at scale what we cannot measure. Anchor practice improvement in disciplined inquiry. Accelerate improvements through networked communities.

http://www.carnegiefoundation.org/improvement-research/approach

The Six Core Principles of Improvement1. Make the work problem-specific and user-centered. It starts with a single question: What specifically is the problem we are trying to solve? It enlivens a co-development orientation: engage key participants early and often.2. Variation in performance is the core problem to address. The critical issue is not what works, but rather what works, for whom and under what set of conditions. Aim to advance efficacy reliably at scale.3. See the system that produces the current outcomes. It is hard to improve what you do not fully understand. Go and see how local conditions shape work processes. Make your hypotheses for change public and clear.4. We cannot improve at scale what we cannot measure. Embed measures of key outcomes and processes to track if change is an improvement. We intervene in complex organizations. Anticipate unintended consequences and measure these too.5. Anchor practice improvement in disciplined inquiry. Engage rapid cycles of Plan, Do, Study, Act (PDSA) to learn fast, fail fast, and improve quickly. That failures may occur is not the problem; that we fail to learn from them is.6. Accelerate improvements through networked communities. Embrace the wisdom of crowds. We can accomplish more together than even the best of us can accomplish alone.

Student View of Records

Records are sortable by term or categoryEach entry is linked to catalog descriptionAdvisor has access to student records

40

Student can select activities (left) to create a personalizedExperiential Learning Record and can store multiple documents (below)

41

Experiential Learning Record

Each entry linked to catalog descriptionTwo ELR format options

42

Lessons LearnedIts not a technology project, primarilyRequires defining educational purpose, intentional program structureRequires calibration of language across multiple departmentsIt is a technology projectNon-Credit Records SystemsSymplicityHandshakeMaxientTutor-TracAdvisor-TracSARSOdysseyCollegiateLinkExcel, Access, Google-DocsAcademic Records SystemBanner

43

Comprehensive Student Record

Links records of educational activities for each student, within and beyond the classroom.Beyond The Classroom Matters

44

P. Kay Lund, Ph.D.Director of the Division of Biomedical Research Workforce, National Institutes of Health

NIH Diversity Priorities

Division of Biomedical Research Workforce (DBRW)Office of Extramural ProgramsOffice of Extramural ResearchOffice of the Director

NIH PerspectiveP. Kay Lund PhD

The NIH is committed to improving the diversity of the biomedical & physician scientist research workforce

Workforce composition affects the quality and impact of biomedical research, health and health disparities. Universities play a key role in recruiting and retaining scholars into these fields.Multiple factors impact success of students from under-represented groups at multiple career levels A multi-faceted strategy is needed to maximize their success.Diversity is a top priority for the Division of Biomedical Research Workforce. Our division has launched a new extramural diversity websiteWe also have an interagency working group on graduate education and interacting with the interagency group re broadening participation in STEM and the biomedical sciences.

NIH Extramural Diversity Websitehttps://extramural-diversity.nih.gov/

New!48Lisa Evans and TAC

Programs to Enhance Diversity By Career Stage:F31, Predoctoral Fellowship to Promote Diversity in Health-Related Research (note due dates April 8th, August 8th, December 8th)T32, Training Program for Institutions That Promote DiversityT34, Maximizing Access to Undergraduate Research Careers (MARC) K01, Mentored Career Award for Faculty at Institutions That Promote Diversity (some IC)G12, Research Centers in Minority Institutions ProgramSC1, Support of Competitive Research (SCORE) Research Advancement AwardR25, Multiple Research Education Programs (Research Initiative for Scientific Enhancement (RISE); Multiple NCI Cancer Education Programs)

Other programsSmall Business Innovation Research & Small Business Technology Transfer (SBIR/STTR) ProgramsNIH Loan Repayment Programs (LRP)49

Research Supplements to Promote Diversity in Health-Related Research

Administrative supplement to an existing research grant, designed to:Support candidates from underrepresented groups who wish to develop research capabilities... participate in career development experiences Support many career stages from undergraduate to facultyDiversify the biomedical research workforce

PA-16-288: https://grants.nih.gov/grants/guide/pa-files/PA-16-288.htmlAdministratively reviewed by the Institute or Center (IC) funding the original grant Note: different ICs have different deadlines and policies

50

50

Diversity supplements at all career stages51

Average Award Size FY2014

Distribution of Awards FY2014

Rigorous evidence on outcomes of diversity strategies will inform policy and programs

Outcomes analyses of Diversity supplementsEvaluation of existing programs: R25 Education programs, F31 Pre-doctoral, F32 Postdoctoral, Career Development (K-Awards)Partnerships between Research Intensive Universities & Institutions serving historically under-represented groupsInstitutional Research and Academic Career Development Awards (positive outcomes: https://www.nigms.nih.gov/News/reports/Documents/IRACDA-outcomes-report.pdf

https://researchtraining.nih.govNIH Research Training WebsiteLaunched in 2015, one stop for funding opportunitiesUseful resource for trainees and early stage facultyModifications and integration with new DBRW website in progress

Separate components separate slide for matrix53

54

For Program LeadersBuilding Participation55

For Trainee ApplicantsCareer Pathways56

Example: NIH R25 Programs at College-Stage

57

Thank YouQuestions/Discussion ?

Please submit questions through the chatbox

Panel Discussion

Contact info:Julia Michaels, Project Managerjmichaels@aplu.org (202) 478-6071http://www.uuhealth.org

Next webinar

Supporting Minority Postdocs Tuesday, April 18, 1:00-2:00 p.m. Eastern Time

Wrap Up

Chart17.678.474.255.28.16.1

Avg Gain+7.7pts+8.5pts+4.2pts+5.2pts+8.1pts+6.1pts

Sheet1Avg GainLearning Com7.67Serv Learn.8.47Study Abr4.25Internship5.2St/Fac Res8.1Capstone6.1To resize chart data range, drag lower right corner of range.

Chart151576471596469745865707643505767

No HIPs1-2 HIPs3-4 HIPs5-6 HIPs

Sheet1No HIPs1-2 HIPs3-4 HIPs5-6 HIPsDeep Learning51576471Gains Gen. Ed.59646974Gains Practical58657076Gains Pers. & Soc.43505767To resize chart data range, drag lower right corner of range.