Annual Report 2016-2017: Key Activities, Accomplishments

and Staff Excellence

The mission of the Office of Assessment, Evaluation, and Research is to: (1) Provide a coordinated approach for data collection and use to foster data-informed decision-making for programs and services within Student Affairs, (2) Serve as the data resource for Student Affairs, (3) Align assessment outcomes with strategic objectives at both the divisional and departmental level, and (4) Serve as a liaison with other institutional assessment and research entities to promote a culture of assessment at the University of Utah.

The values that guide our work are: Integrity, Professionalism, and Collaboration.

Table of Contents Core Objectives ....................................................................................................................................................................... 1

Strategic Planning ......................................................................................................................................................... 2

Assessment and Evaluation .......................................................................................................................................... 4

Research .................................................................................................................................................................... 10

Reporting .................................................................................................................................................................... 10

AER Staff Excellence & Education ............................................................................................................................... 10

Plans for the Future .............................................................................................................................................................. 11

Anticipated Challenges ............................................................................................................................................... 11

Anticipated Opportunities .......................................................................................................................................... 12

Staff Excellence ..................................................................................................................................................................... 13

1

Core Objectives The Office of Student Affairs Assessment, Evaluation and Research (AER) provides coordinated support for these activities within the Division of Student Affairs. During the past year the entire Division of Student Affairs has been engaged in a focused process to make greater connections between programs and services and the assessment of these activities. As part of this larger organizational mapping process, AER has outlined its key activities, goals and outcomes for AER as follows:

Core Objectives Goals Outcomes

Strategic Planning

Facilitate ongoing strategic planning for the Division of Student Affairs to align with institutional strategic planning efforts.

Assist each department within Student Affairs with strategic plan development in a manner that aligns with the Division's strategic plan.

The strategic plan has been drafted and is undergoing review.

Once the Student Affairs strategic plan has been approved, aligned department level planning will begin.

Assessment

Facilitate assessment planning for the Division of Student Affairs and each department within the Division. Increase assessment of processes and services that involve multiple areas.

Implement standardized utilization data collection for the division. Facilitate and support the use of shared assessment resources with other institutional entities.

Integrated utilization data planning within our process and departmental implementation.

Promoted the use of broader outcome measures across departments (e.g. Enrollment Plans Survey) Implemented different tools based on departmental needs that result in uniform utilization data across Student Affairs. Coordinated institutional surveys Developed dashboards for Student Affairs in

2

Collaboration with the BI team

Research

Develop 1-2 research studies annually.

Facilitated data collection across the division and institution. Analyzed data for further research efforts. Learning the capacities of Civitas to develop impact studies for Student Affairs.

Reporting

Increase reporting of data Established assessment newsletter for Student Affairs. Developing initial Tableau dashboards for First Destination and Graduation data.

Staff Excellence and Education

Support professional development for AER staff.

Provide educational opportunities for Student Affairs and other data entities on campus to develop best practices for assessment and share research findings related to students.

Each staff member has a professional development plan with associated costs, goals and outcomes.

Provided training for student affairs staff on relevant AER topics.

A discussion of each core objective, associated goals and outcomes follows.

Strategic Planning Goals: (1) Facilitate ongoing strategic planning for the Division of Student Affairs to align with institutional strategic planning efforts; and (2) Assist each Student Affairs Department with strategic plan development in a manner that aligns with the Division’s Strategic Plan.

3

Over the past year, eight strategic planning committees met, developed strategic objectives, goals and actions utilizing the previous strategic plan framework. Staff from across departments and representing a variety of roles participated. As the Student Affairs Leadership Team reviewed these eight plans, it became clear that modifying the previous framework to discern focal areas from approach (or philosophical) objectives would be useful. It also was important to consider how staff undergird all of our work. The format of the plan has been re-organized, and the proposed strategic objectives, goals and actions are under review. Once this has been finalized, department level alignment implementation will begin.

Student Affairs Strategic Plan: Student Success

4

The following individuals have led development of goals and actions for each of the eight areas:

Strategic Planning Leadership

Student Engagement & Support Tasha Myers

Strategic Enrollment Management Mary Parker

Student Health & Wellness Scott McAward

Diversity & Inclusion Debra Daniels

Partnerships Kari Ellingson

Assessment & Best Practices Stacy Ackerlind

Facilities & Resource Management Jerry Basford

Staff Excellence Perry Hacker

Outcomes: The plan has progressed and is being refined by SALT.

Assessment and Evaluation Goal One: Facilitate assessment planning for the Division of Student Affairs and each department within the Division. With the implementation of Symplicity and tools as part of the “U engage” utilization strategy, a division-level initiative to increase the focus on utilization data was implemented. AER has refined our existing assessment model to focus more resources and effort on utilization data to allow programs, departments and the division as a whole to understand who utilizes our programs and services, the impact of use of these services and outreach to those who are underserved. This data also allows for more complex student life cycle studies to be developed. General outcomes including: (1) sense of belonging, (2) feeling valued, and (3) satisfaction will be built into existing measures to provide a division-level general outcomes strategy.

5

University of Utah Student Affairs Assessment Model

*Ackerlind, 2011

*Ackerlind, Duszak and Masoka, 2016

At Assessment Boot Camp, this model will be reviewed with departments to ask them to align their efforts using the pyramid and as a diagnostic tool to identify where they may be lacking critical information. This assessment will prepare departments for strategic plan alignment. The number of assessment projects has continued to increase. In the previous year, 194 assessment projects were requested whereas this year, 246 were requested. Assessment projects may consist of one element such as a survey or may include multiple elements such as a utilization kiosk and survey, or a focus group and survey or datamining and exploration of linked data sets.

6

Surveys continue to dominate the types of assessment projects requested through AER. With new tools available, utilization data requests have increased as have reports and analysis. With the advent of the PPM system, AER has largely moved away from personnel evaluation support but do continue to provide these on an ad hoc basis, particularly for 360 reviews.

8090

107

141

201186

196211

194

246

0

50

100

150

200

250

AER Assessment Project Trend Data

7

Following is a breakdown of requests by reporting line. A greater number of projects continue to be requested from Enrollment Management. With a greater number of areas reporting to the VPSA, a greater number are coming from her direct reports than have in the past, which has impacted the overall distribution of projects across Student Affairs.

68.3%

10.6%

3.7% 4.5%8.1%

0.8% 2.4% 1.6%

0%

10%

20%

30%

40%

50%

60%

70%

80%

Survey UtilizationKiosk

UtilizationReport

Analysis Report Dashboards Personnelevaluation

Other

Project Types for 2016-2017

4.5%

15.0%

24.4% 24.8%

17.5%

5.3%

2.0%

6.5%

0%

5%

10%

15%

20%

25%

30%

2016-2017 AER Assessment Projects by Reporting Line

8

Outcomes: AER has refined the Student Affairs Assessment Model. AER will implement the new assessment model as a diagnostic tool for departments to prepare for strategic plan alignment with assessment efforts. The increase in requests for analysis, reporting and utilization data suggests that AER time is being used to address other needs within the Division and Institution beyond surveys, which suggests progress in meeting our goal.

Goal Two: Increase assessment of processes and services that involve multiple areas. Over the past year through AER’s provided greater support to Enrollment Management to increase student life cycle data elements. Due to the complexity and demands of those offices as well as personnel changes within AER, this work is ongoing and is the highest priority. Because of the need to focus on Enrollment Management and limited staff, greater collaborative efforts were not initiated for projects beyond key offices for the Enrollment Plans Survey, across the division. Outcomes: Reports established on Student Life Cycle Data. Planning is occurring to continue to deepen this effort in 2016-2017 in collaboration with the BI Team.

Goal Three: Implement standardized utilization data collection for the Division of Student Affairs

As AER has reviewed and mapped office processes, our team has developed a flexible strategy to meet the needs of the departments to collect utilization data. Because of the variability of needs, there is not a “one size fits all” tool. However, with access to the Student Data Warehouse, Symplicity, Campus Labs and department specific tracking strategies (e.g. Excel spreadsheets, Titanium, Medicat), the requirement for what is collected has been standardized. A standardized report format has been developed that departments will receive at the end of each semester or event. Our team is continuing to meet with each department and utilization data collection will be the focus of Assessment and Budget Boot Camp. Outcome: Standardized utilization reporting established and increasing. Specific training will be offered to further develop this data element across Student Affairs

Goal Four: Facilitate and support the use of shared assessment resources with other institutional entities. AER has continued to manage the Campus Labs Contract for Student Affairs and Academic Affairs. Because of the scope of Student Affairs, our Division is the largest user. AER continues to collaborate with OBIA, UGS and the Graduate School on the Graduating Student Survey. We have also developed a process for the First Destination Form that involves Career Services, Registrar’s Office and coordination with colleges. Our team continues to manage the institutional survey distribution and sampling as a coordinated effort to boost response rates and reduce survey fatigue.

9

In order to maximize our resources for analysis, AER has begun to collaborate with the BI Team to reduce the number of manual processes of joining data for assessment projects, most notably the First Destination Form and specialized projects for Enrollment Management.

250

41

34

20

9

0

0 50 100 150 200 250 300

Student Affairs

Undergraduate Studies

Joint Report Projects (managed by SA)

Institutional Surveys (managed by SA)

Other surveys for UU managed by SA

Institutional Surveys (joint management)

Campus Labs Usage 2016-2017

0%

10%

20%

30%

40%

50%

60%

70%

80%

Campus Labs Usage by Area

2011-2012 2012-2013 2013-2014 2014-2015 2015-2016 2016-2017

Note that Joint projects were not tracked based on managing office prior to FY13-14.

10

Outcomes: AER continues to coordinate institutional surveys. Dashboards to share information more efficiently are being developed in collaboration with the BI Team.

Research Goal: Develop 1-2 research studies annually. Our research efforts have continued to focus on Campus Climate (Sexual Assault and Misconduct), collaborating with OBIA on the use of the National Survey of Student Engagement (NSSE) and the National College Health Assessment. With the implementation of Civitas, AER is working collaboratively with campus partners to understand how this tool can be used for research and by whom. Outcomes: (1) Facilitated data collection across the division and institution. (2) Analyzed data for further research efforts. (3) Learning the capacities of Civitas to develop impact studies for Student Affairs.

Reporting Goal: Increase reporting of data. AER has continued to develop reports particularly for Enrollment Management. A comprehensive report that was developed on behalf of Mary Parker for the institution and for colleges, showing demographic characteristics of students who do not attend the University of Utah, after admission and where they attended college. This report was presented and distributed by Mary Parker at the Council of Academic Deans. AER is also developing our data visualization capacities to allow more routine data, particularly utilization data to be available for data consumers. Our initial efforts have focused on visualizations for the First Destination Data and Graduation data in addition to the Dean’s reports discussed above. AER has continued our newsletter to Student Affairs to share assessment projects and highlight the work of our colleagues across the Division. Outcomes: AER has increased the number of formal reports produced, data visualization capacity and informal sharing of information through the Assessment Newsletter.

Staff Excellence and Education

Goal One: Support professional development for AER staff In order to utilize best practices to support the Division in creating a culture of evidence, providing opportunities for learning and continued skill development are critical for the AER team. Each staff member completed a professional development plan that was used as the basis for evaluation. Outcome: Each staff member has a professional development plan with associated costs, goals and outcomes for 2016-207. Goal Two: Provide educational opportunities for Student Affairs and other data entities on campus to develop best practices for assessment and share research findings related to students. Assessment Coffee Hour Seminar Series

11

In 2016-2017, the Assessment Coffee Hour seminar series was continued for Student Affairs Staff. The purpose of the Assessment Coffee Hour is to share data and data related topics in an informal seminar format to engage participants in assessment conversations around selected topics. The attendance at the events varied greatly as detailed below.

Topic Number of Attendees

Where do our graduates go? (Stan Inman, Kelly Dries & Eric Bloomquist, Career Services)

19

Enrollment Plans: Turning Data into Action (Tim Ebner, University of Utah Registrar)

21

National Survey of Student Engagement (Stacy Ackerlind & Elizabeth Duszak, AER)

20

A Closer Look at Counseling Center Data (Lois Huebner & Jake Van Epps) 16

Evaluation data was collected for each event, compiled, reviewed, shared with presenters and used to plan subsequent events. Assessment Boot Camp will be offered in July 2017 to focus on the updated Student Affair Assessment model, annual and budget reports.

Plans for the future The role of Assessment, Evaluation and Research has evolved over time to meet the needs of the Division as well as address gaps at the institution. With the charge to provide Enrollment Management analysis and reports as well as implement a system for utilization data, AER has prioritized these efforts. AER has continued to address institutional survey needs and increased capacity to provide meaningful data visualizations. AER has also played a critical role in the Division’s strategic planning process to support the Student Affairs Leadership Team.

A. Anticipated challenges

1. Prioritization: Our overall number of projects increased dramatically this year, in part due to the focus on utilization data collection and increased number of Enrollment Management data requests and analytics development across the Division. The complexity of analytics that incorporated multiple data sources, will continue to increase, with the demand for actionable data. With institutional demands of AER for initiatives such as Civitas, it will continue to be a balancing act and require us to continue to ensure that what is being requested of us, truly meets a priority. Our concern has been that if we do not provide the data on demand, that departments will develop “work-arounds” that are problematic in terms of accuracy, integrity and/or adherence to policy. A major goal for us is to create a detailed data map of the Division to ensure the best and most accurate data are being used to help retire some of these inefficient and ineffective strategies that are currently in place. While providing excellent customer service to our colleagues is important, we have also realized that we need to offer increased training to staff from other departments, so that they can provide use with more uniform data to maximize our time for analysis and research design.

2. Data consistency and integrity: With the mapping process of student life cycle data for Enrollment Management, AER has gained a great deal of insight as to how data is pulled from PeopleSoft and reported by different departments across the division. There are many “functional silos” with data that have

12

developed to meet operational needs yet, present some challenges for connecting data across the life cycle. With gaining greater insight into how data is currently used and the development of meaningful analytics and routine reports for Enrollment Management leadership, AER has is working closely with the Business Intelligence team to ensure the data we use for analytics is the most accurate data. AER has worked closely with Career Services to establish a uniform process for First Destination data. This process includes multiple collection intervals and periodic data cleaning. Our challenge remains with the colleges who often times want to handle their own first destination data collection, which is understandable with accreditation processes. However, when this process is siloed and different elements are collected, an institution view is lost. AER needs to continue to build our partnerships with colleges and ensure we share data back with them so that they will participate in and see the value of the institutional process.

3. Utilization data process: As we have scaled up our utilization data collection efforts, we have learned that a

uniform collection strategy is not feasible but that a uniform analysis process is. With inconsistent Wi-Fi in many locations on campus, AER has relied on Campus Labs to collect utilization data as Symplicity MSE requires a consistent internet connection. With access to the Student Data Warehouse and the increasing capabilities of data-warehousing, we have expanded our capacity to analyze data collected previously using snapshot data versus the live feed that Symplicity MSE uses. Because our use of MSE is dependent on Career Services’ continued use of the CSM portion, we have started to scope other options as Career Services is exploring other survey options. As a team, we have realized that we can better serve the full division of Student Affairs if we have flexibility with utilization data and also are not tied to the decision-making of another department.

4. Survey Management Software: With more emphasis on data that can be used effectively by departments,

the division and the institution as a whole, AER staff time needs to be used differently and focused more on analysis and thorough conceptualization of complex projects in addition to the mapping described above. We are exploring different survey management tools that will allow more flexibility and seamless integration of data into dashboards. With the number of individual Qualtrics licenses on our campus, the Office of Software Licensing is proposing an enterprise license solution. AER has joined this process and is facilitating conversations as with our role, it could be beneficial to have one license to limit the number of surveys to students but it also adds complexity. This scoping effort is in process and will impact AER’s survey tool plan moving forward.

5. Institutional surveys and lack of formal policy: AER has continued to meet the institutional need for

coordination and administration of surveys. While this has likely reduced some oversampling of students, it does not address the number of surveys that are sent out through the colleges, departments or even by students to students. AER will continue to explore how we might serve the institution through leadership to develop a strategy that will hopefully lead to meaningful policy development and clarity of our role. With the new technology being explored, there may be some ways to merge these two needs. Additionally conversations have begun to clarify the policy for recipients of financial aid with incentives.

B. Anticipated opportunities: 1. New Student Affairs Strategic Plan: With a greater focus on outcomes within the strategic plan, this will

provide AER a more well-defined focus for assessment and prioritization. It will also sharpen our focus for reports to address accomplishment of specific goals.

2. Implementation of new Student Affairs Assessment Model: With expanded capacity to collect utilization data and additional AER staff, our focus will be to implement the model across the division more fully. This will provide a consistent assessment approach at the program, department and divisional level.

13

3. Improved data usability and access: With access to the Student Data Warehouse, PeopleSoft and our

collaboration with the Business Intelligence group, AER is able to provide analytics on existing data which supports our assessment model. This greatly supports our ability to expand the reporting for Enrollment Management and for other areas who have needed more integration of utilization and student records data to inform their work.

4. Further strengthen Student Affairs reporting and budget process: With the establishment of the Four Big

Goals, the new institutional budget process and the new reporting and budget process within Student Affairs, AER has the opportunity to support our leadership through refining data elements and showing the impact of Student Affairs programs and services. With the partnership with the Associate Vice President for Business and Auxiliary Services, Jerry Basford, AER can take an active and supportive role in refining this process for our division.

5. AER leadership with surveys and survey policy: With our leadership around survey technology and management of institution student surveys, we have an opportunity to work closely with our executive leaders to determine a vision for student surveys that will provide necessary data and protect students from excessive requests for information. As we continue to explore the new tools, this is providing structure to support conversations as well as consider integration with other institutional tools such as Tableau and Civitas.

C. Grants and contracts: None received in 206-2017.

Staff Excellence

A. Professionalism

Each AER staff member collaboratively develops a professional development plan that is used to evaluate

performance across the year for both departmental and personal goals. Staff members are encouraged to

enhance core job related skills as well as broader skills within student affairs. Dr. Ackerlind is currently the

National NASPA Assessment, Evaluation and Research Knowledge Community Co-Chair. This position entails

responsibilities at both the national NASPA conference and the Assessment and Persistence conference.

Following are the professional development conferences attended by staff members:

Staff Member Conferences Attended

Stacy Ackerlind, Director NASPA National Conference, 2017

Qualtrics Summit, 2017

NASPA Assessment and Persistence

Conference, 2017

Elizabeth Duszak, Assistant Director Utah NASPA Conference, 2017

Qualtrics Summit, 2017

NASPA Assessment and Persistence

Conference, 2017

Chuck Masoka, Information Coordinator Qualtrics Summit, 2017

14

NASPA Assessment and Persistence

Conference, 2017

Vanessa Johnson, Assessment Analyst Qualtrics Summit, 2017

B. Recruitment and retention of highly qualified staff C. Retirements, new hires, accomplishments

Dr. Stacy Ackerlind has served as the Director for AER since 2006 and as the Special Assistant to the Vice President for Student Affairs since 2012. Dr. Duszak has served as the Assistant Director for AER since 2015. Vanessa Johnson was hired as an Assessment Analyst in December and will leave the position to continue her education. A new search will begin in September. Chuck Masoka’s position was changed from a joint report between AER and the Career and Professional Development Center and is now a joint report between AER and Scholarships and Financial Aid. 1. Awards: None

2. Committees

Ackerlind, Stacy Student Affairs Leadership Team

NWCCU Accreditation Core Team

University of Utah Retention and Completion Tasks Force

Graduating Student Survey Assessment Team (Chair)

Civitas Work Illume Work Group

Advanced Registration Committee

NASPA Assessment, Evaluation and Research Knowledge Community (National), Co-Chair

Duszak, Elizabeth Student Affairs Assessment Cooperative (Chair)

Graduating Student Survey Assessment Team

Student Affairs Professional Development Committee

Civitas Illume Technical Work Group

Masoka, Chuck Student Affairs Diversity Council

3. Presentations and publications

Ackerlind, S. J., Duszak, E., & Masoka, C. (2017). Driven by data or becoming data informed: A case

for organizational change. Presentation at the NASPA Assessment and Persistence Conference, Orlando, FL.

Ackerlind, S. J., & Roberts, D. (2017). Calling all directors of division assessment. Presentation at the NASPA Assessment and Persistence Conference, Orlando, FL.

Ackerlind, S. J., & Roberts, D. (2017). Engage with the Assessment, Evaluation, and Research Knowledge Community. Presentation at the NASPA Assessment and Persistence Conference, Orlando, FL.

15

Duszak, E., & Ackerlind, S. J., (2017). Applying assessment expertise to enhance recruitment and retention efforts: A collaboration between AER and enrollment management. Presentation at the NASPA Assessment and Persistence Conference, Orlando, FL.

4. Faculty appointments Ackerlind, Stacy Adjunct Assist Professor Educational Leadership and Policy