Slide 1

Assessment InstituteOctober 2013

Amber D. Lambert, Ph.D.Angie L. Miller, Ph.D.Center for Postsecondary Research, Indiana UniversityAssessment with Alumni Surveys: Administration Tips and Data Sharing Suggestions

1AbstractAlumni surveys can be important sources of information for institutions, yet many obstacles are involved in their execution. The Strategic National Arts Alumni Project (SNAAP) is a multi-institution online survey of arts graduates from secondary and postsecondary institutions. This presentation shares lessons learned during the first five years of SNAAP, including issues related to accurate alumni contact information, response rates, design factors, and implementation of results. Actual examples from previous SNAAP-participating institutions are shared to promote idea generation among participants for the use of alumni surveys at their own institutions.

Literature Review and Introduction to SNAAP

3Literature ReviewAs funding to higher education institutions continues to be cut, colleges and universities are often required to show measures of their effectiveness (Kuh & Ewell, 2010)

Surveys are used in many areas of higher education (Kuh & Ikenberry, 2009; Porter, 2004)Alumni surveys can provide valuable information on student satisfaction, acquired skills, strengths and weaknesses of the institution, and current career attainment

Literature ReviewA major concern with all surveys, and alumni surveys in particular, is low response rates

Over the last decade survey response rates have been falling (Atrostic, Bates, Burt, & Silberstein, 2001; Porter, 2004)

Alumni surveys often have lower response rates than other types of surveys (Smith & Bers, 1987) due to:Bad contact information Suspicion of money solicitationDecreased loyalty after graduation

SNAAPAs an example, we present some best practices for survey administration and share results from the Strategic National Arts Alumni Project (SNAAP)

What is SNAAP?Online annual survey designed to assess and improve various aspects of arts-school educationInvestigates the educational experiences and career paths of arts graduates nationallyFindings are provided to educators, policymakers, and philanthropic organizations to improve arts training, inform cultural policy, and support artists

SNAAP Basics: Who is surveyed?Participants drawn from..Arts high schoolsIndependent arts collegesArts schools or departments in comprehensive colleges & universities

Broad definition of arts

All arts alumni, all years (since 2011)2008-2010: surveyed selected cohorts

SNAAP began in 2008 and tested the questionnaire and survey operations with three annual field tests that were free to participating institutions. In 2011, we fielded the first national administration, surveying all alumni from each participating school. 7

Increasing Numbers2010 Field TestOver 13,000 respondents154 Institutions

2011 AdministrationMore than 36,000 respondents66 institutions

2012 AdministrationMore than 33,000 respondents70 institutions

2013 AdministrationCurrently underway

Combined 2011 and 2012 respondents to create a SNAAP Database with over 68,000 respondents plan to add 2013 data after this year!

Questionnaire TopicsFormal education and degreesInstitutional experience and satisfactionPostgraduate resources for artistsCareerArts engagementIncome and debtDemographics

The questionnaire is divided into 7 sections.9Participating institutions receive...Institutional Reports: Customized, ConfidentialSeparate reports for undergraduate and graduate alumniBoth quantitative and qualitative dataSpecial report on Recent GradsComparative data with other schools

Complete data file of responses

Workshops/webinars on how to use data

PASS AROUND COPIES OF SAMPLE IRs10Survey Administration Challenges

11Survey Administration Challenges: Locating the LostImportant that contact information is accurate and up-to-date

Encourage proactive effortsNewslettersWebsites, social networking

Alumni trackingContracted with Harris Connect, a direct marketing firm

12Survey Administration Challenges: Response RatesResponse rates are directly related to the accuracy of contact information

Incentives: only minimally effective

Open enrollment features can increase number of responsesSocial networking sitesNeed to verify respondents

13Survey Administration Challenges: Response Rates

Email invitations to participate in the surveyIs it better to have HTML or plain text?

For the 2011 administration, we created visually appealing email invitations in HTML format

We wondered if HTML was causing problems with email platforms. Either getting flagged as spam or requiring too much effort from respondents to enable the html content (an extra click)14Survey Administration Challenges: Response Rates

For the 2012 administration, we systematically compared the effectiveness of HTML invites to plain text invites across the 5 email contacts sent to participants

Results of this experiment suggested that a combination of message types gets the highest response ratesPlain text was more effective for the initial contactHTML was more effective for follow-up contactsPotential reasons: plain text may reach larger numbers, but HTML may give the project legitimacy The results of the first round of messages showed a statistically significant and rather large difference in response rates between the two groups. Those who were sent plain-text message were much more likely to click on the link for the survey and respond than those who were sent the HTML message. In the second round of invitations, those who received the HTML invitation were much more likely to respond than the plain-text counterparts. For alumni at schools of art and design, for whom the visual appearance of a message might be even more influential, the difference was more dramatic. Also, the group that first received a plain-text email was the group with the highest response rate. These results suggest that using both plain-text and HTML messages when contacting your target population may be the best way to increase response rates and that starting with a plain-text invitation might be the best plan. The plain-text messages perhaps reach larger numbers, such as those for whom the HTML messages would go their spam folders or not load properly. In contrast, the HTML messages might give the project legitimacy to survey respondents. Thus, to reach the largest number of your target population, both HTML and plain-text message should be used and the order of these formats might make a difference as well. In addition, for some types of institutions the use of HTML messages might be even more critical in the effort to increase response rates.

15Examples: Sharing on CampusMiami University of Ohio: Assessment Brief published by Institutional Research office after receiving their 2012 SNAAP Institutional Report

Purdue University: Office of Institutional Research published a 4-page report summarizing data from 2011 Institutional Report, integrating both quantitative and qualitative data

Sharing your own results on campus with campus leadership as well as faculty and staff16

One of the first institutions to share their data on-campus was Miami University of Ohio. This Assessment Brief was published by Miamis institutional research office after it received its SNAAP report of 2010 data. 17

Purdue University published a four-page report of its 2011 SNAAP data. 18

Purdue 2 of 419

Purdue 3 of 420Examples: Alumni and Donor OutreachThe University of Texas at Austin: College of Fine Arts promoted SNAAP results in 2011 with the Deans letter in its quarterly print publication, thanking alumni for participating and sharing selected findings

In 2012, UT Austin also used their SNAAP data, integrated with other information sources such as IPEDS, to develop a web page that illustrates some of its findings with info-graphics

21

The College of Fine Arts at the University of Texas at Austin began sharing its SNAAP results in 2011 with the Deans letter in its quarterly print publication. In this example, the dean thanks his alumni for participating and shares selected findings: http://www.utexas.edu/finearts/alumni/alumni-snaap-thank-you-letter22

In 2012, UT Austin developed a web page that illustrates some of its findings http://www.utexas.edu/finearts/about/mission-vision/alumni-snapshot23

UT Austin shared portions of its Institutional Report (being careful to not identify any respondents). http://www.utexas.edu/finearts/sites/default/files/attach_download/utaustinfineartssnaap2011report.pdf24Examples: Alumni and Donor OutreachHerron School of Art + Design (IUPUI): created a website to share selected findings, and integrated SNAAP data with alumni profiles on the website Virginia Commonwealth University: mentions SNAAP participation in their VCU Alumni publication, and incorporated an option for alumni to update their contact info for future surveys

25

The Herron School of Art and Design at IUPUI in Indianapolis created a web site to share selected findings. 26

Herron 2 of 227

The Herron School tells the story of one alumna in a post on its web site28

VCU Alumni publication- online and in print. Also incorporates option for alumni to update their contact info for future surveys. 29Examples: RecruitmentKent State University: created flier for those considering majoring in visual arts using aggregate SNAAP findings

Herron School of Art + Design (IUPUI): created a recruitment brochure based on its alumni achievements, potential careers, and comments from the SNAAP survey

30

This web page from Kent State shares aggregate SNAAP findings to make the case for a visual arts education. 31

The Herron School of Art and Design created a recruitment brochure based on its alumni achievements. This page includes comments from the SNAAP survey. 32

Herron (Creativity) 2 of 233Examples: Program & Curricular ChangeVirginia Commonwealth University: found discrepancies when comparing the business skills alumni needed for their work to the business skills learned at VCUIntroduced a new Creative Entrepreneurship minor to address this

University of Utah: found that many alumni were dissatisfied with career advising their school offeredCollege of Fine Arts developed an Emerging Leaders Program that offers high-stakes internships, mini-grants, and peer-mentoring opportunities designed to prepare students for their transition into the world of work

34

VCU Arts example. Found discrepancies in business skills used vs. skills learned and introduced a new minor to address this.35

New VCU Course Catalog Listings

VCU Arts example continued- Specific courses developed36

University of Utah example: Found that more than half (51%) of undergraduate arts alumni are dissatisfied with the career advising their school offered, as are 43% of graduate arts alumni. Listening to these concerns, the College of Fine Arts developed an Emerging Leaders Program. This program offers high-stakes internships, mini-grants, and peer-mentoring opportunities designed to prepare students for their transition into the world of work.

37ReferencesAtrostic, B. K., Bates, N., Burt, G., & Silberstein, A. (2001). Nonresponse in U.S. government household surveys: Consistent measure, recent trends, and new insights. Journal of Official Statistics, 17(2), 209-226.

Kuh, G. D. & Ewell, P. T. (2010). The state of learning outcomes assessment in the United States. Higher Education Management and Policy, 22(1), 1-20.

Kuh, G. D. & Ikenberry, S. O. (2009). More than you think, less than we need: Learning outcomes assessment in American higher education, Urbana, IL: University of Illinois and Indiana University, National Institute of Learning Outcomes Assessment.

Porter, S.R. (2004). Raising response rates: What works? New Directions for Institutional Research, 121, 5-21.

Smith, K., & Bers, T. (1987). Improving alumni survey response rates: An experiment and cost-benefit analysis. Research in Higher Education, 27(3), 218-225.

*Special thanks to Miami, Purdue, IUPUI, UT Austin, VCU, Kent State, and U of Utah for sharing their examples with SNAAP!

Assessment Brief #62October 12, 2011

Using Feedback from Miami Alumni to Improve Educational Effectiveness

Surveying AlumniMiami students are frequently surveyed throughout their college experiences. However, assessing the long-term impact of students Miami education can also require reaching out to students after they graduate. Feedback from alumni, who are now using the skills they developed at Miami, can greatly improve educational effectiveness. The Strategic National Arts Alumni Project (SNAAP) survey gathers information about fine arts alumni to better understand the relationship between arts education and arts-related occupations. The SNAAP participants from Miami University consisted of 220 undergraduate fine arts alumni who graduated in the following years: 1990, 1995, 2000, and 2005-2009.The survey included questions about institutional experiences and career choices. To capture institutional experiences, the survey prompted alumni to report their overall satisfaction with their education as well as their satisfaction with specific areas (e.g., academic advising, freedom to take risks). In the career section, alumni reported their current and previous occupations, their satisfaction with these jobs, and their current level of fine arts engagement.

To explore the intersection between institutional experience and careers, the survey asked alumni about the skills and competencies they developed at Miami University as well as which skills were most important in their current job.

By reviewing these results, faculty and staff can better understand how students experiences at Miami prepare them for their career.Key Findings Fine arts alumni were satisfied with their experiences at Miami University; 94% of undergraduate arts alumni rated their overall experience as good or excellent.

Arts alumni were especially satisfied with their sense of belonging at Miami and with their instructors.

Respondents were least satisfied with opportunities to network with alumni and others, advice about further education, career advising, and work experience. The vast majority of respondents reported developing critical and creative thinking skills while at Miami and found these skills important in their future careers. Fine arts alumni were less likely to report that Miami helped them to develop business and technological skills related to their field.

Student Satisfaction

RecommendationsThe SNAAP survey highlights the importance of gathering alumni feedback. Such feedback is a valuable resource for assessing educational impact and improving educational effectiveness across the university. The SNAAP survey helps faculty and staff in fine arts by identifying the following: Common occupations and post-secondary degrees among graduates

Skills and competencies that students will frequently use in their careers

Levels of student satisfaction with various aspects of their Miami experience These results can help the division improve retention and graduation rates and better prepare students for their future careers.If you have comments or questions, please contact the Center for the Enhancement of Learning, Teaching and University Assessment at celt@muohio.edu or 513-529-9266. Previous Briefs are available online at: http://www.units.muohio.edu/celt/assessment/briefs/. Miami University

Assessment