from micro to macro: correlating instruction with student learning & success through local and...

From Micro to Macro:

Correlating Instruction withStudent Learning & Success through Local and National

Initiatives

Zoltán Szentkirályi, Director of Assessment

Hollie Gardner, User Experience Librarian

Central University Libraries

Southern Methodist University

North Texas Library

Assessment Group

Spring Meeting 2015

March 20, 2015

GWLA Learning OutcomesConference & Workshop

November 13-15, 2013 at UNLV, Las Vegas

33 GWLA member universities, each encouraged to send a team of three people to discuss SLO’s.

SMU Team:

“Manager in Charge of Library Instruction Program” Elizabeth Killingsworth

“Assessment Representative” Zoltán Szentkirályi

“Instruction Librarian” Hollie Gardner

Student Outcomes Research

GWLAGreater Western Library Alliance

Arizona State UniversityBaylor UniversityBrigham Young UniversityColorado State UniversityIowa State University LibraryKansas State UniversityOklahoma State UniversityOregon State UniversityRice UniversitySouthern Illinois UniversitySouthern Methodist UniversityTexas A&M UniversityTexas Tech UniversityUniversity of ArizonaUniversity of ArkansasUniversity of Colorado at BoulderUniversity of Hawai'i at Mānoa


University of HoustonUniversity of Illinois at ChicagoUniversity of KansasUniversity of MissouriUniversity of Nevada, Las VegasUniversity of New MexicoUniversity of OklahomaUniversity of OregonUniversity of Southern CaliforniaThe University of Texas at AustinUniversity of UtahUniversity of WashingtonUniversity of WyomingUtah State UniversityWashington State UniversityWashington University in St. Louis

The Greater Western Library Alliance (GWLA) is a consortium of33 research libraries located in the central and western United States.

GWLA Student Learning Task Force

Initial Charge Established Spring 2011

Goals for the GWLA Student Learning Task Force:

To investigate and sharelocal evaluation/assessment practicesand evidence obtained, across GWLA institutions

To create a research question and identify waysto assess students, using all GWLA universitiesas a possible sample population



First Phase of the Charge


2011 – 2014

Report Compiled and Submitted



Second Phase of the Charge


To create a research question and identify waysto assess students, using all GWLA universitiesas a possible sample population

Spring 2014 - Call For Participants for Research Project


GWLA Libraries Impact on Student Learning Task Force

Research Project started in 2014

Melissa Bowles-TerryHead of Educational Initiatives, UNLV LibrariesChair of Student Learning Outcomes Task Force

Patty Iannuzzi, Dean of UNLV Libraries, Dean LiaisonJoni Blake, Executive Director GWLA, GWLA Liaison

32 Task Force Members from 20 Participating Institutions


GWLA Student Learning Research Project

Roster of Participating Institutions & Representatives


GWLA Student Learning Research Project & Task ForceInstitution Name PositionArizona State University Lisa Kammerlocher Social Sciences LibrarianBaylor University John Bales Liaison LibrarianBrigham Young University Suzanne Julian Library Instruction Coordinator Brigham Young University Holt Zaugg Assessment LibrarianGWLA Anne McKee Program Offi cer for Resource SharingGWLA Joni Blake Executive DirectorKansas State University Sara Kearns Head of Undergraduate and Community ServicesKansas State University Melia Fritch Instruction Team LeadOregon State University Cheryl Middleton Associate University Librarian for Learning & EngagementOregon State University Laurie Bridges Instruction and Emerging Technologies LibrarianSouthern Methodist University Zoltan Szentkiralyi Director of AssessmentTexas A&M University Stephanie Graves Coordinator of Learning & OutreachTexas Tech University Laura Heinz Head of Research, Instruction & Outreach University of Arkansas Norma Johnson Reference LibrarianUniversity of Hawaii Vicky Lebbin Social Sciences LibrarianUniversity of Houston Christina Gola Head of Liaison Services for Instruction & OutreachUniversity of Illinois-Chicago Paula Dempsey Research Services & Resources Librarian University of Illinois-Chicago Annie Armstrong Coordinator of Bibilographic InstructionUniversity of Missouri Goodie Bhullar Head of Instructional Services University of Missouri Navadeep Khanal E-Learning LibrarianUniversity of Nevada, Las Vegas Melissa Bowles-Terry Head of Educational InitiativesUniversity of Nevada, Las Vegas Patty Iannuzzi DeanUniversity of Nevada, Las Vegas Greg Voelker Resources Workflow ManagerUniversity of New Mexico Mark Emmons Director of Instruction & AssessmentUniversity of Southern California Jade Winn Assistant Dean for Instruction, Assessment and EngagementUniversity of Southern California Carolyn Caffrey Gardner Info Literacy & Educational Technology LibrarianUniversity of Utah Lorelei Rutledge Research and Information Services LibrarianUniversity of Utah Alfred Mowdood Head of Research and Information ServicesUniversity of Washington Jackie Belanger Assessment CoordinatorUniversity of Washington Amanda Hornby Teaching and Learning Program LibrarianUtah State University Britt Fagerheim Head of Reference ServicesWashington State University Steve Borrelli Strategic Assessment Librarian

GWLA Student Learning Taskforce


Timeline for Research Project


May 2014 - Task Force starts meeting every 2 weeksResearch Design subgroup – prepare research designGrants subgroup – seek & prepare grant applications

Fall 2014 – Institutions begin tracking Library Instruction activity, per the specifications of the Research Design

Spring 2015 – each participating institution work with their Institutional Research office to make sure they can get student data, and submit IRB application for the study


Timeline for Research Project


Fall 2015 – Spring 2016Merge Library Instruction Data w/ Institutional Data for all 20 participating institutions, data cleaning, analysis and report of first year results

Leverage first year results to secure grant and continue research project as longitudinal study over the next 4-6 years

Fall 2015 – Request Institutional Data, e.g. student grades & retention, per the specifications of the Research Design

Research DesignTwo-Tiered Approach

Library Instruction Variables - (teaching methods)

Logs of instruction sessions Librarian contribution to

design Methods and characteristics

Institutional Variables - (student success) Student Demographics Courses, Credit Hours, Grades GPA and Retention



Library Instruction Variables



INDIVIDUAL SESSIONS (data collection tool): Session by session level data collection for library instruction activity, entered by librarian/library-instructor after a session.

● Course number (#) associated with this library instruction session ● Course section (#) associated with this library instruction session ● Instructor (last, first) – In most cases, a non-librarian. ● Session Librarian/Library-Instructor (last, first) – Name of person providing the library

instruction for this session. ● The person listed as Session Librarian/Library-Instructor is:

○ A faculty or staff librarian with a library degree ○ A faculty or staff without a library degree ○ A student or graduate assistant who has not yet completed a library degree

● Date of this library instruction session ● Duration of this library instruction session in minutes ● Number of students present during this library instruction session ● The class meeting format for this library instruction session was:

○ Online only ○ In-person only

● Teaching Method for this library instruction session: ○ Lecture only – a presentation and/or demonstration, with or without the help of

projection of the active website, powerpoint slides, handouts, etc., with students listening/watching but not actively practicing the methods.

○ Flipped – Students were assigned material to complete in advance (modules, videos, tutorials, assignments, etc.), then followed by library instruction which covers the material in greater depth or covers other additional material

○ Directed practice – Students followed along on their computers and performed tasks, e.g. using certain search terms and strategies suggested by the librarian.

○ Active Learning/Assignments – Students worked in groups or individually to complete in-depth activities and tasks assigned by librarian instructions or to capture learning.

○ Other_________________________ ○ None – there was no instruction for this session, only orientation or tour.

● One-shot instruction session (yes/no) – Was this session a one-time library instruction for this class, with no other meeting planned?

● Course-integrated (yes/no) – Was the library instruction for this session integrated with, or directly related to, the course content?

● Session Librarian helped design content or assignment (yes/no) – Session Librarian collaborated with instructor to develop the content or assignment related to this instruction session.

Library Instruction Variables 2



SYSTEM-WIDE CHARACTERISTICS: Library instruction program-level data that may apply broadly across courses or sections at the institution, which can be added to the data after-the-fact by someone knowledgeable at each institution, e.g. instruction coordinator and/or GWLA task force member librarian, prior to submitting. Confirm or fill in Course/Section level data collection (compiled and cross-checked by library representative):

● Campus or university ○ Sub level Campus (for institutions that wish to report on more than one campus)

● Course number (#) ● Course section (#) ● Instructor (last, first) – In most cases, a non-librarian. ● Librarian (last, first) – If a single librarian is associated with this course/section. If the

librarian is the only instructor of record, put librarian’s name as both instructor and librarian.

● Librarian helped design course (yes/no) – A librarian helped develop the overall course assignments and/or syllabus through collaboration with the instructor of record for the course.

● Librarian helped design course/section only (yes/no) – A Librarian collaborated with the instructor of record for this section to develop the assignments for the course/section.

● Librarian helped design assignment for course/section (yes/no) – Librarian collaborated with instructor to develop at least one major credit-bearing assignment for the course/section.

● Online Library Tutorial or Learning Objects (yes/no) – Did this course/section have an online library tutorial or learning objects that were integrated as assignment(s) or assessment(s)?

● Course/section is co-taught by librarian/instructor team (yes/no) – A librarian and at least one non-librarian are both instructors of record for the course/section.

● Credit-bearing library class, taught by librarian(s) (yes/no) – Librarian(s) is/are the only instructor(s) of record.

○ Number of units or credit hours (#) ○ Full-credit course or partial-credit course ○ Quarter, semester, or other

Library Instruction Variables 2



SYSTEM-WIDE CHARACTERISTICS: Library instruction program-level data that may apply broadly across courses or sections at the institution, which can be added to the data after-the-fact by someone knowledgeable at each institution, e.g. instruction coordinator and/or GWLA task force member librarian, prior to submitting. Confirm or fill in Course/Section level data collection (compiled and cross-checked by library representative):

● Campus or university ○ Sub level Campus (for institutions that wish to report on more than one campus)

● Course number (#) ● Course section (#) ● Instructor (last, first) – In most cases, a non-librarian. ● Librarian (last, first) – If a single librarian is associated with this course/section. If the

librarian is the only instructor of record, put librarian’s name as both instructor and librarian.

● Librarian helped design course (yes/no) – A librarian helped develop the overall course assignments and/or syllabus through collaboration with the instructor of record for the course.

● Librarian helped design course/section only (yes/no) – A Librarian collaborated with the instructor of record for this section to develop the assignments for the course/section.

● Librarian helped design assignment for course/section (yes/no) – Librarian collaborated with instructor to develop at least one major credit-bearing assignment for the course/section.

● Online Library Tutorial or Learning Objects (yes/no) – Did this course/section have an online library tutorial or learning objects that were integrated as assignment(s) or assessment(s)?

● Course/section is co-taught by librarian/instructor team (yes/no) – A librarian and at least one non-librarian are both instructors of record for the course/section.

● Credit-bearing library class, taught by librarian(s) (yes/no) – Librarian(s) is/are the only instructor(s) of record.

○ Number of units or credit hours (#) ○ Full-credit course or partial-credit course ○ Quarter, semester, or other

ANALYST: course/section level variables created from the individual library instruction session data logged by the librarians. Counts of sessions per, librarians per, combinations of teaching methods used, confirming one-shot status for a course/section (i.e. not more than one session was logged), general data groupings and data management based on data that was logged.

● Count of number of class meetings with a librarian (#) – The number of times a librarian met with the course/section as a class.

● Count of number of individual librarians/library-instructors that interacted with this course/section (#)

● Confirm: One-shot instruction for this course/section (yes/no) – Did only one (1) library instruction session occur for this course/section?

● Confirm: The class meeting format for this course/section was: ○ In-person only sessions (yes/no) – The IL instruction session(s) happened in-

person only, and never online. ○ Online only sessions (yes/no) – The IL instruction session(s) happened online

only, and never in-person. ○ Hybrid (yes/no) – The IL instruction session(s) were a mix of in-person and

online meetings.



Institutional Variables Student's unique de-identified code (permanently assigned for longitudinal purposes)

Student's Gender, Ethnicity, Class Level (fr, so, jr, sr), Age, ESL required

Student's total # of credits earned (cumulative at 1st year, 2nd year, 3rd year, etc.)

Student's Fall 2014 to Fall 2015 Retention designation (Yes/No)

Student's Fall 2014 term GPA, # credits attempted & completed

Student's Winter 2015 term GPA (if applicable), # credits attempted & completed

Student's Spring 2015 term GPA, # credits attempted & completed

Student's Summer 2015 term GPA (if applicable) , # credits attempted & completed

Student's First-Year GPA (2014-2015 Academic Year), # credits attempted & completed

All Courses Taken by the Student during the 2014-2015 Academic Year:

Course Number & Department, Course Section, Course Name, Term taken, Course Grade

Library Instruction:

Course-Sections taughtTeaching Method Used

First Year, First Time Students

Student Grades by Course-SectionStudent Retention after the first

year

All student & library data de-identified in order to obtain IRB

approval

Data Merged by Course-Section



Sample Library Instruction Data Collection Form



Sample Library Instruction Data



Library Instructor 1 Session Date Session Duration (in minutes)Participants Professor / Subject Instructor (Lastname, Firstname)Course SubjectCourse Number Course Section Session: Lecture Session: Flipped InstructionSession: Directed PracticeSession: Active LearningSession: One Shot InstructionSession: Online Tutorial UsedSession: Research Guide UsedSession: Helped Design AssignmentSession: Library Tour Givenlibr_nam sessdate sessdura students inst_nam crs_subj crs_numb crs_sect tm_lect tm_flipd tm_pract tm_activ oneshot tutorial resguide libassgn libtourSMU01 10/9/2014 90 18 Harvell DISC - Discernment & Discourse1313 003 Yes No No No Yes No No No NoSMU01 7/14/2014 60 15 Grumbles (Blackman) DISC - Discernment & Discourse1313 0012 Yes No No No Yes No Yes No NoSMU01 6/17/2014 60 8 Lange DISC - Discernment & Discourse1313 0011 Yes No No No Yes No Yes No NoSMU02 9/16/2014 80 17 Hall, Alane DISC - Discernment & Discourse1313 022 No Yes No No No Yes Yes No NoSMU02 9/16/2014 80 17 Hall, Alane DISC - Discernment & Discourse1313 023 No Yes No No No Yes Yes No NoSMU02 9/29/2014 60 17 Kolberg, Stephanie DISC - Discernment & Discourse1313 028 No Yes No No No Yes Yes No NoSMU02 9/29/2014 60 17 Kolberg, Stephanie DISC - Discernment & Discourse1313 032 No Yes No No No Yes Yes No NoSMU02 9/29/2014 60 17 Kolberg, Stephanie DISC - Discernment & Discourse1313 033 No Yes No No No Yes Yes No NoSMU02 10/2/2014 80 17 Mueller, Mary Katherine DISC - Discernment & Discourse1312 055 Yes No Yes No No No No No YesSMU02 10/2/2014 80 15 Mueller, Mary Katherine DISC - Discernment & Discourse1312 056 Yes No Yes No No No No No YesSMU02 10/2/2014 80 16 Mueller, Mary Katherine DISC - Discernment & Discourse1312 057 Yes No Yes No No No No No YesSMU02 10/9/2014 30 20 Umobong, Rebecca HDEV - Human Development Studies1110 006 Yes No No No Yes No Yes No NoSMU02 10/9/2014 80 16 Newton, Pauline DISC - Discernment & Discourse1313 017 Yes No No Yes Yes No Yes No NoSMU02 10/9/2014 80 15 Newton, Pauline DISC - Discernment & Discourse1313 027 Yes No No Yes Yes No Yes No NoSMU03 9/16/2014 80 17 Newton, Pauline DISC - Discernment & Discourse1313 011 No Yes No No Yes Yes No Yes NoSMU03 9/16/2014 80 17 Newton, Pauline DISC - Discernment & Discourse1313 012 No Yes No No Yes Yes No Yes NoSMU04 11/4/2014 75 17 Whitehead, Barbara DISC - Discernment & Discourse1313 006 Yes Yes Yes Yes Yes Yes Yes No NoSMU01 11/4/2014 90 16 Levy DISC - Discernment & Discourse1313 007 Yes No No No Yes No Yes No NoSMU01 11/12/2014 60 10 Lange DISC - Discernment & Discourse1313 008 Yes No No No Yes No Yes No NoSMU01 11/12/2014 60 16 Lange DISC - Discernment & Discourse1313 009 Yes No No No Yes No Yes No NoSMU01 11/12/2014 60 15 Lange DISC - Discernment & Discourse1313 010 Yes No No No Yes No Yes No NoSMU02 11/6/2014 80 16 Wood, Angela DISC - Discernment & Discourse1313 036 Yes Yes No Yes No Yes No No NoSMU02 11/7/2014 50 7 Maitland, Annie ENGL - English1400 001 Yes No Yes Yes No No No Yes NoSMU02 11/10/2014 50 9 Maitland, Annie ENGL - English1400 001 Yes No Yes Yes No No No Yes NoSMU02 11/11/2014 80 16 Wood, Angela DISC - Discernment & Discourse1313 036 No No Yes No No No No No NoSMU02 11/14/2014 50 17 Seaney, Ona DISC - Discernment & Discourse1312 010 Yes No No No No No No No YesSMU02 11/14/2014 50 15 Seaney, Ona DISC - Discernment & Discourse1312 011 Yes No No No No No No No YesSMU02 11/12/2014 50 17 Diaconoff, Cara DISC - Discernment & Discourse1312 023 Yes No Yes No No No No No YesSMU02 11/17/2014 50 9 Maitland, Annie ENGL - English1400 001 Yes No Yes Yes No No No Yes NoSMU02 11/18/2014 80 17 Polster, Kristen DISC - Discernment & Discourse1313 004 Yes No Yes Yes Yes No No No NoSMU02 11/18/2014 90 17 Polster, Kristen DISC - Discernment & Discourse1313 005 Yes No Yes Yes Yes No No No NoSMU04 11/21/2014 60 23 Cloward, Karisa PLSC - Political Science1340 001 Yes No Yes Yes Yes No Yes No NoSMU02 11/21/2014 50 7 Maitland, Annie ENGL - English1400 001 No No No No No No No No NoSMU02 11/24/2014 50 9 Maitland, Annie ENGL - English1400 001 No No No No No No No No NoSMU05 7/29/2014 90 7 Pocklington DISC - Discernment & Discourse1312 2545 Yes No Yes Yes No No No No NoSMU05 7/22/2014 90 7 Pocklington DISC - Discernment & Discourse1312 2545 Yes No Yes Yes No No No No NoSMU05 7/15/2014 90 7 Pocklington DISC - Discernment & Discourse1312 2545 Yes No Yes Yes No No No No No

Massive Dataset of all first year students across 20

institutions

Planned Analysis

Stratified Control Samples Instruction vs. No Instruction:

Effect? Instruction Methods: Effects?

Likely analysis will consist of hierarchical model, taking into account groupings of similar institutions, similar student groups



Control Group matched on key demographics:

First Year StudentsWith Instruction vs. No

Instruction



Students withLibrary

InstructionStudents without

LibraryInstruction

Effect of Specific Teaching Methods



Students withInstructionMethod A

Students withInstruction Method B

Students withInstruction Method C

Primary Investigator has been named:Dr. Jade WinnAsst. Dean of Instruction, Assessment & EngagementUniversity of Southern California

IRB approval obtained at USC, UNLV, ASU

GWLA member university deans & directorsrecently approved funding to hire data analyst

Southern Methodist University will play a key role in data analysis and secure storage of data forall 20 participating institutions



Research Project – progress made in 2015

Questions?

Thank you!



from micro to macro: correlating instruction with student learning & success through local and...

Documents

university of texas

boulder university of

gwla institutions

gwla universities

gwla liaison

research libraries

student learning success

gwla libraries impact