1

Overview of the 2014/2015

Multi-State Collaborative Pilot Study

2

Ashley FinleySenior Director of Assessment and Research – AAC&U(Finley@aacu.org)

Bonnie OrcuttDirector of Learning Outcomes Assessment Academic Policy - Massachusetts Department of Higher Education(Borcutt@bhe.mass.edu)

Presenters Additional Collaborators

Julie CarnahanSenior Associate - SHEEO

MSC Project Director

(Jcarnahan@sheeo.org)

Terrel RhodesVice President for the Office

of Quality, Curriculum and

Assessment – AAC&U

VALUE Project Director

(Rhodes@aacu.org)

3

Pilot Study Webinar Topics Participants Overview of Pilot Study versus Model Purpose and impacts Timeline IRB and Student Consent Faculty and Campus Responsibilities Overview of pilot study communication structure Outcomes, Assignment Parameters, and the VALUE Rubric Faculty and Institution Benefits from Pilot Study Participation Sampling

◦ Artifact Collection and De-Identification Data Management System for the Pilot Study Scoring, Results Aggregation, and Dissemination of Results

Multi-State Collaborative to Advance Learning Outcomes Assessment

MSC Participants:

CT, IN, KY,

MA, MO, MN,

OR, RI, and

UT

Steering Committee: State point persons from each state with representatives from SHEEO & AACU.

Institution Point Persons: Institution point person from each campus in each state.

Pilot Study Team: Representatives from states, SHEEO, & AAC&U

5

From Model to PilotThe Pilot

Multiple measures of learning outcomes – direct and indirect

Demonstration of learning across multiple modalities such as projects, presentations, papers, performances and other assessed using the AAC&U VALUE rubrics

Assessment of learning at multiple points along students’ academic path

A representative sample of states, institutions, and students in order to create useful benchmarks of learning to guide and inform campus-level work and public accountability

Assessment management system (i.e. database) that will enable electronic uploading of campus-level data, online access of student work for scoring, analysis of state data, generation of reports, while assuring confidentiality & security of data

Two Outcomes: Written communication and Quantitative literacy (Optional: Critical thinking)

Written work only

Assessment of learning for students who have completed 75% of total number of credits to be graduated as of the start of the fall semester

9 states, approximately 70 public institutions including both 2 & 4 year campuses

Assessment management system (i.e. database)– Taskstream -- has been chosen for pilot study

The Model

6

The Pilot Study is Process-Oriented

Focus: Proof of Concept:

◦ Ability to produce (1)useful assessment data for campus use,(2) to measure student learning using student artifacts as assessed with the LEAP VALUE rubrics (3) to organize aggregated data for comparison at the segment level and for presentation

Proof of Feasibility:◦ Ability to (1) scale up (2) to sustain campus effort (3) to develop

reliable and cost-effective, sustainable scoring protocols

Proof of Reliability:◦ Analyze inter-rater reliability for a given level (multistate, state, and/or

local scoring) and between these levels

Purpose of Pilot Study

7

What is the intended long-term effect of the work?

Establish proof of process that direct assessment of student work using the AAC&U VALUE rubrics can be coordinated and meaningfully analyzed to inform efforts to improve student learning across and within states and at the institutional level.

  Establish proof of reliability with respect to centralized and decentralized scoring of

student work

  Establish support for validity of assessment measure through random analysis of

assignment instructions, student work and examination of trends reflected at the multi-state, state, and campus-levels

Establish areas where additional professional development is needed and inform focus of activities.

Generate conversations and/or changes on campus(es) regarding assignment design and/or curriculum

 

Anticipated Pilot Study Impacts

8

Pilot Study Timeline & MilestonesPhase I: Spring/Summer 2014

◦ Selection of pilot campuses (completed)◦ Development of sampling protocols and assignment parameters (finalized)◦ Identification of campuses, state point persons, and campus leads (completed)◦ Faculty development workshops, webinars, and summits (underway)

Fall 2014◦ Development of scoring protocols and guidelines for selecting faculty scorers◦ Selection of faculty scorers◦ Continued faculty development activities◦ Collection, de-identification, coding, and uploading of artifacts◦ Creation of Artifact Information File

Phase II: Spring 2015

◦ Scoring of artifacts◦ Data analysis and presentation of results

9

Ponderings for IRB and for upcoming IRB Webinar What is Assessment Research:Taken from the Belmont Report:“…the term "research' designates an activity designed to test an hypothesis, permit conclusions to be drawn, and thereby to develop or contribute to generalizable knowledge (expressed, for example, in theories, principles, and statements of relationships). Research is usually described in a formal protocol that sets forth an objective and a set of procedures designed to reach that objective.”

Definition of Human Subject:Human subject means a living individual about whom an investigator (whether professional or student) conducting research obtains:

1. Data through intervention or interaction with the individuals, or2. Identifiable private information.

-45CFR46102(f)Observation of individuals without interaction or identity determination does not count as “human subjects research”.It is only private information if one of the following applies:

1. It is gathered through intervention or interaction with the individuals studied, and/or2. It contains identifiable private information.

10

Ponderings for IRB and for upcoming IRB Webinar

Waiver of Informed Consent◦ The research involves no more than minimal risk to the subjects;◦ The waiver or alteration will not adversely affect the rights and welfare of the subjects;◦ The research could not practicably be carried out without the waiver or alteration; and◦ Whenever appropriate, the subjects will be provided with additional pertinent information

after participation.

What is Minimal Risk:

“Minimal risk means that the probability and magnitude of harm or discomfort anticipated in the research are not greater in and of themselves than those ordinarily encountered in daily life or during the performance of routine physical or psychological examinations or tests.”— 45CFR46102(i)

Does Assessment of Student Learning fall under this Category for Exemption?

Educational Practices

“Research conducted in established or commonly accepted educational settings, involving

normal educational practices, such as (i) research on regular and special education

instructional strategies, or (ii) research on the effectiveness of or the comparison among

instructional techniques, curricula, or classroom management methods.” — 45CFR46.101(b)(1)

11

Assignment Parameters and Assignment Coversheet

Broad Assignment Parameters:◦ Facilitate the development and redesign of assignments to increase the

degree of consistency across assignments

◦ Enhance the validity of our measure of student learning

◦ If we impose these parameters, are we now “Teaching to the Test?”

Assignment Coversheet:◦ Allows the faculty to indicate which dimensions of a learning outcome they

designed their assignment to be assessed for

◦ Use of the coversheet for second level analysis

◦ Webinar on Assignment Parameters, Design and Redesign will be offered next week.

12

Taskstream selected as vendor for pilot study

Requirements of selected vendor:◦ To be able to work with any platform a campus may be using◦ To be user friendly◦ To be able to provide direct support to campus point persons◦ To provide analysis across multiple data points

Database will enable:◦ Uploading of artifacts for eventual scoring (2015)◦ Generation of reports at the national, multi-state, state, and individual

campus levels◦ Confidentiality of institutions and anonymity of students and faculty

Data Management System

13

Campus Level Implementation Responsibilities of Faculty

◦ Participation in professional development activities, as needed.◦ Assignment design, redesign and alignment with outcomes◦ Submit assignment, assignment coversheet, student work

Responsibilities of Professional Staff and Administration◦ Provide professional development activities ◦ Collect, de-identify and code student work to ensure confidentiality of the

student and faculty member ◦ Create an artifact information file containing state, institution and student

characteristics◦ Submit coded material to the data management system

14

Who do I go to when I have a question?

Participating Faculty

Institution Pilot Study Leads

State Point Persons

MSC Steering Committee and

Julie Carnahan, SHEEO Project Director

Campus Level Implementation

Institution Pilot Study Lead

State Point Person

MSC Steering Committee and Julie Carnahan, SHEEOProject Director

MSC Pilot Study Team and, if needed, the MSC Subgroups (members of each of these groups come from institutions within each participating state)

15

Sampling guidelines have been finalized to ensure reasonably representative and random samples. To view them, go to:

Sampling Guideline Highlights Eligible Student Population

Sample Size:o Targeted minimum of 75 – 100 artifacts per outcome per institution o Maximum of 10 artifacts per course and/or per faculty member o A single artifact may only be used for one outcomeo Maximum of one artifact per student

Sampling plans will be submitted for review and feedback will be provided

Sampling Consultants: Peter Ewell (NCHEMS) Gary Pike (IUPUI)

Brief Overview of Sampling Guidelines

16

Artifact Collection and

De-Identification

How will confidentiality and anonymity be ensured? The pilot study will involve multiple levels of de-identification:

Campus Level: Confidentiality will be maintained at the campus level. The institution collecting the student work will strip the work of all identifying information – course name, course number, faculty name, student name. Campus will electronically upload artifacts into data management system.

State Level: Faculty and student anonymity and institution confidentiality will be maintained at the state level. Student work assessed at the state-level will be stripped of any identifying information indicating the institution the work came from.

Multistate Level: Faculty, student, and institution anonymity will be

maintained at the multi-state level.

17

Why Participate?Institutional Benefits Faculty Benefits

Advance assessment of essential learning outcomes on campuses

Development of sustainable assessment plans and sampling strategies

Ability to contribute to development of state and multi-state benchmarks for outcomes

Gain insight and understanding into direct assessment of essential learning outcomes using rubrics

Develop frameworks for intentional assignment design

Participate and contribute to faculty learning communities on and across campuses

18

Scoring: Faculty scorers will be blind to the institution and state that artifacts are drawn from. Faculty will be precluded from assessing artifacts they submitted. Aggregation of Results: Analysis of the ability to aggregate results at the state and multi-state level by segment – two-year and four-year – for meaningful benchmarking, state-to-state comparisons and for presentation.

No Public Reporting Out: Pilot study results will not be reported out publically.

Scoring and Results Aggregation

19

Institution results will not be identifiable at the state and multi-state level with the exception of states with a relatively small number of institutions. Student work, assignment instructions and trial data analysis will be coded in order for materials and results to be returned to each participating institution

Identifiable trends at the institution, state and multi-state level will be available while maintaining institution anonymity.

Data disseminated to states will be aggregated by segment. Confidentiality of institutional identity will be maintained.

Dissemination of Results

20

Pilot Study Overview Webinar May 9, 5-6pm (ET) (repeat of May 8 webinar) Assignment Design Webinar

May 13, 4-5pm (ET) and May 14, 4-5pm (ET) Sampling for the Pilot Study

May 21, 4-5pm (ET) and May 22, 4-5pm (ET) Multi-State Assessment: IRB & Student Consent Date and Time TBD Coding, Formatting, Submitting: Using Taskstream Date and Time TBD

Webinars will be recorded and posted to: http://www.sheeo.org/msc

Webinars already posted: Welcome to the MSC

Additional MSC Webinars

21

Questions?