Download - NCLA Learning Outcomes Assessment Workshop
1
Assessing for Improvement
Diane HarveyNCLA-BIG WorkshopMay 21, 2010
2
Learning outcomes for today
At the end of this session, participants will be able to:
1. Describe the learning outcomes assessment process
in libraries.
2. Write learning outcomes for an information literacy instruction session.
3. Choose appropriate assessment methods.
4. Define ways to work with assessment results.
Assessment climate in higher education
3
Spellings Commission http://www.ed.gov/about/bdscomm/list/hiedfuture/index.html
Voluntary System of Accountability (VSA) http://www.voluntarysystem.org/
Accrediting Agencies http://www.sacs.org/
SACS standards
3.3.1 …identifies expected outcomes, assesses the extent to which it achieves these outcomes, and provides evidence of improvement based on analysis of the results
3.8.2 …ensures that users have access to regular and timely instruction in the use of the library and other learning/information resources.
4
Assessing for improvement5
We don’t assess to prove…
…but to improve.D. Stufflebeam
Assessment in libraries6
Focus on collections and services (e.g. ARL LibQUAL)
Focus on standards (e.g. ACRL IL Competency Standards)
What is LOA: learning outcomes assessment?7
Systematic look at what students are learning
Moving from “What am I going to teach today?” to “What do I want students to learn today?”
What LOA is NOT8
NOT evaluation of teaching
NOT evaluation of program
Why assess student learning in libraries?9
Affirm commitment to student learning
Improve effectiveness of instructional programs
Align with campus, disciplinary, and higher ed efforts
Be recognized on campus for contributions to student learning
10
Assessment loop
Develop student learning
outcomes
Set criteria
Devise assessment measures
Do assessment
Work with results
11
Levels of assessment
Institutional (University wide)
Program (e.g. Department)
Course
Session (e.g. IL instruction)
LOA in libraries: constraints12
Lack of consistent, reliable access to students
IL instruction: is it a “program”?
Program assessment vs. session assessment
Difficult to isolate effects of library instruction
Know your IL program!13
Do you offer a for-credit IL course?
Do you provide IL instruction for large programs (e.g. freshman writing)?
Is IL integrated into the curriculum?
Do you offer IL for capstone, thesis or senior seminar courses?
Map your IL program14
First Year Writingand Freshman Seminars
Subject specific instruction
Honors capstone seminars
Where will you assess student learning?15
On the instruction session level?
In a for-credit library skills course?
Across a series of library instruction sessions?
Broadly across the student population?
Remember…16
Library instruction doesn’t work the same way as instruction in an academic department.
Know the shape of your program – be able to explain how and why you assess student learning.
17
18
Some simple approaches19
Develop learning outcomes for your instruction program, and look at where they are taught – “snapshots”
Focus on large programmatic efforts e.g. first year writing
Highlight specific academic departments
Assess individual library instruction sessions – develop shared learning outcomes, compare/contrast
Snapshots20
Outcome: Students will be able to distinguish between popular and scholarly journals.
Assessment: pre/post test in freshman English, one minute paper in senior capstone
21
What is a learning outcome?
A learning outcome is one sentence that indicates what students should represent, demonstrate or produce as a result of what they learn.
- source: Peggy Maki
22
Good learning outcomes:
Focus on what students will learn/know/be able to do
Describe actions or behaviorsAre results orientedAre observable and measurableInclude a time frame
23
ACRL Standards
3 levels: standards, performance indicators, outcomes
Can be used for LOA
Examples: (1)Identifies keywords, synonyms and related terms for the information needed. (2) Selects an appropriate documentation style and uses it consistently to cite sources
Writing outcomes24
Today, we’ll write learning outcomes for a library instruction session.
25
Exercise: What do you cover?
Think of an information literacy instruction session that you teach on a regular basis.
List what you would cover in that session.
26
Bloom’s Taxonomy
Classification of educational objectivesPublished in 1956, revised in 2001 (changes: noun to
verb, synthesis/creating becomes highest level)
Taxonomy = classificationCognitive levels (lower higher)Provides way to express outcomes
27
Bloom’s levels (lowest highest)
Knowledge/RememberingComprehension/UnderstandingApplication/ApplyingAnalysis/Analyzing Evaluation/EvaluatingSynthesis/Creating
28
Verbs for information literacy
Knowledge/Remembering: define, list, recognize Comprehension/Understanding:characterize, describe,
explain, identify, locate, recognize, sort Application/Applying: choose, demonstrate, implement,
perform Analysis/Analyzing: analyze, categorize, compare,
differentiate Evaluation/Evaluating: assess, critique, evaluate, rank, rate Synthesis/Creating: construct, design, formulate, organize,
synthesize
Verbs to avoid29
Understand
Appreciate
Know about
Become familiar with
Learn about
Become aware of
30
Learning outcomes formula
1. Time frame2. Student focus3. Action verb4. Product/process/outcome
31
Using the formula
Time frame: “At the end of the library session…”
Student focus: “…students will be able to…”Action verb: “…identify…”Product/process: “…a relevant database for
their term paper research.”
32
Another example:
Time frame: “After completing the online tutorial…”
Student focus: “…students will be able to…”Action verb: “…differentiate between…”Product/process: “…scholarly journals and
popular magazines.”
33
One more example:
Time frame: “By the time they complete their undergraduate program…”
Student focus: “…students will be able to…”Action verb: “…construct…”Product/process: “…a research question that
can be investigated using primary archival resources.”
34
Examples: bad and better
Bad: Students will understand how to use social science databases.
Better: Students will perform a search in Social Sciences Abstracts that retrieves relevant items.
Bad: Students will be able to search the catalog.Better: Students will use the catalog to identify a book on their
topic.
Bad: Students will appreciate the importance of correct citation.Better: Students will produce citations in correct APA format.
35
Now, write your own!
Choose two items from the list of what you cover in an information literacy session.
Transform them into learning outcomes, using the formula.
Find a partner and critique your outcomes using the following checklist.
36
Checklist for learning outcomes
Includes a time frame?Focuses on students?Uses action verbs?Names a product or process?Is measurable/observable?Prompts a measure/method?Will be useful for you to assess?
Sample size37
Use any convenient sample that will give you useable information.
Make sure that groups of interest are represented.
Sample size is important only if you plan to publish your results.
Do you want to generalize to the entire student population?
Assessment Methods
Ask yourself
“How will I know?”
38
39
Assessment methods
Knowledge testOne Minute paper & variationsBibliography analysisConcept InventoryStandardized test
Knowledge tests40
Tests knowledge and/or skills before and/or after library instruction session.
Can be given at end of library session or later in semester.
Can use clickers to gather data during library session.
Sample questions:1. What is the difference between a library catalog
and a database?2. The Boolean operator “or” narrows a search statement (true/false).
One minute paper & variations41
Sample questions:
1. What is the most important thing about library research you learned today?
2. 3-2-1 (three things you learned, two things you’re still confused about, one thing you’d change about session)
3. What is one question you still have?
4. In your research, what will you do differently after today’s session?
Bibliography analysis42
Look for citations from scholarly/peer reviewed journals
Look for citations for books and journals owned by your library
Look for articles retrieved from your databases.
Concept inventory43
Make a checklist of 3-12 important concepts students need to master.
Ask students to explain each concept in a sentence or two. If a concept is unfamiliar, they should leave blank.
Count good responses for each concept, then plan future instruction.
Standardized tests44
I-Skills
ILAT (James Madison University)
SAILS
I-Skills45
Developed by ETS7 ICT proficienciesTracked to ACRL standardsTask based, 2 sections, 75 minutes
SAILS: Standardized Assessment of Information Literacy Skills
46
45 questions 35-50 minutes Multiple choice Can require cooperation of faculty and administrators Sample question: If you wanted to search for a topic that
has several components, such as nutrition for pregnant women, which operator would you use? (and, or, not, adj)
Source: Project Sails https://www.projectsails.org/abouttest/samples.php
ILAT – James Madison University47
Web based60 itemsTests ACRL standards 1,2,3 and 5
48
Using assessment information
Rewrite learning outcomesChange what you do/how you teachWork collaboratively with colleagues
(librarians and faculty)Revise assessment measuresShare the news in the library and on campus
49
Learning outcomes are:
Fundamental components of a learning assessment program
Focused on student learningIndicate what students will know/be able to doMeasurable, observable, overt
Learning Outcomes Assessment 50
A tool to help librarians
improve student learning
51
LOA resources
Angelo, Thomas (1993). Classroom assessment techniques. Maki, Peggy (2004). Assessing for learning : building a
sustainable commitment across the institution. Middle States Commission on Higher Education(2003).
Developing research & communication skills: guidelines for information literacy in the curriculum http://msche.org/publications/devskill050208135642.pdf
Neely, Teresa Y(2006). Information literacy assessment : standards-based tools and assignments.
Radcliff, Carolyn et.al (2007). A practical guide to information literacy assessment for academic librarians.
Student Learning Outcomes Assessment (University of Virginia)http://www.web.virginia.edu/iaas/assessment/outcomes.htm
52
Let’s assess this workshop!
53
Thank you!
Diane Harvey
Head, Instruction & Outreach
Perkins Library
Duke University