ndn pd day yarnall 2013
TRANSCRIPT
Louise Yarnall, Ph.D.SRI International
Notre Dame de Namur Faculty Professional Development
January 6, 2013
Background
Center for Technology in Learning
Introduction Next Generation Learning
Challenge, Gates Foundation
Domain-Specific Assessment, U.S. Department of Education
Community College Partnerships’ Instructional Impacts, National Science Foundation
Destination Scenario-Based Learning, National Science Foundation
Key Topics What are some innovative higher ed uses of
technology in and out of the classroom?
What technology supports faculty creation and use of formative and summative assessments both in and out of the classroom?
How can I use technology to teach, hone, and/or assess critical thinking skills and information literacy?
Technology at inflection point Good bandwidth, wireless access
Greater access to technology for most students
Broader range of types of learning technologies
Greater portability via mobile phones
More faculty comfortable with technology
Higher ed technology use trends 20% of all undergraduates & 22% postbaccalaureate
students took at least 1 distance learning course (NCES, 2011)
4% undergraduates and 9% postbaccalaureate students took entire program online
95% 4-year undergraduates, 93% graduate students, and 78% 2-year students have home broadband (Pew Internet, 2011)
>94% college students have cell phones; 70% have laptop computer
4-9% college students have tablet or e-books
Innovative higher ed uses of technology “Disruptive” or “Old Wine, New Bottles”?
E-Books
Open Education Resources
E-Books Lower cost
Highlighting & annotation capability
Enhanced features:
Social networking
Quizzes of student comprehension
Shared student notes on chapters
BioBook Navigate content TOC
flexibly
Required comprehension quizzes, prof can review
Interact with other students around text, suggest text revisions
Inquire iPad-compatible, Campbell Biology
Enhanced highlighting, running record of notes
Artificial intelligence system poses “deeper learning” questions
E-Book Status Quo Study of students in 5 universities
(Internet2 eTextbook, 2012)
Appeal Lower cost
Highlighting capability
Down sides Faculty not using enhanced features: sharing notes, tracking
students, question/answer, additional links
No faculty use => No student use
Some problems with text enlargement in some platforms
College bookstores resist because undercuts their business model
Open Education Resources Began a decade ago with
MIT OpenCourseWareand Open University (England)
Image:
Colleges making their most popular lecturers’ talks available online
Disruptive technology to reinvent higher ed
Open Education Resources Also, a way to share text
material with open licenses
Datebase: http://www.oercommons.org/
Materials:
http://www.oercommons.org/landing/oer-training
Open Education Resources Status Quo Fact: Mostly online text
materials
Only 9% of users are educators
42% are independent learners (MIT; NYT, 2012)
42% students at other colleges (MIT; NYT, 2012)
Costly to put a course online ($30K-$40K)
Few offer credentials
Open Education Resources Status Quo Faculty (n = 4,678)
reported they use web to obtain (McMartin et al., 2008): Graphics for instruction
Primary texts for students
Very few seek animations or simulations or teaching/learning support
Barriers are: Lack of time to vet online
materials
Open Education Resources v2. Carnegie Mellon’s Open
Learning Initiative $250K per course Focus on low-income
students, first-time college students
Virtual labs, simulations, continuous feedback
Peer-to-Peer University, University of the People Social support around
materials Some provision of
credential
Open Education Resources v2. Next step for faculty:
Social recommendation system is encouraged: “Learning Registry”
Use them, provide feedback/rating
Open Education Resources Gooru:
http://www.goorulearning.org/gooru/index.g#!/home
Sample of materials that faculty can curate in online datebase
Technology and Assessment What technology supports faculty creation and use of
formative and summative assessments both in and out of the classroom?
Adaptive assessment Identify the learning
experiences that will benefit a particular learner (U.S. Dept. of Education, Draft National Education Technology Plan, 2010)
Knowledge-check quizzes, low-stakes, after lecture/reading
“Muddiest point” checks, after lecture/reading
Short essay prompts asking students to connect concepts after learning activity
Interactive simulations that students manipulate; probes check understanding
National Training and Education Resource (NTER)
Open Source LMS
Tools to create own 3-D simulations
Seek to instrument for learner analytics
Online Assessment DesignTools Prototyped online tool to
improve assessment design and production
Simplified version of PADI system used by professional test builders
Useful for describing hard-to-test knowledge and skills
Identifying range of ways to measure
Technology and Deeper Learning How can I use technology to teach, hone, and/or
assess critical thinking skills and information literacy?
Rubrics for complex tasks and performances Technology for rubrics
Complex tasks and research reports Grade with rubrics
Rubric tool online
http://rubistar.4teachers.org/index.php?screen=NewRubric
Technology and Deeper Learning Online threaded
discussions around student group projects Instructor who “lurks” to
track/coach team decisions
Online concept maps to check students’ planning during complex projects
Design Principles: http://www.edu-design-principles.org/dp/designHome.php
Technology and Deeper Learning Mobile phone reminders
to keep students focused on time management
Remind101: https://www.remind101.com/
The non-technological side of assessing deeper learning All these technological tools depend on a sound use
model
Close: A few points from my R&D work with higher edfaculty around deeper learning
Study their practice:
How standards are defined
How variations in student prep affects coverage of standards
How learning objectives are articulated
How student learning is measured
What faculty most want students to learn
Tradeoffs faculty make in designing assessments
Problems with higher-order assessment Not consistently applied
Every teacher applies rubrics differently: Weak reliability
Global vs. fine-grained
Solution: Group calibration on student work “Knowledge progressions”
Future technology Instructor crowdsourcing
Problems with higher-order assessment Tradeoffs between what learning
faculty value and what learning their assessments emphasize
OK content validity BUT
Poor construct validity
Solution: Group calibration on learning objectives
Balance among lower-order knowledge and higher-order application
Problems with higher-order assessment
Easy
Hard
Improving the construct validity of critical thinking learning standards Research indicates there
are some commonalities of good reasoning across domains
But many important differences
You improve your capacity to meet critical thinking standards by understanding those differences
Evidence that critical thinking is elusive in college
Omnibus Test Results:DSA: R2 = 0.15, F(2, 145) = 12.54, p< .0001Test B: R2 = 0.16, F(2, 129) = 12.43, p< .0001Test C: non-significant
Pairwise Contrast (Tukey’s)B1 vs. B2, B2 vs. B3, B1 vs. B3DSA effect size range: 0.53 to 1.31DSA t statistics: 3.06 – 4.90DSA p values: <0.001 – 0.007
More evidence that critical thinking is elusive in college
Omnibus Test Results:DSA: R2 = 0.29, F(2, 144) = 29.64, p< .0001Test B: R2 = 0.13, F(2, 134) = 9.78, p< .0001Test C: non-significant
Pairwise Contrast (Tukey’s)B1 vs. B2, B2 vs. B3, B1 vs. B3DSA effect size range: 0.62 to 1.60DSA t statistics: 4.11 – 7.52DSA p values: all <0.001
Is there a difference for learning?Between using this test question…
And this one?
Cost description Amount
a. Sleeping bag she bought for camping last year $90
b. Bus ride to the store $10
c. Lost income from a part-time night shift job $80
d. Cost of the console $500
Benjamin wanted to go to a baseball game when his favorite team was playing at home. He was quite upset when he found out that all the tickets were sold out.
So, the game night, he approached scalpers who were selling tickets on the sidewalk. Benjamin asked a few scalpers how much they wanted for a ticket.
The first one wanted much more than the original price, so Benjamin asked another one who wanted even more. He kept on asking around and got different prices.
After an hour, it was getting closer to the game time and he saw that the prices were not lower. He went back only to discover that the first seller had doubled the price.
Using economic terms and reasoning, explain why the prices of tickets went up.
Which of the cost items (a ~ d) are sunk costs?
A girl decides to wait in line all night to be the first to get a new game console.
Solution: Calibrate critical thinking for your subject better
Know your tradeoffs!
Schematic
Declarative Procedural
Strategic
Shavelson et al.
What a college sophomore should know about energy flow and matter cycling…
What a college sophomore should know about supply and demand….
A way to think about assessment to avoid making bad tradeoffs
)(
)(
1 ii
ii
bDa
bDa
e
e
A way to think about assessment to avoid making bad tradeoffs
)(
)(
1 ii
ii
bDa
bDa
e
e
Documentation to help you avoid bad tradeoffs
Attribute Description Model
Focal Knowledge The primary knowledge/skill/abilities targeted bythis design pattern
Student
Additional Knowledge
Other knowledge that may be required by this design pattern
Student
PotentialObservations
Some possible things one could see students doing that would give evidence about student knowledge
Evidence
Potential Work Products
Modes, like a written product or a spoken answer, in which students might produce evidence about student knowledge
Task
CharacteristicTask Features
Aspects of assessment situations that are likely to evoke the desired evidence
Task
Variable Task Features
Aspects of assessment situations that can be varied in order to shift difficulty or focus.
Task
Critical thinking varies by domain Talk to experts who work in industry and use the key
knowledge and skills you teach
Talk to academic researchers and innovators who use the key knowledge and skills you teach
Ask them: What knowledge and what skills do you use most of all?
Think about: What do I want to make sure my students know and use from my class 10 years after they graduate?
