online pedagogy and evaluation candace chou university of st. thomas lhdt548 online teaching and...
Post on 31-Dec-2015
215 Views
Preview:
TRANSCRIPT
Online Pedagogy and Evaluation
Candace ChouUniversity of St. Thomas
LHDT548 Online Teaching and Evaluation
Pedagogical Models
• Pedagogical models are cognitive models or theoretical constructs derived from learning theory that enable the implementation of specific instructional and learning strategies (Dabbagh & Bannan-Ritland, 2005, p. 164).
Examples of Pedagogical Models
• From cognition theory and constructivism:– Learning communities or knowledge-
building communities– Cognitive apprenticeships– Situated learning– Problem-based learning– Microworlds, simulations, and virtual
learning environments– Cognitive flexibility hypertexts, and – Computer-supported intentional learning
environments (CSILEs)
Instructional Strategies
• Instructional strategies are what instructors or instructional systems do to facilitate student learning (Dabbagh & Bannan-Ritland, 2005, p. 203)
• The plan and techniques that the instructor/instructional designer uses to engage the learner and facilitate learning.
• Instructional strategies operationalize pedagiogigcal models
Seven Principles of Good Practice
1. Encourages contacts between learners and faculty
2. Develops reciprocity and cooperation among learners
3. Uses active learning techniques4. Gives prompt feedback5. Emphasizes time on task6. Communicates high expectations7. Respects diverse talents and ways of
learning(Chickering & Gamson, 1987)
Seven Principles and Technology Selection
Seven Principles Tools for evaluation1. Teacher/student contact Email, bulletin, forum, chat
2. Stud. reciprocity/cooperation
Chat, forum, IM, blog, sharing
3, Active learning techniques
Games, simul., interactive tools
4. Give prompt feedback Tutorials, quizzes, self-test
5. Time on task Scheduling and monitoring progress
6. High expectations Online publishing,blogs, wiki
7. Respect diverse talents “Personalisable” online environment
Reference: http://www.tltgroup.org/Seven/Library_TOC.htm
• know how to manage collaborative groups• Know how to leverage questioning strategies
effectively• Have subject matter expertise• Be able to coordinate and involve students in
activities• Have knowledge of basic learning theory• Have specific knowledge of distance learning
theory• Be able to correlate study guide with distance
media• Be able to apply graphic design and visual
thinkingReference: http://www.rodp.org/faculty/pedagogy.htm
What are the characteristics of a successful online instructor?
1. Organizes and prepares course materials2. Is highly motivated and enthusiastic3. Committed to teaching4. Has a philosophy supporting student-centered learning5. Is open to suggestions following pre- and post-learning
evaluations6. Demonstrates creativity7. Takes risks8. Manages time well9. Is interested in online delivery of courses with no real
rewards10. Responds to learners needs within the expectations
stated by instructor
What are the characteristics of a successful online learner?
Manag
es a
nd allo
cate
s tim
...
Pre
fers
linear
learn
ing s
tyle
Dis
plays
technol
ogy sk
ills
Can
dea
l with
tech
nology a.
.
Is an
act
ive
learn
er
Hig
hly m
otivat
ed, s
elf-d
irec...
Dep
ends o
n nat
ure o
f inst
ru...
Has
appro
priate
writ
ing a
n...
0% 0% 0% 0%0%0%
100%
0%
1. Manages and allocates time appropriately
2. Prefers linear learning style3. Displays technology skills4. Can deal with technology and its
frustrations5. Is an active learner6. Highly motivated, self-directed,
and self-starting7. Depends on nature of instructional
methods (group vs. individual tasks)
8. Has appropriate writing and reading skills for online learning
Reference: http://www.uwsa.edu/ttt/kircher.htm
More on Pedagogy
• Pedagogy of online teaching and learning, http://www.rodp.org/faculty/pedagogy.htm
• Pedagogy and Best Practices, http://vudat.msu.edu/breakfast_series/
Best Practices
• Organization guidelines• Assessment guidelines• Instruction/Teaching guidelines
Simonson, M., Smaldino, S., Albright, M., & Zvacek, S. (2009), pp. 155-158
Organization
• Each semester credit = 1 unit• Each unit = 3-5 modules• Each module = 3-5 topics• Each topic = 1 learning outcomes
• A typical three-credit course has 3 units, 12 modules, 48 topics, and 48 learning outcomes.
Assessment Guidelines
• 1 major assignment per unit• 1 minor assignment per two to three modules
• A typical three-credit course has the following assessment strategy:– 1 examination– 1, ten-page paper– 1 project– 3 quizzes– 3 small assignments (short paper, article review,
activity report)– Graded threaded discussions, e-mails, and chats
Instruction/Teaching Guidelines
• 1 module per week• Instructor e-mail to students each
week• 1 synchronous chat per week• 2 to 3 threaded discussion
questions per topic, or 6 to 10 questions per week
Module Design Template
• Objectives• Guiding Words• Readings• Explore (web resources or previous
examples)• Product (or assignment)• Optional standard alignment
Evaluation• Quality Matters: A comprehensive online
(or hybrid) course evaluation rubric in eight categories.– Course Overview and Introduction– Learning Objectives– Assessment and Measurement– Resources and Materials– Learner Engagement– Course Technology– Learner Support– Accessibility
http://www.qualitymatters.org/Rubric.htm
Assessment
• Assessment provides information whether learners have achieve specific learning objectives and goals. Designers and instructors could use the information to revise instruction during the course of instruction. The types of assessment include test, observations, self-check, surveys, etc. (Wiggins & McTighe, 2005)
Evaluation• Evaluation provides
information about the effectiveness of programs, policies, personnel, products, organization, etc.– Formative evaluation focuses
on the review of instructional materials and processes
– Summative evaluation focuses on the effectiveness of the instructional materials for decision on whether to adopt the materials for future instruction or not. (Smith & Ragan, 2005)
Examples
• Formative Evaluation– Conducted before and during the
process– Expert review– One-to-one evaluation– Small group– Field test
• Summative evaluation– Usually done at the end of a
project or class– Outcomes and impact evaluation– End of course evaluation
Informal Formal
Student feedbackStudent experiencesStudent expectationsTeacher-constructed tests and observations
Student feedbackStudent experiencesStudent expectationsTeacher-constructed tests and observations
Comparisons of pre- & post-outcomesIn-depth qualitative observations and interviewsBehavior logs
Comparisons of pre- & post-outcomesIn-depth qualitative observations and interviewsBehavior logs
Comparison studies with control group and nonrandom assignment of participants
Comparison studies with control group and nonrandom assignment of participants
Controlled studies with control group and random assignment of participants and control groups (experimental studies)
Controlled studies with control group and random assignment of participants and control groups (experimental studies)
An impact on evaluator’s practiceAn impact on evaluator’s practice
Insights for other practitioners, researchers, and evaluators to consider
Insights for other practitioners, researchers, and evaluators to consider
Information on changes in learning or performance in the specific setting
Information on changes in learning or performance in the specific setting
Generalizable results that can inform other settings
Generalizable results that can inform other settings
Conclusions based on:Conclusions based on:
Results provide:Results provide:
Evaluation Continuum
Dabbagh & Bannan-Ritland, 2005, p. 236
Kirkpatrick’s Model
•Four Levels of Evaluation
•Reaction•Learning•Behavior•Results
Kirkpatrick (1998). Evaluating training programs.
Kirkpatrick’s Model
• Reaction: how learners perceive online instruction or training
• Examples– Voting (student response system)– Post-training surveys– Personal reaction to the training– Verbal reaction– Written report
Kirkpatrick’s Model
• Learning: the extent to which learners change attitudes, gain knowledge, or increase skill in online learning or training
• Examples– Pre- and post-tests– Interview– Observation
Kirkpatrick’s Model
• Behavior: how learners have changed their behavior as a result of online instruction or training
• Examples– Observation or interview over time– Self assessment (with carefully
designed criteria and measurement)
Kirkpatrick’s Model
• Results: the final results that have occurred at the organization level as a result of the delivery of online instruction or training
• Examples– The reduction of accidents– An increase in sales volume– An increase in employee retention– An increase in student enrollment
Assessment Tools
• Online Assessment Tools https://www4.nau.edu/assessment/main/research/webtools.htm
• Types of Online Assessment http://www.southalabama.edu/oll/pedagogy/assessmentslecture.htm
• Rubrics for Assessment http://www.uwstout.edu/soe/profdev/rubrics.shtml
• Web-based surveys– SurveyMonkey, http://surveymonkey,com– How to use SurveyMoneky video,
http://www.youtube.com/watch?v=pUywfcdrnoU – Zoomerange, http://info.zoomerang.com/index.htm – Google Form, http://docs.google.com
Usability Testing
• The next few slides on Usability are modified from Carol Barnum’s Keynote Speech at E-Learn 2007 Conference with permission
• The original PPT can be found at http://www.aace.org/conf/elearn/speakers/barnum.htm
The Problem
“most major producers of e-learning are not doing substantial usability testing…In fact, we don’t seem to even have a way to talk about usability in the context of e-learning.”
Michael Feldstein, “What is ‘usable’ e-learning?” eLearn Magazine (2002)
UA versus QA
Usability Testing– Focus is on user– User’s satisfaction
with product– Ease of use– Ease of self-
learning– Intuitiveness of
product
QA Testing– Focus is on
product– Functional
operation tests for errors
– Performance/benchmark testing
– Click button, get desired action
What is usability?• “The extent to which a product can be
used by specified users to achieve specified goals in a specified context of use with effectiveness, efficiency, and satisfaction.” (ISO 9241-11 International Organization for Standardization)
• “The measure of the quality of the user experience when interacting with something—whether a Web site, a traditional software application, or any other device the user can operate in some way or another.” (Nielsen, “What is ‘Usability’”?)
HE is one tool• Heuristic Evaluation
– Definition• Heuristic evaluation is done as a systematic
inspection of a user interface design for usability. The goal of heuristic evaluation is to find the usability problems in the design so that they can be attended to as part of an iterative design process. (Jakob, 2005)
– examples• Jakob Nielsen
(http://www.useit.com/papers/heuristic/)• Quesenbery’s 5 E’s (www.wqusability.com)• Dick Miller (www.stcsig.org/usability)
Personas - another tool• Definition• Examples
– Cooper (www.cooper.com/content/insights/newsletters_personas.asp)
• HE + personas = more powerful review– eLearn Magazine
• “Designing Usable, Self-Paced e-Learning Courses: A Practical Guide” (2006) Michael Feldstein
• “Want Better Courses? Just Add Usability” (2006) Lisa Neal and Michael Feldstein
The argument against utesting
• Time is money• Money is money• HE is a cheap alternative
– Discount usability method – Uncovers violations against rules– Cleans up the interface– Satisfies “usability by design”
Let’s hear it from the user
• User experience cannot be imagined• What can the user show us?
– how does the user navigate the online environment?
– How does the user find content?– how does the user respond to content?
• What can the user tell us?– think aloud protocol
• What are the user’s perceptions?– listen, observe, learn– evaluate survey responses with caution
Build UX into process
• How many users does it take?– cast of thousands – engineering
model– five or fewer - Nielsen discount model– RITE method - Rapid Iterative Testing
and Evaluation – Microsoft gaming model
Heuristics suggest test plan
– General navigation within Vista and a class– Consistency with general web design and
hyperlink conventions– Performing class-related tasks, such as
posting assignments– Responding to discussion board messages– Using non-class related tools, such as
Campus Bookmarks, Calendar, To Do List
Click on logo opens new web
page (webct.com). Users may
think of this page as a home
page, since this is the first page
users see after submitting vista
url. They may expect this logo
to represent a “link to home.”
These lines seems to clutter this space and instead of acting to delineate the listing. They cause the text to become less discernable by reducing figure-ground contrast.
This im
portant
instructi
on is not
signific
antly diffe
rent
from th
e adjacent
text.
Not all the items in this list are institutions.
User must scroll to see the complete listing
Extensive use of “Mouse-over” links.
Strangely, the logo is
now not an active link to
webct.com
This text does not have enough size contrast to be effective
Buttons links with
mouse-over
effect.
Colored hypertext links
Mouse-over links.
Inconsistent link design may confuse users; users may not be able to readily distinguish what is a link and what is not.
University identifier now missing
“File tab” functions as non-
traditional “home” button; file
tabs not used to navigate
anywhere else on site;
inconsistent navigation may
confuse users.
Introduction on iconic links;
adjacent text not a link.
Some of these tables have links
and some do not; also, some have
icons and some do not.
Some icons seem to represent their meaning
better than others.
The purpose of these text links and their proximity to the iconic
links is unclear.
The relevance of some content is questionable
The meaning and relevance of some titles are
unclear.
Users may not understand the meaning of these icons.
Videos
• Paper Prototype: http://youtube.com/watch?v=ppnRQD06ggY&feature=related
• http://youtube.com/watch?v=8ip4acENxZ4
References
• Chickering, A. W., & Gamson, Z. F. (1987). Seven principles for good practice in undergraduate education. Retrieved May 1, 2007, from http://honolulu.hawaii.edu/intranet/committees/FacDevCom/guidebk/teachtip/7princip.htm
• Dabbagh & Bannan-Ritland (2005). Online Learning. Upper Saddle River, NJ: Pearson.
• http://del.icio.us/ustmalt/pedagogy• http://del.icio.us/ustmalt/usability • Theory into Practice, http://tip.psychology.org/• Tips for training online instructors:
http://home.sprynet.com/~gkearsley/OItips.htm
top related