1 supporting sakai: lessons from the university of michigan sean demonner jeff ziegler usability...
TRANSCRIPT
1
Supporting Sakai:Lessons from the University of Michigan
Sean DeMonner
Jeff Ziegler
Usability Support and Evaluation Lab
2Usability, Support & Evaluation Lab, Digital Media Commons, University of Michigan
What we’d like to share with you
• Brief history, scale and scope of the implementation
• Support group structure• Descriptive statistics related to supporting a
large scale Sakai implementation• Support metrics and satisfaction indicators• Common issues, and strategies for addressing
them• Best practices in support and lessons learned
3Usability, Support & Evaluation Lab, Digital Media Commons, University of Michigan
Overview - History
• Sakai in use at U-M for about three years
• Successfully migrated from legacy CMS to “CTools” CLE in Summer 2005
• Currently running Sakai 2.1.2
4Usability, Support & Evaluation Lab, Digital Media Commons, University of Michigan
Overview - Scale
• Total existing sites10,760 Course sites
7,217 Project sites
• Winter 06 Statistics2,806 Course sites in use
44,740 unique users
12,700 daily users (avg)
4,190 concurrent users at peak
5Usability, Support & Evaluation Lab, Digital Media Commons, University of Michigan
Overview - Scope• Email and phone support available during
standard business hours
• Remote email support WeekendsEvenings (when warranted)
• 24x7 system availabilityexcept maintenance windows
6Usability, Support & Evaluation Lab, Digital Media Commons, University of Michigan
Usage Statistics - NetTracker
7Usability, Support & Evaluation Lab, Digital Media Commons, University of Michigan
Daily Usage Pattern
8Usability, Support & Evaluation Lab, Digital Media Commons, University of Michigan
Weekly Usage Pattern
9Usability, Support & Evaluation Lab, Digital Media Commons, University of Michigan
Support Structure
• Two-tiered Tech Support Model1st Tier – 90 hours/wk temporary staff
• Front line technical issues
2nd Tier - 1.5 FTE professional staff• Policy issues• Issues escalated by 1st Tier
• Training and Documentation1.25 FTE professional staff
• .5 Staff Training• .75 Help Documentation
10Usability, Support & Evaluation Lab, Digital Media Commons, University of Michigan
Support Structure
• Quality Assurance30 hours/wk temporary staff
• Local Unit SupportCTools Affiliate program
Department IT staff
Sites computing consultants
11Usability, Support & Evaluation Lab, Digital Media Commons, University of Michigan
User Training
• Ongoing, with a pre-term peak
• Instructor focused
• Group/department presentations preferred
• Low-stakes student introduction
• Coordination with release timing desirable
12Usability, Support & Evaluation Lab, Digital Media Commons, University of Michigan
Documentation
• Do not assume “no one reads the docs”16% of faculty & 7% of students list docs as “most effective way to get help”
• Include docs in localization effort
• Occasional request for manualOnline and task based PDF “chapters”
• Flash visual tutorials are being implemented
13Usability, Support & Evaluation Lab, Digital Media Commons, University of Michigan
Quality Control
• Internal staff tests local instanceRisk based testing
Test integration points with campus systems
Test across campus browser pool
1-2 weeks preferred; semi-formal reports; go/no-go meetings
Cross-test w/Sakai
• Operations Group runs load testingEvolution of protocol
Automated load testing
14Usability, Support & Evaluation Lab, Digital Media Commons, University of Michigan
Tech Support Performance Metrics
• Number of support requestsEmail (91%)
Phone (9%)
• Response Time
• Number of touches
• Satisfaction data from annual survey
• Daily and weekly status reports
15Usability, Support & Evaluation Lab, Digital Media Commons, University of Michigan
Issue Tracking
• FootPrints Electronic queuing mechanism
Feeds metrics gathering and reporting
Issue classification:
16Usability, Support & Evaluation Lab, Digital Media Commons, University of Michigan
Support Requests - Email
LS&A mandate
Fall site creation Winter site
creation
17Usability, Support & Evaluation Lab, Digital Media Commons, University of Michigan
Support Requests - Email W06
18Usability, Support & Evaluation Lab, Digital Media Commons, University of Michigan
Response Time• Service level goal: 2 hours or less
• Business hours constrain this goal
• *On average* response of 15 min. or less
• Peak times see increases in wait times
• Daily “clear the queue” push
19Usability, Support & Evaluation Lab, Digital Media Commons, University of Michigan
Touches Per Ticket
N= 5518; does not include site creations
These numbers should see some improvement due to “up front” requests for user info
20Usability, Support & Evaluation Lab, Digital Media Commons, University of Michigan
2006 Support Survey - Instructors
• 1,357 respondents
• 33% have contacted support
• 18% report email is “most effective way” they get help
• Instructors are “very satisfied” with support services (4.21 out of 5 quality rating)
21Usability, Support & Evaluation Lab, Digital Media Commons, University of Michigan
2006 Support Survey - Students
• 2,485 respondents
• 15% have contacted support
• 36% report “keep trying on my own” is the most effective way to get help
• Students are “moderately satisfied” with Support Services (3.7 out of 5 quality rating)
22Usability, Support & Evaluation Lab, Digital Media Commons, University of Michigan
2006 Support Survey - Comparison
23Usability, Support & Evaluation Lab, Digital Media Commons, University of Michigan
Support Request by Feature
24Usability, Support & Evaluation Lab, Digital Media Commons, University of Michigan
Status Reports
• Daily and weekly communications
25Usability, Support & Evaluation Lab, Digital Media Commons, University of Michigan
Voice of the Customer
• Customer feedback informs organization:
26Usability, Support & Evaluation Lab, Digital Media Commons, University of Michigan
Voice of the Customer
• Policy Issues often come through Support:
27Usability, Support & Evaluation Lab, Digital Media Commons, University of Michigan
Common Issues
• Sites created before registrar data is available
• Maintenance windows
• Large files & upload/download problems
• Site reuse vs. creating new
28Usability, Support & Evaluation Lab, Digital Media Commons, University of Michigan
Common Issues
• Integration with immature or unsupported services/browsers
• Lack of Auditing Tools
• Distributed product delivery (e.g. Identity management, File systems, Registrar data)
29Usability, Support & Evaluation Lab, Digital Media Commons, University of Michigan
Common Issues
• Managing large resource collections
• Managing peak support loads
30Usability, Support & Evaluation Lab, Digital Media Commons, University of Michigan
Lessons Learned
• Training is key
• Low-stakes introduction to tools
• Use a ticket tracking system Multi-tiered support staffing works well
• Hire and train Tier 1 staff early (July for Fall)
31Usability, Support & Evaluation Lab, Digital Media Commons, University of Michigan
Lessons Learned• Distributed support model (Affiliates in units)
• Escalation of Issues / Communication Flow
• Advisory committee(s) / Policy making bodies
• Embrace the workaround
• Establish Support Accounts and use Jira• Combine MOTD with Known Issues
32Usability, Support & Evaluation Lab, Digital Media Commons, University of Michigan
Lessons Learned
• Know browser targets and test local instance• Migration never ends• “Eat your own dogfood” • Changing expectations of online systems
There’s no substitute for talking with customers, attending trainings, etc.
• Support is the ear of the organization and should “have a spot at the table”
33Usability, Support & Evaluation Lab, Digital Media Commons, University of Michigan
Questions?• Sean DeMonner [email protected]
• Jeff Ziegler [email protected]
34Usability, Support & Evaluation Lab, Digital Media Commons, University of Michigan
What else?
• FootPrints demo
• NetTracker demo
• Selecting support staff
35Usability, Support & Evaluation Lab, Digital Media Commons, University of Michigan
Abstract• Sakai has been in use at the University of Michigan for the past 3 years, serving tens
of thousands of customers. Hear the in and outs of supporting a large scale Sakai deployment from the folks who answer the phones and respond to the email.
Description: This presentation will review the experiences of the team supporting Sakai at the University of Michigan, including:
- brief history, scope and scale of the implementation- support group structure & relations with other groups (Development, Operations, QC)- descriptive statistics related to supporting a large scale Sakai implementation- common issues and strategies for addressing them- best practices in support and lessons learned- support metrics and satisfaction indicators
Attendees will come away with quantitative and qualitative information related to supporting a large scale Sakai implementation.