© Blackboard, Inc. All rights reserved.

System Administration Best Practices & Troubleshooting TechniquesJeremy PortzerTechnical Support ManagerBlackboard Inc.

July 19th 11:30 AM

2

And now a word from our lawyers…

» Any statements in this presentation about future expectations, plans and prospects for Blackboard and other statements containing the words "believes," "anticipates," "plans," "expects," "will," and similar expressions, constitute forward-looking statements within the meaning of The Private Securities Litigation Reform Act of 1995. Actual results may differ materially from those indicated by such forward-looking statements as a result of various important factors, including the factors discussed in the "Risk Factors" section of our most recent 10-K filed with the SEC. In addition, the forward-looking statements included in this press release represent the Company's views as of July 18, 2005. The Company anticipates that subsequent events and developments will cause the Company's views to change. However, while the Company may elect to update these forward-looking statements at some point in the future, the Company specifically disclaims any obligation to do so. These forward-looking statements should not be relied upon as representing the Company's views as of any date subsequent to July 18, 2005. Blackboard, in its sole discretion, may delay or cancel the release of any product or functionality described in this presentation.

3

Objectives

- Provide Blackboard® System Administrators with an overview of their roles, responsibilities, and their impact on the success of the Blackboard Academic Suite™ at their institutions

- Highlight best practices for keeping your Blackboard Academic Suite running smoothly and problem free

- Improve and standardize the troubleshooting process on campus. Methods for better troubleshooting.

- Describe ticket submission and escalation process within Blackboard

4

System Administrator Roles I

» Back-end administratorInstallations; webserver admin; database configuration and maintenance; application monitoring; snapshot and data integration; and even OS maintenance tasks

» Front-end GUI administratorCourse shell creation; user enrollment; export/import/copy; control over feature availability and design; set policies and business rules for account creation, content preservation etc.; portal administration

» System Extension DeveloperCreation of Building Blocks, Application Bridges and Portal Modules using Java and Blackboard’s APIs

5

System Administrator Roles II

» CMS Help Desk ManagerTroubleshooting issues reported via email, phone, or a ticket system, or self-discovered; escalation of issues to Blackboard Technical Support; training and supervision of general help desk staff

» Trainer & Documentation AuthorOne-on-one training and seminars for faculty, staff, and students; creating online documentation and tip sheets

» Instructional DesignerIn some cases the system administrator may also train instructors and staff on pedagogically sound uses of technology, and participate in the creation of course materials and the tracking and evaluation of student performance

6

Show of hands

Which roles do you fill at your institution?

» Back-end administrator» Front-end administrator» System Extension Developer» CMS Helpdesk manager» Trainer / Documentation Author» Instructional Designer

7

User community mailing lists

» Back-end administrator» Front-end administrator» System Extension Developer» CMS Helpdesk manager» Trainer / Documentation

Author» Instructional Designer

» BBADMIN-L» BLKBRD-L» BB-OPEN_SRC

» BB-FS

» BB-ID

8

Evolution of SystemAdministration Roles & Practices

MISSION CRITICAL

EXPLORATORY SUPPORTED STRATEGIC TRANSFORMATIVE:THE NETWORKED

LEARNING ENVIRONMENT

Phase I Phase 2 Phase 3 Phase 4 Phase 5

INS

TIT

UT

ION

AL

GR

OW

TH

Multiple PeopleSingle Server

Single Admin

Integrated Production

Environment

Monitoring, Backup &

Maintenance Practices

Ticketing System

Student /Faculty Training Programs

System Update Deployment Plans

Custom Instructional

Tools

Long Range Planning

Help Desk Management

High Performance /

High Availability

Strategic Owner

Business Continuity

Identified system manager

Help Desk

Change management

practices

9

Best Practices

» Be Prepared» Avoiding is better than solving» Have a well-established troubleshooting process on

campus» Train your users to be aware of pitfalls» Document local policies and differences» Top 10 tips for System Administrators

10

Best practices - Top 10 tips for System Administrators

1. Maintain current Operations Workbook2. Document all server and application changes 3. Maintain a Test server environment4. Set up and test a Disaster Recovery Plan5. Plan ahead for increased usage6. Train users well to avoid problems7. Keep it clean - purge old records8. Manage and monitor your application logs9. Graph system metrics (MRTG, cricket)10.Participate in user community

12

Network diagram(workbook)

• Use Visio templates or simple flowcharting tool

• Make sure to label machines with function, hostname and IP-address

• Indicate ports used on each connection

• Include servers, firewalls, routers, proxies, load-balancers, storage devices

• Keep it up-to-date

13

Disaster Recovery Plan

» Ready adequate replacement hardware» Set up full daily backups to it» Test recovery regularly» Train multiple staff on recovery» Document all necessary steps in standard location

(Workbook, Backup/Recovery guide, local customizations)

» Don’t commit replacement system to other production-grade tasks

14

Sample performance graphs

CPU-busy statsweekly graph

weekly graph fortotal number of logins per dayand number of recent logins(last 10 mins)

© Blackboard, Inc. All rights reserved.

Part II

Troubleshooting

16

Troubleshooting Process on Campus

The SOAP Method - A Basic Troubleshooting Method

Subjective - Who?, What?, Where? and When? Objective - Steps to recreate, Error messages, log collection and

access information Assessment - Review steps, errors, logs and assess whether it can

be resolved or has to be escalated Plan - Resolution steps or Workaround

Examples of SOAP interaction: - Used by the Helpdesk- troubleshooting end-user issues - Used by the System Administrator - recognizing, analyzing, reporting

system-issues, focused collection of info for escalation

17

SOAP at the Helpdesk

» User- or course-specific issue?» if so, need exact steps and location

» Browser-specific?» test on different machines and browsers

» Recent changes?» OS or software update, configuration change, malware

infection» Restore known-good configuration

» system restore, spyware removal, virus scan

Troubleshooting User-Reported Issues

18

Example:Resolving a login problem

» Account exists?» Correct password used? Caps?» If campus-wide pwd, does it work elsewhere?» Correct URL used?» Machine-specific?» Browser configuration (SSL, JS, cache)» Spyware and virus check» Create account, reset password, clean machine

19

SOAP for the GUI admin

» Word-of-mouth vs. direct report» Issue affects multiple users?» Issue repeatable?» Issue known in user community?» Useful info in logs?» Report results of this investigation when

filing support case

20

Example:Disappearing forum posts

» Group forum or course forum?» Who has forum admin rights?» Can authors delete own posts?» whodunnit: check modperl access logs for POST

requests in timeframe in question» Inform users that forums are not personal

mailboxes and read messages should not be deleted

» Restore backup to test machine and send text of lost messages to users for re-posting

21

SOAP for the back-end admin

» Intermittent problem vs. event driven » Issue affects server performance?» Issue repeatable?» Issue tracking in the system logs?» Useful info in server OS logs?» Report results of this investigation when

filing Blackboard support case.

22

Example:Database Connection Unavailable

» Blackboard application not working» Users reporting error messages upon connection» Problem is not browser specific.» Review of Log files

− Server OS, DB Logs and Blackboard log files» Log results

» Blackboard log files revealed ‘full transaction logs’.» Server OS logs reported low drive space on DB server.

» Database Maintenance Plans» System Administrator scheduled Database maintenance

routines, Data integrity checks, Backup and truncation of logs. Updated Blackboard maintenance routines to include nightly and weekly DB backups

23

The Escalation Process within Blackboard Client Support

» Clarifying the process – towards realistic user expectations-sample case lifecycle-sample case interaction between customer and Blackboard support

» How and where can you as the system administrator help to resolve cases quickly and smoothly? - SOAP

24

Sample case lifecycle

» Customer: case creation on Behind the Blackboard®

» TSM: replicates problem» TSM: researches KB, Wiki, tracking system for similar cases

and known resolution» TSM: obtains access info from customer» TSM: creates bug report and escalates case» PSE: verifies problem and attempts resolution» PSE: escalates to Development» Dev, PSE: Analysis of customer impact; prioritization» Dev: Developer assigned; target release version determined» Dev: Issue is replicated; root cause is determined» PSE: communicates between customer and development» Dev: solution is created» Dev: fix goes to QA» Customer: Beta-release testing» Dev: Software updated released

25

Roadblocks in the lifecycleor: why do things take so long?

» Communication failures» Everyone is too busy» Incorrect or incomplete information» Customer impact not clear» Issue is not high priority» Other issues hold up a release

26

Sample case interaction

» C: Our system keeps going down» S: Symptoms please? Logs? Access?» C: Supplies some logs & access info» S: Need recent log and perf-report» S: Access not working, SSH blocked» C: now SSH works» S: escalating» S: perf-report useless, no DB info in it» … you get the picture…

27

Or how it should be…

» C: occasionally modperl crashes with CPU exhaustion» C: logs and perf-report from latest crash, access info» S: this has been reported by other customers – have

you recently updated any building blocks?» C: yes, block XYZ» S: any courses with course menu links to this BBB? If so,

export sample such course, restore to test server, and see whether accessing that course makes modperl CPU usage rise over all bounds

» C: verified, here’s sample course export» S: workaround: remove bad menu items in DB

28

How can YOU help?

» Clear problem description» Precise steps for replication» Preliminary analysis (SOAP)» Supply requested data promptly» Check case status online» Focus on important issues» We are all only human

29

Evolution of SystemAdministration Roles & Practices

MISSION CRITICAL

EXPLORATORY SUPPORTED STRATEGIC TRANSFORMATIVE:THE NETWORKED

LEARNING ENVIRONMENT

Phase I Phase 2 Phase 3 Phase 4 Phase 5

INS

TIT

UT

ION

AL

GR

OW

TH

Multiple PeopleSingle Server

Single Admin

Integrated Production

Environment

Monitoring, Backup &

Maintenance Practices

Ticketing System

Student /Faculty Training Programs

System Update Deployment Plans

Custom Instructional

Tools

Long Range Planning

Help Desk Management

High Performance /

High Availability

Strategic Owner

Business Continuity

Identified system manager

Help Desk

Change management

practices

30

Wrapping up…

» Innovating Together in ‘05:» Use SOAP method for troubleshooting» Preparation is better than problem solving

» Resources Available:» Operations Workbook

http://behind.blackboard.com/b3/sysadminas/refcenter/r61/wkbk.htm» Training Center

http://behind.blackboard.com/b3/sysadminas/training/index.htm− Web Forums

http://behind.blackboard.com/b3/sysadminas/forums/index.htm

» Follow up Contact(s):− Support site http://behind.blackboard.com− Product Support - Domestic 1-888-788-5264, International 1-202-715-6019− jportzer@blackboard.com, psmgr@blackboard.com

» IF YOU ONLY REMEMBER 1 THING:» Know your system well!

31

Thank You!