![Page 1: L643: Evaluation of Information Systems Week 11: March 17, 2008](https://reader031.vdocuments.us/reader031/viewer/2022032205/56649eb55503460f94bbe34c/html5/thumbnails/1.jpg)
L643: Evaluation of Information Systems
Week 11: March 17, 2008
![Page 2: L643: Evaluation of Information Systems Week 11: March 17, 2008](https://reader031.vdocuments.us/reader031/viewer/2022032205/56649eb55503460f94bbe34c/html5/thumbnails/2.jpg)
2
Measurement Instrument
It is not necessary to use the instruments discussed in class
However, it would be helpful to consider the existing instruments Review the ones we discussed Search the IS World survey instrument database:
http://www.isworld.org/surveyinstruments/surveyinstruments.htm
![Page 3: L643: Evaluation of Information Systems Week 11: March 17, 2008](https://reader031.vdocuments.us/reader031/viewer/2022032205/56649eb55503460f94bbe34c/html5/thumbnails/3.jpg)
3
Pilot Testing of Instruments
Instrument providers: Give instructions to the testers
E.g., their roles, actions, etc Provide your instrument Provide a feedback sheet/ or ask a list of questions
afterwards (debriefing) Testers
Try to answer the questions Ask questions when instructions are not clear Be honest and sincere Provide constructive feedback
![Page 4: L643: Evaluation of Information Systems Week 11: March 17, 2008](https://reader031.vdocuments.us/reader031/viewer/2022032205/56649eb55503460f94bbe34c/html5/thumbnails/4.jpg)
4
Pilot Testing of Instruments
First Pilot Test Session
Second Pilot Test Session
Unicoop (5) Zipcar (5)
Need to send an elec copy to [email protected] and cc’ the msg to [email protected]
Library Thing (3) Evergreen (4)
John Fluevog Boots & Shoes (4)
Quadrem (4)
![Page 5: L643: Evaluation of Information Systems Week 11: March 17, 2008](https://reader031.vdocuments.us/reader031/viewer/2022032205/56649eb55503460f94bbe34c/html5/thumbnails/5.jpg)
5
Announcements
The Evaluation Design Proposal, Measurement Instrument, and Instrument Memo will be due at 5pm on Thursday, March 26th, instead of next Monday
Any group will need a computer lab?
![Page 6: L643: Evaluation of Information Systems Week 11: March 17, 2008](https://reader031.vdocuments.us/reader031/viewer/2022032205/56649eb55503460f94bbe34c/html5/thumbnails/6.jpg)
6
Evaluation Project ProcessProblem/Purpose
Hypotheses or Questions
Definitions
Literature Review
Sample
Instrumentation
Procedures/Design
Data Collection & Analysis
Recommendation
Pilot (test)
See also UCLA’s report
![Page 7: L643: Evaluation of Information Systems Week 11: March 17, 2008](https://reader031.vdocuments.us/reader031/viewer/2022032205/56649eb55503460f94bbe34c/html5/thumbnails/7.jpg)
7
Survey Research (Babbie, 2004)
Choose appropriate question forms Open-ended questions (qualitative) Closed-ended questions (quantitative)
The response categories provided should be exhaustive The answer categories must be mutually exclusive
When would be appropriate to use open-ended Qs? When would be appropriate to use closed-ended
Qs?
![Page 8: L643: Evaluation of Information Systems Week 11: March 17, 2008](https://reader031.vdocuments.us/reader031/viewer/2022032205/56649eb55503460f94bbe34c/html5/thumbnails/8.jpg)
8
Survey Research (Babbie, 2004)
Questions: Avoid asking multiple questions in a single item Qs must be relevant Qs must be concise Try to avoid confusions/ biased items & terms
Respondents: whether they are capable of answering the
questions must be willing to answer
![Page 9: L643: Evaluation of Information Systems Week 11: March 17, 2008](https://reader031.vdocuments.us/reader031/viewer/2022032205/56649eb55503460f94bbe34c/html5/thumbnails/9.jpg)
9
Survey Research (Babbie, 2004)
Format Clear instructions Easy to understand Ordering items Pre-testing
![Page 10: L643: Evaluation of Information Systems Week 11: March 17, 2008](https://reader031.vdocuments.us/reader031/viewer/2022032205/56649eb55503460f94bbe34c/html5/thumbnails/10.jpg)
10
Survey Research
Types of surveys Telephone surveys
http://www.pewinternet.org/reports.asp (Pew Internet & American Life)
Mail questionnaire Low response rates How to increase response rates?
![Page 11: L643: Evaluation of Information Systems Week 11: March 17, 2008](https://reader031.vdocuments.us/reader031/viewer/2022032205/56649eb55503460f94bbe34c/html5/thumbnails/11.jpg)
11
Survey Research (Babbie, 2004)
Administration of survey: Self-administration:
Via snail mail Via Web (e.g., Survey Monkey)
Interview survey Telephone survey
![Page 12: L643: Evaluation of Information Systems Week 11: March 17, 2008](https://reader031.vdocuments.us/reader031/viewer/2022032205/56649eb55503460f94bbe34c/html5/thumbnails/12.jpg)
12
Assessing Program Impact: Alternative Designs (Rossi, et al., 2004)
Bias in estimating program effects Selection bias (nonequivalent comparison design)
Targets drop out Targets refuse to participate
Environmental trends Interfering events, e.g., natural disaster Maturation – people grow
![Page 13: L643: Evaluation of Information Systems Week 11: March 17, 2008](https://reader031.vdocuments.us/reader031/viewer/2022032205/56649eb55503460f94bbe34c/html5/thumbnails/13.jpg)
13
Risks of Quantitative Studies (Nielsen, 2004)
List 3 major problems of quantitative studies Considering these problems, how would you
treat most of the IS research based on quantitative studies? (disregard them or accept as it is?)
Can you think of any situations in which quantitative studies would be useful?
For your group project, what type of research design would be useful? Why?
![Page 14: L643: Evaluation of Information Systems Week 11: March 17, 2008](https://reader031.vdocuments.us/reader031/viewer/2022032205/56649eb55503460f94bbe34c/html5/thumbnails/14.jpg)
14
Quality and Effectiveness in Web-based Customer Support Systems (Negash, et al., 2003)
Information Quality
•Informativeness
•entertainment
System Quality
•Interactivity
•AccessService Quality
•Tangible
•Reliability
•Assurance
•Responsiveness
•Empathy
Effectiveness
•User satisfaction
H1
H2
H3
![Page 15: L643: Evaluation of Information Systems Week 11: March 17, 2008](https://reader031.vdocuments.us/reader031/viewer/2022032205/56649eb55503460f94bbe34c/html5/thumbnails/15.jpg)
15
Quality and Effectiveness in Web-based Customer Support Systems (Negash, et al., 2003)
SERVQUAL: Tangibles: physical features of the system Reliability: the system’s consistency of
performance and dependability Responsiveness: readiness of the system to
provide service Assurance: knowledge & courtesy expressed in
the system Empathy: the care and individualized attention
http://www.12manage.com/methods_zeithaml_servqual.html
![Page 16: L643: Evaluation of Information Systems Week 11: March 17, 2008](https://reader031.vdocuments.us/reader031/viewer/2022032205/56649eb55503460f94bbe34c/html5/thumbnails/16.jpg)
16
Quality and Effectiveness in Web-based Customer Support Systems (Negash, et al., 2003)
Information Quality
•Informativeness
•entertainment
System Quality
•Interactivity
•AccessService Quality
•Tangible
•Reliability
•Assurance
•Responsiveness
•Empathy
Effectiveness
•User satisfaction
H1
H2
H3-.051
.449
.348
![Page 17: L643: Evaluation of Information Systems Week 11: March 17, 2008](https://reader031.vdocuments.us/reader031/viewer/2022032205/56649eb55503460f94bbe34c/html5/thumbnails/17.jpg)
17
Zones of Tolerance (Kettinger & Lee, 2005)
Perceived Service
Adequate Service
Desired Service
Expected Service
SERVQUAL ZOT SERVQUAL
Minimum service level
Desired service level
Perceived service level
Perceived adequacy
Perceived superiority
![Page 18: L643: Evaluation of Information Systems Week 11: March 17, 2008](https://reader031.vdocuments.us/reader031/viewer/2022032205/56649eb55503460f94bbe34c/html5/thumbnails/18.jpg)
18
E-service quality in libraries (Hernon & Calvert, 2005)
Adaptations of SERVQUAL to include e-SERVQUAL instrument
SERVQUAL focuses on 5 dimensions: Assurance Empathy Reliability Responsiveness Tangibles
![Page 19: L643: Evaluation of Information Systems Week 11: March 17, 2008](https://reader031.vdocuments.us/reader031/viewer/2022032205/56649eb55503460f94bbe34c/html5/thumbnails/19.jpg)
19
E-service quality in libraries (Hernon & Calvert, 2005)
LibQUAL focuses on 4 dimensions: Access to information Affect of service Library as place Personal control
![Page 20: L643: Evaluation of Information Systems Week 11: March 17, 2008](https://reader031.vdocuments.us/reader031/viewer/2022032205/56649eb55503460f94bbe34c/html5/thumbnails/20.jpg)
20
E-service quality in libraries (Hernon & Calvert, 2005)
Customization/personalization Importance #4 Unimportance #2
Security/privacy/trust Importance #5 Unimportance #3
Web site aesthetics Importance #10 Unimportance #1
![Page 21: L643: Evaluation of Information Systems Week 11: March 17, 2008](https://reader031.vdocuments.us/reader031/viewer/2022032205/56649eb55503460f94bbe34c/html5/thumbnails/21.jpg)
21
E-service quality in libraries (Hernon & Calvert, 2005)
ideal
perceived
Retain
Improve
Reallocate
resources
Revisit
![Page 22: L643: Evaluation of Information Systems Week 11: March 17, 2008](https://reader031.vdocuments.us/reader031/viewer/2022032205/56649eb55503460f94bbe34c/html5/thumbnails/22.jpg)
22
Choosing Measures to Evaluate … (Ryan, et al., 2001)
Are you measuring something simply because you can? Consider cost & time Just why you are measuring something
Pre-test the measure’s proposed definition Training the relevant staff (see the 5 Qs on p.
123)
![Page 23: L643: Evaluation of Information Systems Week 11: March 17, 2008](https://reader031.vdocuments.us/reader031/viewer/2022032205/56649eb55503460f94bbe34c/html5/thumbnails/23.jpg)
23
Choosing Measures to Evaluate … (Ryan, et al., 2001)
Data collection methodologies: Qualitative techniques:
Case study Content analysis Policy analysis Focus groups Interviews
Quantitative techniques: In-house, mail or electronic surveys
![Page 24: L643: Evaluation of Information Systems Week 11: March 17, 2008](https://reader031.vdocuments.us/reader031/viewer/2022032205/56649eb55503460f94bbe34c/html5/thumbnails/24.jpg)
24
Choosing Measures to Evaluate … (Ryan, et al., 2001)
Data collection methodologies: Automated techniques:
Pop-up surveys Network traffic measures Web log file analysis
Are there confidentiality and privacy concerns?
How can the burden of data collection be balanced ore reduced?