![Page 1: 1 L643: Evaluation of Information Systems Week 4: January 28, 2008](https://reader035.vdocuments.us/reader035/viewer/2022062422/56649e695503460f94b657bd/html5/thumbnails/1.jpg)
1
L643: Evaluation of Information Systems
Week 4: January 28, 2008
![Page 2: 1 L643: Evaluation of Information Systems Week 4: January 28, 2008](https://reader035.vdocuments.us/reader035/viewer/2022062422/56649e695503460f94b657bd/html5/thumbnails/2.jpg)
2
Measurement
Measurement is “the assignment of values to outcomes following a set of rules” E.g., Measurement of tastiness
Good Bad
E.g., measurement of size 2.5 inches 1 inch
![Page 3: 1 L643: Evaluation of Information Systems Week 4: January 28, 2008](https://reader035.vdocuments.us/reader035/viewer/2022062422/56649e695503460f94b657bd/html5/thumbnails/3.jpg)
3
Conceptualization (Chambliss & Schutt, 2006)
Define the concept (or purpose) e.g., PDA vs. Laptop in a hospital
Identify variables that correspond to the concept e.g., ????
Determine how we can measure these variables [use available data, construct Qs, make observations, content analysis] e.g., ????
![Page 4: 1 L643: Evaluation of Information Systems Week 4: January 28, 2008](https://reader035.vdocuments.us/reader035/viewer/2022062422/56649e695503460f94b657bd/html5/thumbnails/4.jpg)
4
Scales of Measurement (Salkind, 2007)
Nominal – attributes are named E.g.: nationality (American, Canadian, British)
Ordinal – attributes can be ordered E.g., level of use (Low, High)
Interval – distance is meaningful E.g., temperature in Fahrenheit (11, F, 30 F, 50 F)
Ratio – absolute zero is included E.g., # of clients in 6 months (0, 10, 30)
![Page 5: 1 L643: Evaluation of Information Systems Week 4: January 28, 2008](https://reader035.vdocuments.us/reader035/viewer/2022062422/56649e695503460f94b657bd/html5/thumbnails/5.jpg)
5
Reliability of Measurement
A reliable measurement is one that gives consistent results Test-retest reliability Parallel forms reliability Internal consistency reliability Interrater reliability
![Page 6: 1 L643: Evaluation of Information Systems Week 4: January 28, 2008](https://reader035.vdocuments.us/reader035/viewer/2022062422/56649e695503460f94b657bd/html5/thumbnails/6.jpg)
6
To Increase Reliability
Standardized instructions Adequate sample size Avoid unclear items/vague statement Adjust difficulties Minimize external factors
![Page 7: 1 L643: Evaluation of Information Systems Week 4: January 28, 2008](https://reader035.vdocuments.us/reader035/viewer/2022062422/56649e695503460f94b657bd/html5/thumbnails/7.jpg)
7
Validity of Measurement
Validity = It measures what it is supposed to measure Content validity (by expert) Criterion validity (by a set of existing criteria) Construct validity (by conceptual theory)
![Page 8: 1 L643: Evaluation of Information Systems Week 4: January 28, 2008](https://reader035.vdocuments.us/reader035/viewer/2022062422/56649e695503460f94b657bd/html5/thumbnails/8.jpg)
8
![Page 9: 1 L643: Evaluation of Information Systems Week 4: January 28, 2008](https://reader035.vdocuments.us/reader035/viewer/2022062422/56649e695503460f94b657bd/html5/thumbnails/9.jpg)
9
Reliability vs. Validity
![Page 10: 1 L643: Evaluation of Information Systems Week 4: January 28, 2008](https://reader035.vdocuments.us/reader035/viewer/2022062422/56649e695503460f94b657bd/html5/thumbnails/10.jpg)
10
Population & Sample (Chambliss & Schutt, 2006)
A
B C
Others
A
B C
Others
PopulationSample
![Page 11: 1 L643: Evaluation of Information Systems Week 4: January 28, 2008](https://reader035.vdocuments.us/reader035/viewer/2022062422/56649e695503460f94b657bd/html5/thumbnails/11.jpg)
11
Sample Size
Large
Small
Small Large
Errors
Size of Sample
![Page 12: 1 L643: Evaluation of Information Systems Week 4: January 28, 2008](https://reader035.vdocuments.us/reader035/viewer/2022062422/56649e695503460f94b657bd/html5/thumbnails/12.jpg)
12
Sampling
What is a sample? In a research context, “any group on which
information is obtained” The larger group to which one hopes to apply the
results is called the population E.g., All 700 students at a State University who are
majoring in mathematics, constitutes a population E.g., 50 of those students constitute a sample
Random sampling vs. non random sampling
![Page 13: 1 L643: Evaluation of Information Systems Week 4: January 28, 2008](https://reader035.vdocuments.us/reader035/viewer/2022062422/56649e695503460f94b657bd/html5/thumbnails/13.jpg)
13
Random Sampling
the premise that a sample represents a population is based on the assumption that the sample has been selected at random from the population of interest
![Page 14: 1 L643: Evaluation of Information Systems Week 4: January 28, 2008](https://reader035.vdocuments.us/reader035/viewer/2022062422/56649e695503460f94b657bd/html5/thumbnails/14.jpg)
14
Definition of An Information System (Davis, 1994)
A set of hardware, software, data, procedural, and human components that work together to generate, collect, store, retrieve, process, analyze, and/or distribute information
The purpose of an IS is to get the right information to the right people at the right time
Davis, W. S. (1994). Business Systems Analysis and Design. Belmont, CA: Wadsworth.
![Page 15: 1 L643: Evaluation of Information Systems Week 4: January 28, 2008](https://reader035.vdocuments.us/reader035/viewer/2022062422/56649e695503460f94b657bd/html5/thumbnails/15.jpg)
15
Information System Objectives
Why do we need to consider system objectives?
Each system will be designed to meet certain performance objectives
These objectives affect the design of evaluation: They are the evaluation criteria
![Page 16: 1 L643: Evaluation of Information Systems Week 4: January 28, 2008](https://reader035.vdocuments.us/reader035/viewer/2022062422/56649e695503460f94b657bd/html5/thumbnails/16.jpg)
16
Information System Objectives
Which objectives are mentioned or implied in each of the cases?
John Fluevog Boots & Shoes Zipcar LibraryThing
![Page 17: 1 L643: Evaluation of Information Systems Week 4: January 28, 2008](https://reader035.vdocuments.us/reader035/viewer/2022062422/56649e695503460f94b657bd/html5/thumbnails/17.jpg)
17
A Taxonomy of Information Systems Success
System Quality Measures of the information processing system itself
Information Quality Measures of information system output , e.g.,
accuracy, meaningfulness, and timeliness Information/System Use
Recipient consumption of the output of an information system
![Page 18: 1 L643: Evaluation of Information Systems Week 4: January 28, 2008](https://reader035.vdocuments.us/reader035/viewer/2022062422/56649e695503460f94b657bd/html5/thumbnails/18.jpg)
18
A Taxonomy of Information Systems Success
User Attitudes Satisfaction Recipient response to the use of the output of an
information system
Individual Impact The effect of an information system on the behavior of
the recipient
Organizational Impact The effect of the information product on organizational
performance
![Page 19: 1 L643: Evaluation of Information Systems Week 4: January 28, 2008](https://reader035.vdocuments.us/reader035/viewer/2022062422/56649e695503460f94b657bd/html5/thumbnails/19.jpg)
19
Observations by DeLone & McLean (1992)
There is no consensus on particular measure of information systems success
Simplifying different dependent variable measures would contribute to the MIS research (the results can be compared)
Few field studies to measure organizational performance
6 success categories indicate MIS success is a multidimensional construct & that it should be measured as such (Only 28/100 studied multiple categories)
![Page 20: 1 L643: Evaluation of Information Systems Week 4: January 28, 2008](https://reader035.vdocuments.us/reader035/viewer/2022062422/56649e695503460f94b657bd/html5/thumbnails/20.jpg)
20
Independent vs. Dependent Variables
Examples of variables Survey format—online, paper-based (IV) Response rates (DV)
IndependentVariable(s)
(presumed or possible cause)
affectsDependent Variable(s)
(presumed results)
![Page 21: 1 L643: Evaluation of Information Systems Week 4: January 28, 2008](https://reader035.vdocuments.us/reader035/viewer/2022062422/56649e695503460f94b657bd/html5/thumbnails/21.jpg)
21
Measuring the value of information (Ahituv, 1989)
Ahituv, N. (1989). Assessing the value of information: Problems and approaches. Proceedings of the Tenth International Conference on Information Systems, December 4-6, Boston, 315-325.
RealWorld
Data Informationsystem
Decisionmaker
Decisions,actions
Outcomes
Point of measurementperceived value of
information
Point of measurementrealistic value of
information
![Page 22: 1 L643: Evaluation of Information Systems Week 4: January 28, 2008](https://reader035.vdocuments.us/reader035/viewer/2022062422/56649e695503460f94b657bd/html5/thumbnails/22.jpg)
22
IS Success Model (DeLone & McLean, 1992)
![Page 23: 1 L643: Evaluation of Information Systems Week 4: January 28, 2008](https://reader035.vdocuments.us/reader035/viewer/2022062422/56649e695503460f94b657bd/html5/thumbnails/23.jpg)
23
I/S Success Model, Expanded(based on Hwang, Windsor, & Pryor, 2000)
Systemquality
Informationquality
Use
Usersatisfaction
Individualimpact
OrganizationalimpactInfoSys
Userenvironment
Organizationalenvironment
0.37
0.47
0.37
0.36
0.43
![Page 24: 1 L643: Evaluation of Information Systems Week 4: January 28, 2008](https://reader035.vdocuments.us/reader035/viewer/2022062422/56649e695503460f94b657bd/html5/thumbnails/24.jpg)
24
Updated D&M IS Success Model (2002, 2003)
InformationQuality
System Quality
ServiceQuality
IntentionTo Use
Use
UserSatisfaction
NetBenefits
Creation Use Consequences
![Page 25: 1 L643: Evaluation of Information Systems Week 4: January 28, 2008](https://reader035.vdocuments.us/reader035/viewer/2022062422/56649e695503460f94b657bd/html5/thumbnails/25.jpg)
25
![Page 26: 1 L643: Evaluation of Information Systems Week 4: January 28, 2008](https://reader035.vdocuments.us/reader035/viewer/2022062422/56649e695503460f94b657bd/html5/thumbnails/26.jpg)
26
Evaluation
The man with the tusk states that an elephant is like a spear. The man with the tail argues that the elephant is like a rope. The man with the trunk says, no, its like a snake. The man with the side thinks its more like a wall. But the man with the leg is sure the elephant is like a tree.
The flaw in all their reasoning is that speculating on the WHOLE from too few FACTS can lead to VERY LARGE errors in judgment.
How does this fable relate to System Evaluation?
![Page 27: 1 L643: Evaluation of Information Systems Week 4: January 28, 2008](https://reader035.vdocuments.us/reader035/viewer/2022062422/56649e695503460f94b657bd/html5/thumbnails/27.jpg)
27
![Page 28: 1 L643: Evaluation of Information Systems Week 4: January 28, 2008](https://reader035.vdocuments.us/reader035/viewer/2022062422/56649e695503460f94b657bd/html5/thumbnails/28.jpg)
28
Values in Evaluation (Davidson, 2005)
Suppose that a client does not like the findings of your evaluation and says, “Well, that’s just your opinion about the program. Evaluations are always just so subjective.” How would you respond?
![Page 29: 1 L643: Evaluation of Information Systems Week 4: January 28, 2008](https://reader035.vdocuments.us/reader035/viewer/2022062422/56649e695503460f94b657bd/html5/thumbnails/29.jpg)
29
Class Activities: Evaluation of IUCAT
1. Write down the assigned category on the worksheet. Individually fill out the worksheet #1 - #3 (p. 1)
2. Work with a group of 5 people to fill out the worksheet for the category (p.2)
3. Present your discussions to the whole class
4. Use the worksheet (p.2) to jot down the main points