1 l643: evaluation of information systems week 5: february 4, 2008

1

L643: Evaluation of Information Systems

Week 5: February 4, 2008

2

Team Formation for Group Assignments

Will be announced next week

Teammate evaluation will be conducted at the end of the semester

3

Individual Assignment: Measurement Memo

A summary of the selected category A description of an information system & its

objectives (select a different IS for each memo)

A purpose of the evaluation (measurement) What to measure; How to measure Limitation of the proposed measurement References (APA style, see Resources links

on the course website Assignment page)

4


Write no more than 1 page (excluding the references)

Use MS Word default margin, i.e., 1 inch top/bottom & 1.25 both sides

Use no smaller than 10 points Times New Roman

5


Due is at the beginning of the class Assignment samples are available Pick two weeks to write two measurement

memos

6

Evaluating Sources

Be careful with wording The inequality of available resources between the

research and teaching universities is an important issue for the center.

7

Evaluating Sources

Be careful with wording The inequality of available resourcesThe inequality of available resources The

differences in culture between the research and teaching universities is an important issue for the the center center inter-university centers.

8

Evaluating Sources



a more forceful approach

9

Evaluating Sources



a more forcefulforceful explicit approach

10

Writing

The art of revising Take advantage of the Writing Tutorial

Services http://www.indiana.edu/~wts/

Do me a favor Have someone read your writing At least write it on the night before the due date,

read it next day, and revise it

11

Determining Importance (Davidson, 2005)

Dimensional evaluation E.g., DeLone & McLean’s IS Effectiveness Model

Component evaluation E.g., policies, programs (e.g., Teen program in a

library) Holistic evaluation

Personnel, product, service

12

Reporting of Evaluation

I. Executive Summary

II. Preface III. Methodology

1. Background & Context

2. Descriptions& Definitions

3. Consumers

4. Resources

5. Values

6. Process Evaluation

7. OutcomeEvaluation

8 & 9. Comparative Cost-Effectiveness

10. Exportability

11. Overall Significance

12. Recommendations& Explanations

13. Responsibilities 14. Reporting& Follow-up

15. Meta-evaluation

13

Exercise: Survey Instrument

Read the survey instrument for Executive Involvement and Participation in the Management of Information Technology

How many criteria do they use to evaluate executive involvement?

What are strengths & weaknesses of the survey instrument?

How would you improve/modify the survey instrument?

14

The Basics of Research

15

Hypothesis Testing

State Null Hypothesis

Come upWith a ResearchQuestion

DetermineProbability

Retain NullHypothesis

Reject Null Hypothesis

16

Normal Curve Divided into Different Sections (Salkind, 2007)

100908070 110 120 130(Mean)

Raw score

Standarddeviations

-3 -2 -1 0 1 2 3

Standard deviation = 10

17

Distribution of Cases Under the Normal Curve (Salkind, 2007)

100908070 110 120 130(Mean)

Raw score

Standarddeviations

-3 -2 -1 0 1 2 3

34.13%

13.59% 2.15

% 0.13%

34.13%

13.59%2.15

%0.13%

18

Distribution of Cases Under the Normal Curve (Salkind, 2007)

100908070 110 120 130(Mean)

Raw score

Standarddeviations

-3 -2 -1 0 1 2 3

34.13%

13.59% 2.15

% 0.13%

34.13%

13.59%2.15

%0.13%

95.44%

19

Significant Differences (Salkind, 2007)

The research hypothesis specifies the predicted outcome of a study

The null hypothesis most commonly used specifies there is no relationship in the population

E.g., there is no difference between the population mean of users who used a unix-based system and the population mean of users who used a web-based system (= the difference between the means of the two

populations is zero) to process students’ grades

20


Then, the researcher proceeds to test the null hypothesis

Basic assumption is: the sampling distribution is normal

Instead of using the obtained sample value (sample means) as the mean of the sampling distribution, we use zero (i.e., z score)

21


Determine the probability of getting a particular sample value (e.g., an obtained difference between sample means) by seeing where such a value falls on the sampling distribution

If the probability is small, the null hypothesis is rejected providing support for the research hypothesis

The results are said to be statistically significant

22

Different Types of Errors (Salkind, 2007)

Accepted the null hypothesis

Rejected the null hypothesis

The null hypothesis is really true

Accepted the null when no difference between groups

Type I error – rejected the null when no difference between groups

The null hypothesis is really false

Type II error – accepted a false null hypothesis

Rejected the null hypothesis when there are differences

23


What counts as “small”??? What constitutes an unlikely outcome?

It is customary in social science research to view as unlikely any outcome that has a probability of .05 (p=.05) or less

This is referred to as the .05 level of significance

When we reject a null hypothesis at the .05 level, we are saying that the probability of obtaining such as outcome is only 5 times (or less) in 100 (i.e., 5%)

24

Different Types of Stats (Salkind, 2004)

See the chart, Figure 8.1, p. 186.

25

Practicality of the Significance (Salkind, 2007)

Look at the raw scores

Web-interface (75.6) vs. unix-based interface (75.7)

Sample size = 10,000 T-test the difference between the two means is

statistically significant at the .01 level

26

Linear Regression: Y=f(X)

1 2 3 4

1

2

3

4

0Population

Crime rate

27

Path Analysis

Systemquality

Informationquality

Use

Usersatisfaction

Individualimpact

OrganizationalimpactInfoSys

Userenvironment

Organizationalenvironment

0.37

0.47

0.37

0.36

0.43

28

Factor Analysis

Identify the general dimensions represented by a collection of variables E.g., DeLone & McLean (1992) table 7 Cf., Babbie, p. 475

29

One-Way Analysis of Variance (ANOVA)

Determine the extent to which the groups differ from one another based on dependent variables E.g., Figure 16-9 (Babbie, p. 477)

1 l643: evaluation of information systems week 5: february 4, 2008

Documents