Comments on “Expert System Life Cycle …”:

(Or what happened to expert systems research?)

Daniel E. O’LearyUniversity of Southern California

©2013

Outline

1. AI/ES– Science– Accounting/Auditing

2. Selected Previous Research on which Gray et al. is based – Rogers / Gartner– O’Leary / Moore

3. Gray et al. – Data– Research Questions– Alternative Interpretation

4. Alternative “Paradigm” / Approach?5. What happened to expert systems?

1-Expert Systems

• Sound almost magical!• Capture expertise … put it in a system– Replace the expert with the system …– Worked really well to help solve scientific

problems• Artificial intelligence has a similar magic flavor

What is an expert system?

• When I talk about an expert system, I will use it to represent systems that typically were built by AIS and auditing researchers Circa 1983-1998– Rule-based “if … then …”– Developed for “consumption” by people, e.g., auditors– People were “eyes and ears” of the system– System would ask people for inputs about the state of the

world and provide recommendations based on those inputs

– There were limitations associated with virtually all of them.

Use was not the same as math, chemistry and geology

• In accounting and business, the interesting issues required human judgment.– The above rule requires assessment of a number of

concepts– Unfortunately, different people make different

assessments given the same situation• Often proposed as models of, e.g., auditor

behavior– As a result, ES found an initial home at the USC Audit

Judgment Conference

Sample Rule – Requires an “environmental assessment”

If (1) Message control software is complete and sufficient, and (2) Recovery measures are adequate, and (3) Adequate documentation is generated to form a complete audit trail Then there is strong suggestive evidence (.8) that controls over data loss are adequate.

Assessments often are inconsistent

2. Gray - Previous Research … O’Leary

• O’Leary (2008, 2009) used Roger’s life cycle models, Gartner’s technology “hype” cycle and other life cycle models to investigate issues associated with research in information systems– Different data (artifacts) are available at different points in a technology’s

life cycle … (data availability)– Different methodologies are used at different points in the life cycle

because the available data differs over time• Design science early, case studies, empirical later, etc.

– Different researchers are likely to research different aspects based on their strengths or interests

– Similar to this paper, I used expert system research as an example and used research to illustrate the occurrence in the literature and how examples fit into the life cycle models

Gartner Hype Cycle

Artifacts differ across the life cycles

5 %

20-30%

0%

DesignScience

Empirical

Cases

Biased Data

(O’Leary 2008)

Gray - Previous Research … Moore

• Moore (2002), a marketing guru, studied the adoption of new products.– Book is interesting, led to development of a marketing

consulting firm.– Moore argues that different adoption groups exist at different

points in product life cycles– Moore argued that there were “chasms” of firms in the life

cycles• Adoption groups are not smoothly distributed across the life cycle

– One view is that marketing discovered Kuhn’s work on the “structure of scientific revolutions”• Technology and technology research as a paradigm shift

Moore’s Contribution

Chasm

3-a-Gray et al. paper

• Searches previous AIS expert system research• Studies characteristics of the resulting database of

AIS ES papers• Considers mapping that research over time into

different life cycles– Rogers, Gartner, O’Leary, Moore

• I particularly liked the data about Ph. D. dissertations.– To me this is an interesting data set– Ph. D. students are a signal of research and innovation

Top #9 Expert Systems Dissertation (Institutions) Dissertation # %Texas A&M University 3 6.82%Georgia State University 2 4.55%Kent State University 2 4.55%Louisiana Tech University 2 4.55%Michigan State University 2 4.55%Oklahoma State University 2 4.55%The University of Nebraska - Lincoln 2 4.55%The University of Texas at Arlington 2 4.55%University of South Carolina 2 4.55%

1983

1984

1985

1986

1987

1988

1989

1990

1991

1992

1993

1994

1995

1996

1997

1998

1999

2000

2001

2002

2003

0

1

2

3

4

5

6

7

1

0

3

1

3 3 3 3

6

4

3 3

5

2 2

1

0 0 0 0

1

Dissertation #

Star

t 198

3

End

1998

Ph. D. Students

What does data tell us about expert system research?

AAA Meeting Data

Author Publication #Baldwin-Morgan, Amelia Annette 4McDuffie, Robert Stephen 4Murphy, David Smith 4Eining, Martha McDonald 4Steinbart, Paul John 4Changchit, Chuleeporn 3Mauldin, Elaine Gay 2Lenard, Mary Jane 2Back, Barbro Christina 2Wensley, Anthony Kevin 2Meservy, Rayman David 2

1998 – 2006 vs (1983-1998)1998 PH. D.

Dissertations Done, AAA Starts?Implication are …?

Year Presentation #1998 21999 22001 32002 12003 12004 12006 1

Contributors

Top Expert Systems Arthors Publications %

1 Carol E. Brown 7 3.00%2 Daniel E. O’Leary 7 3.00%3 Robert H. Michaelsen 6 2.58%4 Alan Sangster 6 2.58%5 Mary Ellen Phillips 5 2.15%6 Mohammad J. Abdolmohammadi 4 1.72%7 Andrew D. Bailey, Jr. 4 1.72%8 Amelia Annette Baldwin-Morgan 4 1.72%9 Martha M. Eining 4 1.72%

10 James V. Hansen 4 1.72%11 Clyde W. Holsapple 4 1.72%12 R. Steve McDuffie 4 1.72%13 David S. Murphy 4 1.72%14 Paul J. Steinbart 4 1.72%

3-b-Gray et al. Research Questions

• RQ1: Did expert systems research in accounting and AIS domain go through a similar industry life cycle over time?

• RQ2: Did the type of research evolve over time as would be predicted by the industry life cycle and the Gartner hype curve?

• RQ3: Did the type of researcher evolve over time as would be predicted by the adoption life cycle?

• RQ4: Did the evolution of the type researcher encounter a chasm that slowed or stopped expert system research as would be predicted by Moore?

RQ1: Did expert systems research in accounting and AIS domain go through a similar industry life cycle over

time?• How are you measuring ES research?

– Does it refer to the research artifacts?– Does this mean the approach laid out by O’Leary: design science,

case studies, empirical …?– How much research is needed to count as “research”? Does one

occurrence count?• I am not sure what “… go through a similar industry life

cycle …” refers to.– What from industry is being measured?– What is being compared here?– Does this mean research is like industry?– How would this be measured?

RQ2: Did the type of research evolve over time as would be predicted by the industry life cycle and the

Gartner hype curve?• I am not sure what this means.• What is “type of research?”

– Research artifacts … taking predictable formats as was illustrated in O’Leary (2008,2009)?

• What does evolution mean here?– Research uses previous research, is that what “evolution” means?

• What predictions?– Research is different than industry– Gartner, Moore, Rogers talk about adoption by companies but

what would they predict about expert system research?• How do we measure/describe research evolution?

RQ3: Did the type of researcher evolve over time as would be predicted by the adoption life cycle?

• What does it mean for individual “type of researcher” to evolve?– Is this the individual researcher?– Does it mean that groups of researchers change or the

composition of the group or their knowledge or …• How do we measure “type of researcher”?– We can see the type of research, but researcher?

• How does adoption life cycle allow us to predict “type of researcher”?– What does it mean to be a “laggard?”– See next slide …

“Laggards would be those researchers who have no interest in the technology per se, but they feel somewhat compelled to do some research in this area or to at least acknowledge this research in their own research portfolio.”

?

Applying all of the life cycle schemes do not make a lot of sense to me.

Would a new Ph. D. student be a “laggard”?Who is in the “late majority?”

This scheme refers to the reaction time, but researchers constantly stream in, e.g., Ph. D. students

RQ4: Did the evolution of the type of researcher encounter a chasm that slowed or stopped expert system research as would be predicted by Moore?

• What does it mean to talk about the “evolution of the type of researcher?”– How do we measure that evolution?

• What is a “chasm” here? When is the start of a chasm or the stop of a chasm?– How can we measure that a “chasm” stopped or slowed expert

systems research?• Doesn’t Moore talk about marketing?

– How would we use Moore to predict here?• Perhaps researchers

– realized that many of the issues had been addressed– realized that there were fatal flaws in expert systems?

What is a chasm here?What does it mean in research terms?

Are these faculty?Do faculty act like firms?

How does a researcher become part of the late majority?Why become a laggard?

Is this reaction time, cluelessness?

The issue is the research question, right? How does that fit?

3-c-Alternative Interpretation of Moore?

• Rather than researchers, perhaps an alternative interpretation is that these are journals.– Those journals are more to less receptive of new or technology research– A chasm would mean no research journals are interested– If no journals go for the new technology, methodology, etc. then

researchers need to make “new” journals, ESR, ISAFM, JIS, IJAIS– Was the "gap” after those journals?

More LessGAP

ESR

JAE?

4. Perhaps we should look somewhere else for guidance …Kuhn’s Paradigms?

• Perhaps we should ask what the expert system “paradigm” is or was? – Was there an AIS ES paradigm?– Was there a competing paradigm? – What was the shared paradigm of expert systems?

• When we went to AIS ES we assumed that we could use the same approach taken with scientific knowledge BUT in discipline laden with non scientific judgments.– We took the expert system paradigm and did not question it– There are “Layers” of paradigms in applied research such as

AIS expert systems, that need to be accommodated

Life cycles are models of moving a theory to become a paradigm

• “Pre-paradigm schools”– Early in the life cycle, “emerging technologies”– E.g., rule-based systems.

• Did we prepare people for the new paradigm?– “Men whose research is based on shared paradigms are

committed to the same rules and standards for scientific practice.”

• “To be accepted as a paradigm, a theory must seem better than its competitors”– Is expert systems a better model of auditor behavior?– What were the competitor paradigms?

Tracking the Expert System Paradigm

• Did older schools begin to disappear?• If it was a new paradigm then there is “paradigm building”

– “ … the successive transition from one paradigm to another via revolution is the usual developmental pattern of mature science.”

• The paradigm affects the practice of the group that practices the field.– Some “cling” to the old views … do we see anyone clinging to the

old views of, e.g., expert systems?– “Those unwilling or unable to accommodate their work to it must

proceed in isolation or attach themselves to some other group.”

When is it a paradigm …?• “In the sciences … the formation of specialized journals, the

foundation of specialists’ societies, and the claim for a special place in the curriculum have usually been associated with a group’s first reception of a single paradigm.”– Sounds descriptive of accounting AIS / ES

• “Paradigms gain their status because they are more successful than their competitors in solving a few problems that the group of practitioners has come to recognize as acute.”– Did AIS / ES help us solve any new problems?

• Anomalies disrupt paradigms … causing chasms of sorts and breaking up paradigms– There was empirical research that served to attack the paradigm

• Kuhn talks of “revolutions” as changes of world views

Types of Research via Kuhn

• Theoretic work comes first, followed by different types of empirical research

• “First is that class of facts that the paradigm has shown to be particularly revealing of the nature of things.”

• “A second usual but smaller class of factual determinations is directed to those facts that, though often without much intrinsic interest, can be compared directly with predictions from the paradigm theory”

• Third, “ … empirical work undertaken to articulate the paradigm theory, resolving some of its residual ambiguities and permitting the solution of problems to which it had previously only drawn attention.”

5. What happened to Expert Systems? (An Evolutionary View)

• Expert systems were replaced by “knowledge-based systems”– Researchers realized that the focus was broader than experts.– Expert systems got a “bad name” – they just did not work the way

we wanted them to.• Other forms of AI became interesting, e.g., CBR, qualitative

reasoning, etc.• The web came along and changed everything …• Then it was recognized that it was still useful to have people

involved so the world got interested in “knowledge management”

• Researchers began investigating “ontologies” that were at the basis of both human and machine systems

What really happened to ES?(My personal research view)

• “One big 4 representative called … (expert systems) a ‘reality mugging,’ meaning what worked in the research lab did not work in the field.”– My own research found that expert systems did not really work,

even in the lab.• In experiments I found that such systems led to inconsistent decisions• Researchers made errors implementing systems and characteristics, e.g.,

uncertainty factors

– I think there must have been a confirmation bias in some of the research

– People were more concerned with building the systems than seeing what could go wrong, if the systems actually worked and when they broke.

What did the “chasm expert” Moore say about “artificial intelligence”

• “Today, however, AI has been relegated to the trash heap. Despite the fact that it was—and is—a very hot technology, and that it garnered strong support from the early adopters, who saw its potential for using computers to aid human decision making, it has simply never caught on as a product for the mainstream market. Why? When it came time for the early majority to absorb it into the mainstream, there were too many obstacles to its adoption …. So AI languished at the entrance to the mainstream, for lack of a sustained marketing effort to lower the barriers to adoption, and after a while it got a reputation as a failed attempt. And as soon as that happened, the term itself became taboo.”

Should we mourn expert systems?

• I do not think so.• There are many problems with expert systems

that do not seem very easy to fix.• More interesting issues are around now– Other technologies have emerged

• I would bet that some expert system technology is embedded in a range of other technologies around us

Can I recommend a paper?

• I did a Google Search on “Why haven’t expert systems lived up to their promise?” – “Auditor Environmental Assessments” IJAIS, 2003

Questions?