B. Ross Cosc 4f79 1

Knowledge acquisition (Ch. 17 Durkin)

• knowledge engineering: building expert systems

• knowledge acquisition: process of extracting knowledge from an expert, organizing it, and encoding it into a knowledge base

• knowledge elicitation: extracting knowledge from an expert

• knowledge acquisition is the principle bottleneck in expert system development

• many techniques and theories about how to best do this

• more tools are appearing to help in this

– early example: inductive inference tables

• active research area

– psychologists are especially interested in elicitation issues, as it is a fundamental problem of human psychology

B. Ross Cosc 4f79 2

Knowledge acquisition

Expert

End user

Knowledgeengineer

KNOWLEDGEBASE

Formalizedstructuredknowledge

knowledgeconceptssolutions

data, problems, questions

Prototypes,needs queries

Needs,usability,feedback

Also: other experts, literature

B. Ross Cosc 4f79 3

Some problematic phenomena

1. Paradox of expertise: The more competent a domain expert is, the less able she is to describe the knowledge they use to solve problems.

- studies & experience shows that experts are experts because they compile their vast knowledge into compact, efficiently retrievable form

- as a result, they ignore lots of details about how they derive conclusions --> intuition is prevalent; structured principles are ignored

- for example, experts use lots of generalization and pattern matching to solve standard and new problems

2. Experts make bad knowledge engineers

- domain experts are the worst people for formalizing their own knowledge - non-objective, unfamiliar with AI technology, ...

- need an objective view of knowledge, which isn’t possible from expert

- eg. try to formalize how you go about creating a computer program to solve some problem

B. Ross Cosc 4f79 4

Some problematic phenomena

3. Don't believe everything experts say.

• experts rely on intuition, compiled knowledge • unaware of the deep reasoning; use shallow reasoning

ie. often short-term memory isn’t used;rather, long-term memory as obtained via past experiences is relied upon ---> huge gaps in knowledge

• because experts don't know the formal structure of their knowledge, their descriptions will likely be wrong

- they aren’t used to verbalizing their expertise!

• therefore, knowledge engineer must watch for knowledge that is... - irrelevant, incomplete, incorrect, inconsistent

- knowledge engineer will formalize an expert's knowledge, and then test it to see whether it is logically consistent

B. Ross Cosc 4f79 5

Steps in knowledge acquisition

1. Collect: (elicitation)

- getting the knowledge out of the expert

- most difficult step

- lots of strategies

2. Interpret:

- review collected knowledge, organize, filter

3. Analyze:

- determining types of knowledge, conceptual relationships

- determining appropriate knowledge represention & inference structure

4. Design:

- extracting more knowledge after using above principles

Lets look at these in more detail...

B. Ross Cosc 4f79 6

Tasks of main players

Durkin 17.4

B. Ross Cosc 4f79 7

Preliminary steps

Durkin 17.7

B. Ross Cosc 4f79 8

Interviews and questions

• Interacting with the expert is the primary means of eliciting knowledge

17.9, 17.10

B. Ross Cosc 4f79 9

Interview strategies

• there are different interview techniques; some are suited to different phases of the elicitation process

• Funnel sequencing technique: interview progresses from general, exploratory questions, to detailed questions

Prompts Indirect Beginning of topic ; General

Probes

Direct End of topic ; Concrete

SUMMARIZE INTERVIEW

B. Ross Cosc 4f79 10

1. Unstructured interview

• a spontaneous, natural means to let expert talk freely on anything in domain

• expert verbalizes responses to general questions asked by KE

• stream of consciousness sometimes used

– KE keeps a minimal level of focus on topics discussed

• goal: not to let KE unduly influence early explorations of knowledge

17.14, 17.15

B. Ross Cosc 4f79 11

2. Structured interview

• much more focussed and disciplined than unstructured interview

• KE’s task is to discover concrete information about specific questions

• topic to be explored has been established at earlier sessions

• not as exploratory as unstructured --> better for advanced phases

17.18, 17.19

B. Ross Cosc 4f79 12

To interview or not to interview

• Interviewing is primary means of knowledge elicitation.

• However, there are weaknesses:

– procedural knowledge difficult to verbalize• easier to “do” than to describe• plus some knowledge (physical, artistic) not easily verablized

– ineffective long-term memory• expert just doesn’t remember details of problems• compiled knowledge is difficult to reconstruct

• Case studies: another strategy useful in concert with interviews

B. Ross Cosc 4f79 13

3. Retrospective case study

• ask expert to review and explain a solved case

• expert goes over all the steps, explaining as she or he goes

• KE will record the protocol: the sequence of problem-solving steps or strategies used by expert

• types of case studies:

a) familiar case: a typical “vanilla” case• general info is obtained• best for early phases when foundations are sought

b) unusual case: a new problem hereforeto unseen by expert• good way to get deeper, detailed, more introspective expert

feedback• best for intermediate, later stages

17.22, 17.23

B. Ross Cosc 4f79 14

4. Observational case study

• rather than giving expert the whole case, just supply the problem description

• then watch & record the expert as he or she solves the problem

• stream of consciousness useful

• both familiar and unfamiliar problems can be used

– familiar: more general knowledge obtained

– unfamiliar: detailed, deeper insight into problem solving obtained

17.26, 17.27, 17.30, 17.31

B. Ross Cosc 4f79 15

Summary: strategy effectiveness

17.32, 17.33, 17.34

B. Ross Cosc 4f79 16

Analyzing the knowledge

1. data from expert interview & observation is then transcribed into text form

– important to document all data: date, who, what,...

2. the text is interpreted

– identifying “chunks”: labelling key parts of the knowledge• what portions of knowledge? what are they?

3. Analyzing (“sorting”) the knowledge:

– interrelating the knowledge with previous sessions

– determining it’s representation in domain-friendly notation

– converting it to KB language

– this is done iteratively and incrementally• must pass it by expert for confirmation and corrections

• knowledge dictionary: akin to “data dictionary” in DB systems

– a system document that indexes all terms, rules, etc

B. Ross Cosc 4f79 17

Example transcript (step 1)

17.11

B. Ross Cosc 4f79 18

Interpreted Transcript (step 2)

17.12

B. Ross Cosc 4f79 19

Interpreting transcript

17.36, 17.37

B. Ross Cosc 4f79 20

Knowledge analysis

• Graphical representation of knowledge is an effective means of organizing it

• both KE and expert can understand

– idea is that graphical notations close the “semantic gap” between expert knowledge and formalized form

• Some techniques

– cognitive maps: hierarchical, frame-like graphs

– inference networks/trees: AND-OR tree

– flowcharts: great for procedural knowledge

– decision tree

– example table (from which decision tree, neural net derivable)

• contemporary knowlege engineering tools incorporate graphical denotations of KB

B. Ross Cosc 4f79 21

Graphical representations

17.7, 17.8, 17.9, 17.10

B. Ross Cosc 4f79 22

Conclusion

• research in AI, psychology is forming models of how people & experts organize knowledge, learn, and do problem solving

- these models will give means for determining the best way to extract knowledge from experts, and encode it into a KB

• in the meantime, knowledge engineers (experts themselves) rely on experience for acquiring knowledge and constructing expert systems

- what about: an expert system for creating expert systems?

• KE is quite an interesting and challenging- lucrative profession- active research area