closing some loose ends sources: david w. aha my own thomas h. davenport, laurence prusak, 1998
TRANSCRIPT
Closing Some Loose Ends
Sources:
• David W. Aha• My own• Thomas H. Davenport, Laurence Prusak, 1998
Classifiers
Classifiers
• Case-based reasoning (CBR) classifier
• Induction of decision trees (IDT)
• CBR+IDT classifier
• Others (e.g., covered in the Data Mining course): Support Vector Machines Linear regression Neural networks …
So which one is best?
No Free Lunch Theorem
• Each of these classifiers have a bias
• To explain the bias, let us examine a situation where instances (or cases) are pairs of numeric features and a binary classification problem:
((x,y),class)
• Let us draw the space: CBR, K-d trees (K=2), Support vector machines
• Let us construct examples where each of these classifiers works best
How does the other classifiers work on these examples?
• Formulation of the no free lunch theorem
Knowledge Management
The Beginning: The Apollo 13 Situationhttp://www.youtube.com/watch?v=nEl0NsYn1fU
• The oxygen tanks had originally been designed to run off the 28 volt DC
• The tanks were redesigned to also run off the 65 volt DC
The Changing Game
The New EconomicsManufacturing ServiceTangible IntangibleConsumable InconsumableStructural Intellectual
Tobin’s Q ratio company’s stock market value / value of its physical assets
Is increasing dramatically. What does this mean?
Increasing importance of intellectual capital in the United States (Barr & Magaldi, 1996)
Knowledge Management (KM)
An increasingly important new business movement that promotes the creation, sharing, & leveraging of knowledge
within an organization to maximize business results.
An increasingly important new business movement that promotes the creation, sharing, & leveraging of knowledge
within an organization to maximize business results.
Effective tools to capture, leverage & reuse knowledge
Effective tools to capture, leverage & reuse knowledge
Technology
Develop a culturefor knowledge sharing
Develop a culturefor knowledge sharing
Organizational Dynamics
Needs
Financial constraintsLoss of organizational knowledge
Financial constraintsLoss of organizational knowledge
Needs
Problems:
Knowledge Management: Issues
• Technical and Business Expertise:ProficienciesKnow-HowSkills
• Work Practice Execution:ProcessesMethodologiesPracticesLessons learned
Why Knowledge Management?
• Leverages Core Business Competence
• Accelerates Innovation (Time to Market)
• Improves Cycle Times (Market to Collection)
• Improves Decision Making
• Strengthens Organizational Commitment
• Builds sustainable differentiation
CBR: The Knowledge Management Plunge
“Case-based reasoning programs have been shown to bring about marked improvements in customer service.”
- Thomas H. Davenport, Laurence Prusak, 1998 - Working Knowledge: How Organizations Manage What They Know
“Case-based reasoning programs have been shown to bring about marked improvements in customer service.”
- Thomas H. Davenport, Laurence Prusak, 1998 - Working Knowledge: How Organizations Manage What They Know
KM
CBRWorks
eGain eService Enterprise (E3)
KM Project Domains: CBR Applicable? (KM World, 1/99, Dan Holtshouse, Xerox)
1. Sharing knowledge and best practices2. Instilling responsibility for knowledge sharing3. Capturing and reusing past experiences4. Embedding knowledge (products/services/processes)5. Producing knowledge as a product 6. Driving knowledge generation for innovation7. Mapping networks of experts8. Building/mining customer knowledge bases9. Understanding/mining customer knowledge bases10. Leveraging intellectual assets.
KM Domains/Tasks CBR Applicable?YesNoYesYes Yes
No
YesNoYes
No
1999 AAAI KM/CBR Workshop
~45 attendees: Siemens, Schlumberger, Motorola, NEC, British Airways, General Motors, Boeing, Ford Motor Company, World Bank
~45 attendees: Siemens, Schlumberger, Motorola, NEC, British Airways, General Motors, Boeing, Ford Motor Company, World Bank
Goals:1. Explain KM issues to CBR researchers2. Report on recent CBR approaches for KM tasks3. Share cautions, knowledge, & experiences
Goals:1. Explain KM issues to CBR researchers2. Report on recent CBR approaches for KM tasks3. Share cautions, knowledge, & experiences
Some observations:1. Embedded/integrated in knowledge processes2. Benefits of semi-structured case representations3. Interactive (“conversational”) systems
Some observations:1. Embedded/integrated in knowledge processes2. Benefits of semi-structured case representations3. Interactive (“conversational”) systems
Limitations of CBR for KM(from the 1999 AAAI KM/CBR Workshop)
1. Main limitation is time and effort? (Wess/Haley)1. Main limitation is time and effort? (Wess/Haley)
2. Limitations from working with simple representations (Haley)– Becoming less problematic (e.g., with development of textual CBR)
2. Limitations from working with simple representations (Haley)– Becoming less problematic (e.g., with development of textual CBR)
3. Rule-based integrations– Suffer from old problems of rule acquisition– But KM problem-solving techniques are combating this (Studer)
3. Rule-based integrations– Suffer from old problems of rule acquisition– But KM problem-solving techniques are combating this (Studer)
4. More intuitive case authoring capabilities 4. More intuitive case authoring capabilities
5. Tools for working with heterogeneous data sources 5. Tools for working with heterogeneous data sources
Panel: Lessons & Suggested Directions
CBR Roles:– Accumulate, extend, preserve, distribute, reuse corporate knowledge– Extracting tacit knowledge– Customer relationship management
CBR Roles:– Accumulate, extend, preserve, distribute, reuse corporate knowledge– Extracting tacit knowledge– Customer relationship management
Lessons & Observations:– Integrate CBR with KM tasks & task models– Integrate case retrieval with presentation with tools/workplaces– Integrate case construction/indexing with work product development– Need more advanced (automated) case authoring tools– Must consider effects on user groups, time, organizational impact– CBR not a complete KM solution
Lessons & Observations:– Integrate CBR with KM tasks & task models– Integrate case retrieval with presentation with tools/workplaces– Integrate case construction/indexing with work product development– Need more advanced (automated) case authoring tools– Must consider effects on user groups, time, organizational impact– CBR not a complete KM solution
Experience Management vs CBR
Experience Management
CBR
(Organization)
(IDSS)2. Reuse3. Revise
4. Retain
Case Library
1. RetrieveBackground Knowledge
Experience base
Reuse-related
knowledge
Problem acquisition
Experience evaluation and retrieval
Experience adaptation
Experience presentation
Complex problem solving
Developm
ent and M
anagement
Methodologies
BO
OK
Relating KM with AI
AI
Knowledge-BasedSystems
HumanFactors
KM BusinessProcessing
CBR
Distinguishing KM from Data Mining
KDD Focus:• Large databases• Autonomous pattern recognition
Knowledge Discovery from Databases Process:
Database AcquisitionDatabase Acquisition
Data WarehousingData Warehousing
Data CleansingData Cleansing
Data MiningData Mining
Data MaintenanceData Maintenance
KM Focus:• Capturing organizational dynamics processes• Interaction (i.e., decision support)
Process-Oriented CBR
Most KM tasks are performed in the context of a well-defined (e.g., business) process, and any techniques
designed to support KM must be embedded in this process
Most KM tasks are performed in the context of a well-defined (e.g., business) process, and any techniques
designed to support KM must be embedded in this process
KM examples (many):• Enterprise resource planning (O’Leary)• Project process (Maurer & Holz)
KM examples (many):• Enterprise resource planning (O’Leary)• Project process (Maurer & Holz)
CBR examples (few):•Leake et al.: Feasibility assessment in design process•Moussavi, Shimazu: Cases represent processes•Reddy & Munoz-Avila: Project Planning
CBR examples (few):•Leake et al.: Feasibility assessment in design process•Moussavi, Shimazu: Cases represent processes•Reddy & Munoz-Avila: Project Planning
Motivation for Design Project
• Embedding CBR into existing tools has been shown to be an effective way to insert CBR into KM processes
We saw it this year in a number of projects: Help-desk for LTSRecommender for university eventsCompanies processes
• We discuss two applications They have a similar flavor to most of the design
projects
Two Examples
EXTERNAL MONITORING
AlertsSpiders
Workflow
Scheduling
CollaborationSuspenses
Records Management
Document Management
OA tools
Library catalog
Online databases
E-journals
How-to guides
Document Delivery Service
Bulletin boards
Buckets
Profiles
MIS
INFORMATION SOURCES
WORKSPACE
PERSONAL PORTAL
AFRL Proposed KM Environment
(multi?) impersonal
Personalization
Semantic Web Ontologies
DS1
DS2
DS3
Distributeddata sources
AssistantAgent
Case Repository
Causal ModelCurrent Problem
User Ontologies
Personal Portal/Workspace
InformationSources
NEWS
BULLETINBOARDS
SUSPENSES/
TASKS RESEARCH ASSISTANT
CALENDAR/SCHEDULING
4 5 6 81 2 3 7
WORKSPACE
WHO’SWHO
GUIDES
FAVORITEWEB SITES
Microsoft Word.lnk Micro sof t Pow erPo int.ln k Microsoft Excel.lnk
Individualized Portal
Information Domains
Data Systems
Virtual Library
BucketsFinance
Personnel
A B C D
Executive Information System
Out-of-Family Disposition (OOFD) Process
NASA-Kennedy Space Center: Shuttle Processing Directorate
KM expertise
CBR expertise
Topic: Performing project tasks outside range of expertise• Lack of task familiarity
Motivations: Downsizing, employee loss, technology paceResources: Interim problem reports
• Standardized text documents for reporting problems/solutions• Given: 12 of these reports
Topic: Performing project tasks outside range of expertise• Lack of task familiarity
Motivations: Downsizing, employee loss, technology paceResources: Interim problem reports
• Standardized text documents for reporting problems/solutions• Given: 12 of these reports
Pre-flight, launch, landing, recovery
Prof. I. Becerra-Fernandez
Has this data alreadybeen gathered? If so,
WHERE? NO, needto gather data
NO, need
new mission
ScienceDataNeed
YES, here is the DATA! YES, recommend thisOBSERVATORY!
Science Mission Parameters
Science Mission Assistant and Research Tool (SMART)
Intelligent DataProspector (IDP)
Intelligent ResourceProspector (IRP)
Design Assistant
(IMDA)
Intelligent Mission
Can anexisting resource
obtain the data for me?If so,
WHAT?
I would like toformulate
a new mission…HOW?
Example KM Aplication: SMART KM Portal
SMART: Science Mission Assistant & Research ToolCategorization: An interactive, web-based tool suitePurpose: Reduce time/cost required to define new science initiatives
SMART: Science Mission Assistant & Research ToolCategorization: An interactive, web-based tool suitePurpose: Reduce time/cost required to define new science initiatives
Uncertainty
SMART is Architected as a Web Portal
SMART User
WebBrowser
http://smart.gsfc.nasa.gov
SMART
Intelligent Data Prospector Find data sets
Intelligent Resource Prospector Find an observatory
Intelligent Mission Design Asst Design a science mission
http://smart.gsfc.nasa.gov/irp/
Browse Observatory Knowledge Base Map
TreeObservatory Lists
Search Observatory Knowledge BaseWord/Phrase Search Interactive Dialog
DiscussionsExperts
SMARTIntelligent Resource Prospector
http://smart.gsfc.nasa.gov/imda/
Browse Mission Knowledge Base MapTreeMission Lists
Search Mission Knowledge BaseWord/Phrase Search Interactive Dialog
DiscussionsExperts
Design a Mission
SMARTIntelligent Mission Design Asst
SMARTConcept Map Viewer: Observatories
SMARTHierarchical DirectoryViewer
SMARTDatabase Views
SMARTConversational CBRQuestion/ResponseInterface
SMARTCollaborativeDiscussions Interface
SMART IMDADesign a Mission
Create/Edit a MissionValidate Design
Power Design AdvisorThermal Design AdvisorCommunications
Design Advisor…
InvokeDesign
ValidationAgent
(applet)
(serverDBaccess)
(applet)
(KM toolservice)
(KM toolservice)
(expertsystems)
Searching for Missions Using CCBR
SMARTConversational Mission Search Engine
Describe what you are looking for:
“I’m looking for astronomy missions in low-Earth orbit.”
Ranked questions:Score Answer Name Title
“X-ray” Q17 What portion of the spectrum is observed?60 Q7 What launch vehicle?50 Q32 What mission phase?20 Q23 Low or high inclination orbit?10 Q41 Cryogenically-cooled instrument?
Ranked cases:Score Name Title90 XTE X-Ray Timing Explorer90 AXAF Chandra X-Ray Observatory30 GRO Gamma Ray Observatory30 EUVE Extreme Ultra-Violet Explorer
Question: Q17
Title: What portion of the spectrum is observed?
Description: What portion of the electro-magnetic spectrum are you interested in?
Select your answer: Visible light Infra-red Ultra-violet Microwave X-Ray Radiowave Gamma Ray
Lessons Learned
Keywords: Philippines, evacuation, disaster relief, c2, NEO, Fiery Vigil, etc.Keywords: Philippines, evacuation, disaster relief, c2, NEO, Fiery Vigil, etc.
Observation: Assignment of air traffic controllers to augment host country controllers was critical to safe evacuation airfield operation.
Observation: Assignment of air traffic controllers to augment host country controllers was critical to safe evacuation airfield operation.
Discussion: The rapid build-up of military flight operations…overloaded the civilian host nation controllers. Military controllers maintained 24 hour operations. ...
Discussion: The rapid build-up of military flight operations…overloaded the civilian host nation controllers. Military controllers maintained 24 hour operations. ...
Lesson Learned: Military air traffic controllers are required whenever a civilian airport is transformed into an intensive military operating area for contingency operations.
Lesson Learned: Military air traffic controllers are required whenever a civilian airport is transformed into an intensive military operating area for contingency operations.
Recommended Action: Ensure controllers and liaison teams are part of the evacuation package, and establish early liaison with host nation to coordinate an agreement on operational procedures.
Recommended Action: Ensure controllers and liaison teams are part of the evacuation package, and establish early liaison with host nation to coordinate an agreement on operational procedures.
What
How
When
Joint Unified Lessons Learned System (JULLS)
Database: 908 “scrubbed” lessons from the CINC’s (1991-)– Unclassified subset: 150 lessons (Armed Forces Staff College)
• 33 relate to NEOs
Database: 908 “scrubbed” lessons from the CINC’s (1991-)– Unclassified subset: 150 lessons (Armed Forces Staff College)
• 33 relate to NEOs
Lesson Format: 43 attributes– e.g., ID Number, submitting command, subject, date– Unified Joint Task List number– Content attributes: All in text format
6 Keywords6 Observation6 Discussion6 Lesson learned6 Recommended action
Lesson Format: 43 attributes– e.g., ID Number, submitting command, subject, date– Unified Joint Task List number– Content attributes: All in text format
6 Keywords6 Observation6 Discussion6 Lesson learned6 Recommended action
Some Lessons Learned Centers/SystemsAir Force o Air Force Automated Lessons Learned Capture and Retrieval System o Air Force Center for Knowledge Sharing Lessons Learned o Air Combat Command Center for Lessons Learned o Automated Lessons Learned Collection & Retrieval SystemArmy o Center for Army Lessons Learned (CALL) o SARDA: Contracting Lessons Learned o US Army Europe - Lessons Learned SystemCoast Guard o Coast Guard Universal Lessons LearnedJoint Forces o JCLL: Joint Center for Lessons LearnedMarine Corps o Marine Corps Lessons Learned SystemNavy o NDC: Navy Doctrine Command Lessons Learned System o NAWCAD: Navy Combined Automated Lessons Learned o NAVFAC: Naval Facilities Engineering Command Lessons Learned System Government (non-military) o NASA Lessons Learned Information System o International Safety Lessons Learned Information System o NASA-Goddard: RECALL: Reusable Experience with CBR for Automating Lessons Learned) o NIST: Best Practices Hyperlinks o DoE: US Department of Energy Lessons Learned Other o Canadian Army Lessons Learned Centre o United Nations: UN Lessons Learned in Peacekeeping Operations
Air Force o Air Force Automated Lessons Learned Capture and Retrieval System o Air Force Center for Knowledge Sharing Lessons Learned o Air Combat Command Center for Lessons Learned o Automated Lessons Learned Collection & Retrieval SystemArmy o Center for Army Lessons Learned (CALL) o SARDA: Contracting Lessons Learned o US Army Europe - Lessons Learned SystemCoast Guard o Coast Guard Universal Lessons LearnedJoint Forces o JCLL: Joint Center for Lessons LearnedMarine Corps o Marine Corps Lessons Learned SystemNavy o NDC: Navy Doctrine Command Lessons Learned System o NAWCAD: Navy Combined Automated Lessons Learned o NAVFAC: Naval Facilities Engineering Command Lessons Learned System Government (non-military) o NASA Lessons Learned Information System o International Safety Lessons Learned Information System o NASA-Goddard: RECALL: Reusable Experience with CBR for Automating Lessons Learned) o NIST: Best Practices Hyperlinks o DoE: US Department of Energy Lessons Learned Other o Canadian Army Lessons Learned Centre o United Nations: UN Lessons Learned in Peacekeeping Operations
Lessons Learned Repositories: Functionality
Center forLessons Learned
Center forLessons Learned
Documented Lessons
Decision-SupportTool
Decision-SupportTool
RetrievalTool
Interface
RetrievalTool
Interface
LessonsLearned
Repository
LessonsLearned
Repository
Lessons Learned System
Search queriesRelevantlessons
Lessons Learned Systems: Unrealistic Assumptions
The decision maker
1. has time to search for lessons,
2. knows where to search for lessons,
3. knows how to search for lessons, and
4. knows how to interpret retrieved lessons for their current decision-making context.
The decision maker
1. has time to search for lessons,
2. knows where to search for lessons,
3. knows how to search for lessons, and
4. knows how to interpret retrieved lessons for their current decision-making context.
Decision SupportTool
UserInterface
Active Lessons Learned Repositories
Center forLessons Learned
Center forLessons Learned
Documented Lessons
RetrievalTool
Interface
RetrievalTool
Interface
LessonsLearned
Repository
LessonsLearned
Repository
Lessons Learned System
LL Agent: (CBR)• Relevance
Assessment• Retrieval• Interpretation
Search queriesRelevantlessons
Issues for Active Lessons Learned
Documented Lessons
LL Agent(CBR)UserUser
Case Library
Case extraction
Decision SupportTool
Decision-Making Process
1. Case extraction methods2. Case representation3. Choice of decision support tool4. Embedded LL agent behavior
1. Case extraction methods2. Case representation3. Choice of decision support tool4. Embedded LL agent behavior
Lessons Learned: NEO Critiquing Example
Compose an Intermediate Stage Base
Tasks
Scenario:• 50 miles from ISB #1• 30 miles from ISB #2
• Commercial airfield
Resources:• Transport vehicles
• …• Joint Air Command
• Military air traffic controller• ...
Objects:1. Planning tasks2. Resources3. Assignments4. Task relations5. Scenario
Objects:1. Planning tasks2. Resources3. Assignments4. Task relations5. Scenario
Coordinatewith localsecurity forces
Coordinate withairfield traffic controllers
...
Lesson Learned #13167-92740:• Index: Coordinate w/ traffic controllers• Lesson: If ISB is a commercial airfield,
then assign military air traffic controllers to the evacuation package
Lesson Learned #13167-92740:• Index: Coordinate w/ traffic controllers• Lesson: If ISB is a commercial airfield,
then assign military air traffic controllers to the evacuation package
Transport militaryair traffic controller to ISB
KM/CBR: Possible Future Directions
1. Applications– e-Commerce– Decision support systems
• Personalized– Knowledge discovery for databases?
• Yet KDD stresses need for many automated tasks
1. Applications– e-Commerce– Decision support systems
• Personalized– Knowledge discovery for databases?
• Yet KDD stresses need for many automated tasks
2. Multimodal systems– e.g., Shimazu: Audio tapes of customer dialogues– Information gathering– Learning assistants
2. Multimodal systems– e.g., Shimazu: Audio tapes of customer dialogues– Information gathering– Learning assistants
3. Process-focused emphases:– Retrieval, adaptation, and composition of processes
3. Process-focused emphases:– Retrieval, adaptation, and composition of processes