text analysis for constructing design representations

ELSEVIER PII: SO954-1810(96)00036-2

Artificial Intelligence in Engineering 11 (19971 65-75

0 1996 Kluwer Academic Publishers. Published by Elsevier Science Ltd. All rights reserved

Printed in Great Britain

0954-1810/97/%17.00

AID 96 Prize Paper

Text analysis for constructing design representations*

Andy Dong & Alice M. Agogino

University of California at Berkeley, Department of Mechanical Engineering, 5136 Etcheverry Hall, Berkeley, California, USA

An emerging model in concurrent product design and manufacturing is the federation of workgroups across traditional functional ‘silos’. Along with the benefits of this concurrency comes the complexity of sharing and accessing design information. The primary challenge in sharing design information across functional workgroups lies in reducing the complex expressions of associations between design elements. Collaborative design systems have addressed this problem from the perspective of formalizing a shared ontology or product model. We share the perspective that the design model and ontology are an expression of the ‘meaning’ of the design and provide a means by which information sharing in design may be achieved. However, in many design cases, formalizing an ontology before the design begins, establishing the knowledge sharing agreements or mapping out the design hierarchy is potentially more expensive than the design itself. This paper introduces a technique for inducing a representation of the design based upon the syntactic patterns contained in the corpus of design documents. The association between the design and the representation for the design is captured by basing the representation on terminological patterns at the design text. In the first stage, we create a ‘dictionary’ of noun-phrases found in the text corpus based upon a measurement of the content carrying power of the phrase. In the second stage, we cluster the words to discover inter-term dependencies and build a Bayesian belief network which describes a conceptual hierarchy specific to the domain of the design. We integrate the design document learning system with an agent-based collaborative design system for fetching design information based on our ‘smart drawings’ paradigm. 0 1996 Kluwer Academic Publishers. Published by Elsevier Science Ltd. All rights reserved.

Key words: natural language processing, information retrieval, product data modeling, computer-aided design.

1 MOTIVATION

The design of complex mechanical systems requires an

intimate understanding of the interactions among the different disciplines and subsystems so that cross- disciplinary tradeoffs can be made. Any change that might have been precipitated explicitly by modifying a requirement or implicitly by observing a failed simula- tion will propagate a chain of interaction between designers, manufacturing engineers, process planning

*Awarded the Prize for the Best Paper at Articial Intelligence in Design 1996. The judges said, ‘We think that the winning paper identifies an important and general problem in artificial intelligence in design. It presents a novel solution to this general problem, together with results which clearly demon- strate its feasibility and the viability of its further significant development and application. It is a well organised and a well

engineers, and sales and marketing professionals. Knowing the role of individual functional and physical design elements and their association with other elements in the overall design helps the product design team ‘understand’ the design from the perspective of other members. A process critical to design decision- making is the elaboration of qualitative information of the interactions of design elements.’

In reality, to ‘know’ the interaction between design elements, designers expend a considerable amount of

written paper and stands as an excellent example in its presentation of the method adopted in the work reported and in its treatment of related work’. The paper was originally published in Art$cial Intelligence in Design, 1996, Kluwer Academic Publishers, Dordrecht, 1996, pp. 21-38, and is printed here with kind permission of Kluwer Academic Publishers.

65

66 A. Dong, A. M. Agogino

effort accessing and absorbing design information. One can characterize this scenario roughly as a three-step process. First, the designer looks for possible related elements such as inter-dependent design functions or physical components. Next, the designer analyzes and interprets the relations between them, relations that might be explicitly stated in mathematical equations and rules, or implied by design standards and ‘best- practices’. Finally, the designer decides which of the associations is plausible in some sense. If there is no reason to reject or defer, then the association is accepted.’ Unfortunately, few computer-aided design (CAD) applications have begun to address the problem of reducing the time designers spend understanding the design, including absorbing design information, keeping up with design changes and reconciling problems or sharing information2 According to Akman et al.3 only systems which embed advanced reasoning capabilities will be able to deal with the complexity arising from the management of large quantities of design data.

The research question then is how best to support this type of assessment, that is, the ‘reasoning capabilities’ the design information management system should possess. A review of the current research suggests the use of structured design representations and expert- derived (design committee-based) ontologies. Empirical evidence, however, has shown that the preferred mode of assessment is reading natural language texts such as memos and design specifications associated with the design model4 rather than perusing structured design models. Further, especially in conceptual design, Baya and Leifer’ show that 93% of the time designers assess information on a non-quantitative level of abstraction. Structured representations and design models have typically been geared towards quantitative reasoning. Both conclusions of protocol studies suggest that information-handling tools should concentrate more on bringing to the attention of designers documents within the contextual locality of information requested rather than the representation itself. The system should improve the recall of pertinent design information by guiding the designer towards information related to or critical for the analysis of the current problem, leaving decision analysis to the designers themselves. If designers spend 80% of the time generating and retrieving their data,s then information tools should offer external memory aids to retrieve that data. Doing so, however, requires dealing with unstructured design documentation.

This research introduces an automated technique to acquire a representation of the design based upon contextual clues in the design documents. By allowing the current context of the design to influence the representation, we eliminate the a priori determination of a structured hierarchy or design language and permit dynamic updating of the design vocabulary. The research was motivated by a desire to take advantage of existing design information to assist in collaborative design. Current CAD tools adequately capture the final

design details such as specifications and analysis results. Still, we need to develop tools that learn the inter- connections between well-documented design elements so that federated workgroups can have access to relevant information without necessarily having to be an expert in each area of the design. The underlying aim of the research then is to discover the terminological patterns in design text as a basis for constructing a meaningful engineering model of the design.

2 PRIOR RESEARCH

2.1 Structured design representations

A design representation is a linguistic structure that forms a basis for the structure of thought in the design process. An ontology is a shared vocabulary among the designers which typically represents a shared understanding of the functionality and behavior of the product. Together, the design representation and corresponding ontology form the kernel of design information systems. The evolving Standard for the Exchange of Product Model Data (STEP)6 highlights the thrust towards product modeling and a common ontology in product models. Product modeling-based systems have been quite successful at setting up complex rules which describe in detail the possible underlying structures of a design7’* at the same time ontology-based systems are trying to define semantic relations for representing stereotypical information and to model the functional and behavioral structures underlying the synthesis of a design.9X10 A design-document learning system similar to the one proposed here is being pursued by Reich et al.” except that the relationships between words are not learned but rather negotiated by the designers.

We agree that an ontology coupled with a structured design representation provides a means for sharing information. The approach presented in this research, however, differs from those approaches focusing on product models using specialized grammars and/or shared ontologies 91’2 in that it derives from full-text design documents or natural language annotations to CAD drawings. We distinguish, then, between structured design representations such as ontology-based or product modeling-based representations and unstructured design representations, the richest of which is natural language design text.

2.2 Unstructured design representations

Information models should capture and represent product information to give the reader an ‘understanding’ of the design the model represents. But, they must also be dynamic to reflect the evolutionary nature of design. Even though one could argue that the addition of new ontology and negotiated agreements

Text analysis for design representations 61

makes the ontology-based or product modeling-based systems dynamic with the design, since the ‘meaning’ of design elements changes with an evolving design, modifying the model or adding new ontology to reflect the changes in real-time might be difficult. In fact, the evolutionary and uncertain nature of design require representations that operate on meaning, not expression.13

Part of the problem of structured representations is that they assume that the ‘meaning’ of a design could be computed as a function of the constituents. To ‘understand’ a design, designers must take advantage of a variety of mechanisms that use all sorts of knowledge to fill in any necessary information. In making a computer model of design knowledge, this presents a serious problem. On the one hand, it is impossible to isolate all aspects of domain-dependent knowledge from the others. On the other hand, it is clearly undesirable to give the program all the knowledge related to the design. In this research, the dilemma is resolved by inferring plausible conclusions by relating the various elements of the design using a sufficiently complete and current set of design documents as an accurate representation of the current state of the design.

However, using natural language design text as a model of the design alone would be inefficient from the perspective of reasoning, computation, search and retrieval. Efficiency demands that some structure (representation) be imposed on this full-text model of design in a form that suits the type of reasoning operations described in section 1. This research presents a methodology for making sense of unstructured design representations by inducing a structured representation which supports the process of forming associations between the various design elements and assisting designers in retrieving original design information within the contextual locality of their information needs. We propose the architecture of an intelligent agent in a collaborative design environment which dynamically learns the current status of the design. One application of the agent is the retrieval of relevant information to the current needs of the designer. The system achieves the learning and understanding of the design using the design documents as the ‘model of the world’. We present a theory of design discourse as a theoretical premise for generating a model of the design based on the design documents, and illustrate how to integrate learning the design within a collaborative design framework for bringing relevant design information to the decision-maker based on the ‘smart drawings’ system presented in a prior paper.14

3 METHODOLOGY

3.1 General theory

In discourse, people take advantage of a variety of mechanisms that depend upon the existence of an

intelligent hearer who will use all sorts of knowledge to fill in any necessary information.i5 To make an intelligent agent understand the design as communicated by the designers through design documents, then, we must construct a framework within which the agent has a sufficient search space to formulate an adequate understanding of the design. In order for the agent to fill in necessary information regarding the design, though, it must learn the connections between the functions or components of the design. Currently, the solution strategy9’16 is to have experts construct both the ontology and describe the decomposition of the design to the agent. However, we argue that this information is in fact available and contained in the design documents themselves. Research in full-text retrieval systemsI verify how certain syntactic patterns in documents refer to meaningful concepts, and how language-oriented techniques for information retrieval can build the relationships between categories, category instances and relations of those concepts. These categories can define a model of the design.

In building our algorithm, we assume to a first- approximation that the linguistic content (words) of the design documents provide a useful index to the composition and structure of key design concepts at the current state of the design. Second, it is assumed that every statistical association derives from causal interaction; therefore logical coherence is based on statistica! coherence. Based upon these assumptions, we propose the following theory of design discourse as the theoretical foundation for the learning algorithm:

A theory of design discourse. The content of design documents is related to a conceptual structure of the design, whose communication comprises the goal of the designer.

The claim is the agent can induce a model of the design, including the functions and components of the design and their relations by learning over the design text associated with the product. Eastman et al.” identify several criteria for describing engineering product models: (1) the semantics, which describe the functions, components and attributes of the design; and (2) class structures, which describe both the general- izations (properties relevant to any design element), and the decomposition (how functions and components are interrelated) of the design.

While the product model derived by our algorithm is not the same as that proposed by Eastman et al., these criteria serve as a guideline for learning the design model. The sequence of operations in the program are (as shown in Fig. 1) to: (1) extract the natural language text annotations to CAD drawings to excerpt the semantics of the design; (2) generate the class structures describing which properties are relevant to any function of a design using clustering; and (3) build a decomposition of the design which in this method is accomplished with belief networks. The clusters and belief network


- Annotate Drawings

-----+I NLP/IR Parse

Index

Term Relevance Scoring

1 Machine Learning

Conceptual Clustering

4 I

I Belief Network I

Fig. 1. Process flow chart - the figure outlines the sequence of operations of the program in learning the content and structure of the design. The research proposes a methodology for annotating CAD documents to create ‘smart drawings’, techniques for extracting the design vocabulary from the design text using natural language processing and information retrieval, and model learning and inference by applying

machine learning.

offer external memory aids to alert designers to common

elements (clusters) and the interaction of various design

elements (belief net).‘”

3.2 Stage 1: text analysis

Our general method to discover terminological patterns in the design documents, which act as a basis for

constructing the design model, is to parse the document text, cluster inter-term dependencies and build a conceptual hierarchy.

First, the text is passed through a parser and indexer, freeWAIS-sf2’t w&index, which returns a dictionary of every word in the text except for common ‘stop words . 3$

We then filter this set of terms to develop a set of

content-carrying terms. The filtering process is based upon a word score metric similar to that described by the CLARIT method.2’ The scoring equation is based

on the freeWAIS-sf term relevance score (TRS) metric shown in eqn (1). The primary statistics include (1) a

frequency count of the number of times the word was encountered in individual documents in the corpus, and (2) an inverted weighted distribution measurement for the number of documents containing a particular term. The idea is that the frequency measurement correlates with the text semantics. Words that occur often in a text

are better indicators of what the text is about. More terms can always describe the document concepts better,

t One advantage of using a WAIS (wide area information server) program such as freeWAIS-sf for full-text document parsing, indexing and retrieval is that documents can be queried and retrieved over the Internet using the 239.50 V2 protocol. $ Stop words include conjunctions and articles such as ‘a’, ‘the’, ‘since’ and other words frequently used in natural language to connect terms but not necessarily to distinguish topics or provide contextual cues for topics.

but too many terms dilute the importance of any

individual concept. Thus the distribution (or inverted document frequency) of the terms in the documents

captures the intuition that words which have high frequency across documents are ‘general’ in the domain

and do not serve as good discriminators of concepts.

TRS = (log(q) + 10) x idf

number_qf_terms_in_a_document

(1)

0.5 x c word

tf = 0.5 + dot max c word

~klloc

idf = 1

c word ctm

In eqn (1) - the freeWAIS-sf TRS metric - the TRS

metric is based upon the term frequency {f which counts

the number of times the word appears over all documents, the inverted document frequency idf which counts the number of documents containing the word (a

measurement of distribution) and normalized by the number of terms in a document, to account for the rarity of a word.

The score does not account for variations in author style or the presentation of the text. For example, one might score words which are typed in bold face or italicized or words from more recent design documents higher than others. Other factors such as the person who

wrote the document, paragraph headers or document titles could be used as additional word weights; however, the efficacy and numerical value of these weights is difficult to quantify. Further, this complicates the clustering. For example, ‘recent’ terms might be associated by time rather than meaning which violates the purpose of the algorithm. Thus the algorithm has

limited sensitivity to the organization and presentation of the text.

Then, the program computes the average score and

standard deviation. Words whose score falls above the mean become the inventory of index terms for the

corpus, the certified terminology. The system filters out words which occur relatively frequently, have less value in forming good topic discriminators than relatively rare words, and words which are seldom used since they are

probably not conditionally dependent upon the concepts described and vice versa. We will explain later why this conditional relevance is important in building a dependency matrix of concepts which forms the basis of the

representation. Finally, based on the set of certified, content-carrying

terms, the system determines their contextual similarity by measuring the frequency of co-occurrence of any two of the certified terms in the documents. That is, the program generates a n x n matrix, where n is the number of certified words. which scores how ‘often’ the certified words co-occur. This matrix is created by executing a ~rai.syucr~ consisting of the query string ‘[word-A] AND


[word-B]‘. The query sums up the score for similarity between the query string and the document base. The conjecture is that if the query string appears frequently over the entire document base then the words have a shared contextual dependency. In freeWAIS-sf, document similarity is measured as a vector product formula. The similarity between the query string Q and the document D is given by

similarity(Q, D) = x(w+ x wdk) (2) k

where w+ is the weight assigned to term k in the query and w& is the weight assigned to term k in the document D.

3.3 Stage 2: clustering and inducing a bayesian network

Once the system has developed a prescribed vocabulary, the program maps the terms into context descriptors. The words themselves have no ‘meaning’ outside the context in which they appear. Thus, we need to map the relevance between the assigned terms and the context in which they appear.

For this process, we apply two machine learning techniques. In the first portion, we classify related terms into ‘conceptual cells’ using unsupervised learning. These cells represent terms which are self-similar in the documents. This determination is based upon the observation that terms which appear together (in the same context) in documents typically connote similar meaning. ‘7.22 Since the matrix measures closeness based on the spread of data or distance between words, a convenient distance-based clustering technique is the K-means algorithm shown in Table 1.23 The variable x, is the score in the matrix for the pairwise occurrence of two words in the document collection.

Next, the goal is to obtain a decomposition that explicitly reveals as much information regarding the conditional independence of design elements as possible. The key feature of belief networks is their explicit representation of the conditional independence among events.24 That is, they can explicitly and compactly represent the dependency of design elements. In the

network, the possible states on each node is binary, 1 if the element exists in the current analysis, 0 otherwise. Topological transformations (through arc reversals and node absorption for example) can answer questions concerning possible causal relations or dependencies between design elements. Since the Bayesian network conveys an intuitive understanding of how the reasoning process works, the designer can also follow the reasoning process of the design based upon the dependencies/ independencies of the events to determine how the change in any one element might affect any other element.

The general method for constructing belief networks is to draw arcs from causal nodes to effect nodes and then attach a probability to that arc.25 While techniques exist for constructing the most probable belief network B, given a database D of instances (often called the maximum a posteriori structure) based on assumptions of a uniform distribution of belief network structures,26 the Bayesian Dirichlet likelihood equivalent metric,27 and minimum description length (MDL),28 the problem remains NP-complete and the definition of a ‘good’ belief network, i.e. one that adequately captures the underlying causal model of the physical system yet supports efficient reasoning, remains undecideable. We generate an initial network using a heuristic approach. We plan to apply one of the metrics to optimize the network locally about a network structure which correctly represents the design.

The heuristic used to construct the Bayesian network is based upon the conjecture that seeing a lower TRS word with respect to a word that it shares contextual similarity causes the system to update the belief that the higher TRS word will appear.21 This causal influence and contextual similarity is found by pairing words with the highest TRS in the co-occurrence matrix. The strategy for building the network is to link the highest associated words in their own clusters first then to link the words between clusters. The algorithm is outlined in Table 2.

3.4 Agent architecture for design information retrieval

Figure 2 depicts the agent architecture for learning the

Table 1. K means algorithm

procedure K-MEANS (Initialize the cluster centers wi, j = 1,2, . , N,) (repeat

; Group the patterns with the closest cluster center (for all xi do

(Assign x, to Q,*, where wi* = m,in 11 X, - W, 11 endloop)

; Compute the sample means (for all wj do

M’j _+&xi J x, tj

endloop) until there is no change in cluster assignments from one iteration to the next)

end ; {K-MEANS)


Table 2. Network algorithm - in the first part, the table illustrates the general method for creating belief networks based on expert knowledge. In the bottom second, the table outlines the heuristic algorithm employed by the program

General procedure

(1) Choose the set of relevant variables Xi that describe the domain (2) Choose an ordering on the variables (3) While there are variables left:

(a) Pick a variable Xi and add a node to the network for it (b) Set parents (Xi) to some minimal set of nodes already in the net such that the conditional independence property is satisfied

(direct causal influence) (c) Define the conditional probability table for X,

(1)Define a variable Xi for each word Network algorithm

(2) Order the variables Xi in their respective clusters by ascending TRS (3) While there are variables left in the cluster

(a) Select the variable Xi with the lowest TRS and add a node to the network for it (b) Set 4 as parent of (Xi) where Xi and X, have the highest similarity in the co-occurrence matrix and TRS(q) > TRS(X,) (c) Select next node in ordering as Xi+, and continue; repeat for each cluster

(4) Order the clusters by ascending cumulative TRS (5) While there are variables left in the cluster

(a) Select a variable Xi from the lowest TRS cluster (b) Set Xi as parent of (Xi) as the node from the next cluster with the highest similarity in the co-occurrence matrix (c) Select next node in ordering as Xi+, and continue; repeat for each node and cluster

(6) Define the conditional probability table for X,

ICAD Drawings 1

I Cluster Belief Network I

+ Document Parser

Fig. 2. Agent architecture - the user annotates and adds design documents to the document database. The agent interacts with the document database by parsing and scoring the words in the document. The agent uses the data to create the clustering and belief network to learn the connections between the design elements. The user can then ask for relevant information with respect to current

information needs by having the agent search for related design components.

design based on the documents. The architecture augments the ‘smart drawings’ system presented in a previous paper.14

The agent environment consists of the database of

design documents, including the CAD drawings, design specifications, design notes and memos and e-mails written between designers. The agent reads the text periodically to generate the list of content-carrying words. By manipulating the list and using the document database for additional data, the agent constructs the inter-term clusters and belief networks to build a model of the design. The model helps the agent to understand the design by finding out what properties are relevant to a function in the design and the decomposition of the design. In response to requests from the user, the agent can retrieve relevant design information.

4 EXPERIMENTATION AND RESULTS

4.1 Experimentation

In order to test our algorithm on a corpus of design documentation, we created a machine-rea

Y able form of

The Mechanical Engineers’ Handbook,29 which was

scanned and run through an optical character recogni- tion (OCR) software (‘dirty’) $ to output the final text.

tAlthough the ultimate test of our algorithm would be a set of design documentation from a complex, industrial product development project, the advantage of using the handbook is that other researchers can compare algorithms against the pm! database.

‘Dirty’ OCR refers to documents un-modified after the OCR process, i.e. no spell check.


((system design controller control) (transfer function time error state signal response plant) (zero integral gain order damping performance steady action) (stable root frequency process model loop) (valve pressure power pneumatic motor displacement) (variable value position feedback input output) (disturbance diagram constant) . . . )

Fig. 3. Cluster results - the cluster results for the chapters on controller design.

The program was then run on the chapters on controller design to derive a model of controllers.

The cluster results are shown in Fig. 3. These clusters indicate which properties are relevant to any particular function or element of the design, giving the agent knowledge of relevant issues in the design. The clusters indicate, for example, that the main content of the documents is the design of a controller or the control of a system. The third cluster reveals that the performance of the system is influenced by the gain and order of the control as well as any damping in the system while the sixth cluster indicates that the position seems to be the variable to be controlled in the system as it is tightly related to the feedback, input and output. One critique of the clusters is that zero appears with performance and root appears with stable, whereas it is known that both the zero and root of the system affect the stability. However, in the document collection, zero statistically appears more often with performance and root with stable since, for example, the documents discuss more often that a zero affects steady-state error (a measurement of controller performance) whereas the closed-loop

roots determine the stability of the system. The cluster results agree with known knowledge of the relevant properties of the functions and attributes of controllers.

Finally, the system generates the belief network shown in Fig. 4 and the conditional probability table associated with the network. The states for each of the event nodes (words) are 0, when the word (or design element) is not present in the document, and 1 otherwise. The conditional probability table for the network is based on frequency counts. For example, the probability of the word controller co-occurring with the word design is given by:

P(controller/design) =

No. occurrences controller and design

No. occurrences design

For readability, not all nodes and their associated arcs are shown in Fig. 4. For the nodes shown, the arcs are complete. One can read some interesting inferences off of the network.

The first inference expresses the dependency of the

design elements. The expression of dependency describes the decomposition of the design.

(1) The system to be controlled is characterized by the desired response and the controller design. The control law is conditioned on the transfer function, the error and the desired response of the system.

The second inference illustrates the degree of dependency between design elements. These types of inferences relate both information and the degree of relevance based on the amount of evidence available.

(2) The concept of system response is more dependent upon gain in this controller design than the specific input criteria.

The third is perhaps the most interesting since it shows

Fig. 4. Initial belief network (partial) - this figure illustrates parts of the belief network generated using the heuristic algorithm. For readability of the figure, not all arcs and nodes

are shown.


,___._ ‘.,_.._._. _ _ _ _ _ re obtalned, and all of the linear system analysis techniques can be applied ts

predict the system's performnce. If the performance is unsatisfactory. a new control law is tried and the P~OCBSS repeated. Mien this process falls to achieve an acceptable design. awe systematic Method of altering the systew's structure am needed; they are discussed in later sections.

We have used step functions as the test signals because they are the cost co~~"on and perhaps represent the severest test of system performance. Impulse. rap. and sinusoida test signals are also ewloyed. The type to use should be made clear in the design specifications.

40.7 CCd0ROLLER HI\ROHARE

The control la nust be iaplenanted by a physical device before the control ngineer's task is complete. he earllest devices were purely kinematic and were mechanical elements such as ears, levers, and iaphra IS that usually obtalned their power fror the controlled vanable. ncist ontrol 9 ers nor are nalog electronic, hydraulic. pneumaf~~. or digital electronic devices. He now

consider the analog electronic type. Digital control is taken up at the end of the chapter.

CoMnand: _MDL_QUERY 40.7.1 Feedback Compensation and ~antroller Design

Command: Cl

_MDL_QUERY Search concluded Mast controllers that irplement versions of the PID algorithn are based on the

Fig. 5. Smart drawings desktop - the agent learns the content of the design data based on the design documents using the learning methodology outlined above. Then, when prompted, the agent can retrieve relevant design information based on the current

information needs of the designer using the information content of the active document as the query.

how the system could infer the interaction of several

elements in the design which produce a certain function. Therefore, if the designer were interested in increasing

the pressure in the controller, one of the design elements to modify is the motor and followed by checking if the valve could handle the increased stress. More notable is

that without explicitly telling the system these design element connections or the design topic, the system correctly extracted from the text that these chapters

discussed controller design using pneumatic devices.

(3) The motor changes the displacement of the valve which affects the pressure.

While the arc directions could change through topological transformations, the above network and associated inferences illustrate two important ideas. First, inspection of the network indicates that the heuristic generates a network with arcs between elements in the direction of perceived engineering causality. As illustrated by the third

example, the motor causes displacement rather than vice

versa. Second, the network illustrates the more important

problem of capturing the dependencies between design elements. By capturing these dependencies, the system is more efficient in searching for meaningful and relevant design information. The combination of the cluster information and the belief network augments the search by finding closely associated design elements (cluster information) which may not actually appear in the designer’s query while removing less relevant information

if less evidence supports the association between the design elements (belief network).

4.2 Validation

The program was integrated with a ‘smart drawing’ system14 as shown in Fig. 5. Preliminary tests were conducted to evaluate the effectiveness of the learned model in improving the recall of pertinent design information. Using the terminology from the clusters and belief net learned over relevant sections of handbook,

Text analysis .for design representations 73

Table 3. Recall improvement validation (percent recall)

Query freeWAIS-sf Augmented Percent increase

1 40 80 100 2 17 33 94 3 14 71 407 4 33 66 100 5 83 83 0 6 7 33 33 0 8 50 50 0 9 60 60 0

10 20 60 300

graduate students in mechanical engineering were asked to form queries and select the relevant documents from

the handbook. To circumvent coding bias, the relevancy judgements were performed on every document in the

target corpus relative to the entire range of initial test queries and independent of the actual test. The query-

document relevance assessments were then applied in an algorithmic manner to the results of freeWAIS-sf and

our algorithm. Table 3 shows the recall results for all the queries, comparing ‘raw’ freeWAIS-sf to augmented

queries with terms added from direct parents and

children of nodes in the belief net. We apply the standard information retrieval definition of recall as the percentage of documents retrieved that are a subset of

the pre-determined relevant documents. A document from the return set is deemed ‘relevant’ if it had a similarity score of 500 or higher. One of the tests asked the system to retrieve relevant information to the ‘step response of the transfer function. t Based on the belief

network results, the program should know that the

response of the system is characterized by the gain and is measured in units of time. By automatically expanding the query to include closely related terms which in this example indicate closely related attributes to the step response of the system, the program can find documents related to the original information request. That is, the

dynamically learned design structure augmented retrieval to include information not explicitly cast in the query but which should be reported together by virtue of

design dependency. Query number 6 introduced a request for which there

did not exist a match in the document set to ensure proper operation of the freeWAIS-sf software. The system experienced difficulties in improving recall for queries 5 and 7-9 because of a combination of factors; almost all relevant documents were located without

query expansion and there existed few relevant docu-

tOne esthetic limitation of the current implementation is that the user is given only the path to the document rather than the document title, for example. By selecting one of the documents, though, the system automatically brings up a viewer for the document type, such as a text file or AutoCAD drawing.

ments in the database. Therefore, we did not expect the

query expansion to markedly improve recall. However,

these preliminary results indicate the utility of a learned

representation in marked improvement of the recall of historical design documentation, such as for case-based

reasoning.i3

4.3 Discussion

The role of the clusters and belief network for design

information retrieval is similar to the purpose of the

decision dependency network presented by Garcia et aL3’ in the explanation interface to their active design

document (ADD) system. The explanation interface displays related information by retrieving documents

that are generally reported together. The key differentia- tion is that the dependency network is based on a pre-

processed parametric design model for the design domain which seems to violate their thesis that the

evolution of the design description via documents relates to the evolution of the design. For example, to capture

design rationale, ADD prompts the designer for decisions which deviate from the preferred norm. This

strategy for design rationale capture suggests that changes in the design affect how the design should have

been modeled or parametrized, that, in fact, the design model dynamically evolves with the design.

Systems such as ADD and the one proposed which address the problem of accessing design information by employing a structured design model to augment the retrieval of unstructured design documents can improve recall over those which have only an unstructured model

(such as freeWAIS-sf) or only structure (ontology-based

systems). However, the important metrics for evaluating these systems should include both the overhead for creating the structured model to account for the dynamic nature of the design as well as the performance in retrieving relevant information compared to baseline

systems which employ no structure. The design learning

methodology proposed illustrates a preliminary system which addresses both metrics, and results were shown to validate the recall claim. The critical test for any such

system, though, is to compare the learned representation to a ‘gold standard’ such as an expert-generated representation. A future paper will discuss this issue in-depth.

While this is only a preliminary test of how well the

system learns the design data, what these tests suggest is the ability to augment design information search by finding related information based upon meaning, not just

how the search request is expressed in the query. This type of query would be of particular advantage for case-

based reasoning systems. Second, the clustering and belief network open the possibility of organizing the retrieved data in a manner which is more meaningful to the designer than just straight frequency metrics, such as ordering by related concepts.

74 A. Dong, A

5 SUMMARY AND FUTURE DIRECTIONS

This research develops a computable learning method to extract the content of the design model to facilitate information sharing among designers. The premise of the methodology is that the design specifications and solutions as communicated through design documents are related to a model of the design. Certain combinations of the chosen properties of the design give rise to the corresponding combinations of design descriptions in the design text. Therefore, by learning these descriptions (words) through text analysis, the system induces a model of the design. The learning algorithm is based upon natural language processing text analysis to extract content-carrying terms, and then applying techniques from machine learning to cluster inter-dependent terms and decompose the design into dependent elements using belief networks.

The model derived for the controller design example was plausible and correct based upon knowledge of controllers. Further, initial tests showed that augment- ing queries over historical design documentation using the learned representation generally improved the recall of relevant information. The learned representation contextualized the search query by adding terms describing related design concepts. The contextualization directed the search engine towards a larger subset of relevant information than might not have been located without its assistance.

What this research emphasizes is that CAD systems cannot ignore the communication of design information with respect to the current and relevant information needs of the design based on the annotation of the drawings. ” That is, the effect of techniques which implement inductive learning techniques such as the one proposed to generate new knowledge structures about the design rather than techniques that improve the efficiency of problem-solving (explanation-based learning techniques) is tantamount to improving CAD systems. By putting the knowledge of design components in a form in which we can explicitly express the connections between the different parts of the system’s knowledge, we enrich the possibility of interaction for collaborative design.

The methodology explored in this paper only begins to explore the possibilities of full-text analysis for deriving a model of the design and its application. For example, one could augment the learned design structure with formally derived ontologies or use the learned structure as the basis for a formal ontology. However, if we decide to use full-text documents as the representation of the design rather than a computable, decideable and structured representation, we must look at the issue of efficiency in reasoning over the representation. Although the designer can always gather more information by running more queries, the designer also incurs a cost (perhaps measured by time) in running more queries and reading more documentation. Even if the

M. Agogino

system can achieve 100% precision in retrieval, i.e. retrieval of only relevant information, how can the designer be sufficiently appraised that no further searches are warranted? That is, we must assess the utility of the design documents in the current database in matching the designer’s information requests, with the expected value of information of the document. The belief network could then serve as a guide for directing the designer towards topics around the current context when the value of doing so is warranted. Each query term or phrase has a specific value in retrieving relevant information. Computing this value and then modifying the query to optimize the value of information returned is the next area for research.

ACKNOWLEDGEMENTS

The authors would like to acknowledge William H. Wood III for his intellectual contributions from his doctoral dissertation which led to this work and valuable comments on this paper. Dr Wood also converted the scanned document images into ASCII text. We would like to acknowledge John Wiley and Sons, for their permission to scan and OCR the text used for research and testing. We would like to thank in particular our industrial partners, Sun Microsystems and Autodesk, not only for financial and equipment support but for valuable collaboration. This research was sponsored by the NSF Concept Database grant No. DDM-9300025.

REFERENCES

1.

2.

3.

4.

5.

6

Baya, V., Gevins, J., Baudin, C., Mabogunje, A., Toye, G. & Leifer, L., An experimental study of design information reuse. In Proceedings of the ASME Conference on Design Theory and Methodology, DE-Vol. 42, pp. 141-7. Toye, G., Cutkosky, M. R. & Leifer, L., SHARE: a methodology and environment for collaborative product development. Post-Proceedings of the Second IEEE Work- shop on Enabling Technologies: Infrastructure for Colla- borative Enterprises. Center for Design Research Technical Report CDR-TR No. 19930507, Standford University, CA, 1993. Akman, V., ten Hagen, P. J. W. & Tomiyama, T., Desirable functionalities of intelligent CAD systems. In Intelligent Systems in Design and Manufhcturing, ed. Cihan H. Dagli & A. Kusiak. ASME Press, New York, pp. 119- 38. Ullman, D. G., Dietterich, T. G. & Stauffer, L. A., A model of the mechanical design process based on empirical data. Artificial Intelligence in Engineering Design and Manufacturing, 1988, 2(l), 33-52. Baya, V. & Leifer, L., Understanding design information handling behavior using time and information measure. Proceedings of the 1995 Design Engineering Technical Conferences, ed. A. C. Ward, ASME DE-Vol. 83, Vol. 2, 1995, pp. 555-562. Altemueller, J., PDESISTEP Implementation Levels. IS0 TC184/SC4/WGl Document N282, October. Also described in http://elib.cme.nist.gov/nipde/stds/pdes.html.


7. Szykman, S. & Cagan, J., A computational framework to support design abstraction. In Proceedings of the ASME Conference on Design Theory and Methodology, DE-Vol. 42, 1992, pp. 27739.

8. Wong, A. & Sriram, D., SHARED: an information model for cooperative product development. Research in Engi- neering Design, 1993, 5, 21-39.

9. Olsen, G. R., Cutkosky, M., Tenenbaum, J. M. & Gruber, T. R., Collaborative engineering based on knowledge sharing agreements. Concurrent Engineering: Research and Applications, 1995, 2(3), 145-59.

10. Shah, J. J., Bliznakov, P. & Urban, S. D., Development of a machine understandable language for design process representation. In Proceedings of the ASME Conference on Design Theory and Methodology, 1993, DE-Vol. 53, pp. 15-24.

11. Reich, Y., Konda, S. L., Levy, S. N., Monarch, I. A. & Subrahmanian, E., New roles for machine learning in design. Artificial Intelligence in Engineering, 1993,8, 165-81.

12. Gruber, T. R., Tenenbaum, J. M. & Weber, J. C., Toward a knowledge medium for collaborative product development. In Proceedings of the Second International Conference in Arttficial Intelligence in Design, Pittsburgh, ed. John S. Gero, 1992.

13. Wood III, W. H. & Agogino, A. M., A case-based conceptual design information server for concurrent engineering. Computer-Aided Design, 1996, 28(5), 361-9.

14. Dong, A., Agogino, A. M., Moore, F. & Woods, C., Managing design knowledge in enterprise-wide CAD. Preprints Advances in Formal Design Methods ,for CAD, ed. J. S. Gero and F. Sudweeks. Key Centre of Design Computing, University of Sydney, Sydney. 1995, pp. 330@ 47.

15. Wilensky, R., Planning and Understanding: A Computa- tional Approach to Human Reasoning. Addision-Wesley. Reading, MA, 1983.

16. Gruber, T. R.. Toward principles for the design of ontologies used for knowledge sharing. Formal Ontology in Conceptual Anal_vsis and Knowledge Representation, ed. Guarino & Poli, Kluwer Academic Publishers, Dordrecht. Technical Report KSL 93-04, Knowledge Systems Labora- tory, Standford University, 1993.

17. Lewis, D. D., Croft, W. B. & Bhandaru, N., Language- oriented information retrieval. International Journal of Intelligent Systems, 1989, 4, 285-318.

18. Eastman, C. M., Bond, A. H. & Chase, S. C., A formal approach for product model information. Research in Engineering Design, 1991, 2, 65-80.

19. Ullman, D. G., Wood, S. & Craig, D., The importance of drawing in the mechanical design process. Computers and Graphics, 1990, 14(2), 263374.

20. Pfeifer, U., FreeWAIS-sf http://ls6.informatik.unidort- mund.de/ir/projects/freeWAIS-sf/.

21. Evans, D. A., Ginther-Webster, K., Hart, M., Lefferts, R. G. & Monarch, I. A., Automatic indexing using selective NLP and first order thesauri. In Intelligent Text and Image Handling Proceedings of a Conference on Intelligent Text and Image Handling ‘RIAO 91, Barcelona, Spain, ed. A. Lichnerowicz, 1991, pp. 624-43.

22. Gardiner, D., Riedl, J. & Slagle, J., TREC-3: Experience with conceptual relations in information retrieval. Pro- ceedings of the Third Text Retrieval Conference (TREC-3)) Gaithersburg, MD, 1994.

23. Duda, R. 0. & Hart, P. E., Pattern Classtfication and Scene Analysis. John Wiley, New York, 1973.

24. Pearl, J., Probabilistic Reasoning in Intelligent Systems: Networks of’ Plausible Inference. Morgan Kaufmann, San Mateo, CA, 1988.

25. Russell, S. J. & Norvig, P., ArtiJicial Intelligence: A Modern Approach. Prentice Hall, Englewood Cliffs, NJ, 1995.

26. Cooper, G. R. & Herskovits, E., A bayesian method for the induction of probabilistic networks from data. In Machine Learning, Vol. 9. Kluwer. Dordrecht, 1992, pp. 309947.

27. Heckerman, D. & Geiger, D., Learning bayesian networks. Microsoft Corporation Technical Report MSR-TR-95-02, 1995.

28. Lam, W. & Bacchus, F., Using causal information and local measures to learn bayesian networks. Proceedings of the Ninth Conference on Uncertainty in Arttjicial Intelli- gence, ed. D. Heckerman & A. Mamdani, Morgan- Kauffman, San Mateo, CA, 1993, pp. 2433250.

29. Myer, K. (ed.), Mechanical Engineers’ Handbook. John Wiley, New York.

30. Garcia, A. C., Howard, H. C. & Stefik, M. J., Improving design and documentation by using partially automated synthesis. In Arttficial Intelligence in Engineering Design and Mamtfacturing, ed. C. L. Dym, 1994,6(l), 335- 354.

text analysis for constructing design representations

Documents