Fl,-,HEWLETTa:~ PACKARD

The Application of a ModifiedCard Sort in the Creation of anOrder Management Knowledge-Base

Paul Eccleson, Christoph Jakfeld*Information Management LaboratorylIP Laboratories BristollIPL-92-76June, 1992

knowledgeacquisition,knowledgeelicitation

Generalized knowledge acquisition tools have beendeveloped which will support the elicitation ofknowledge from human experts. This paper will showhow the adaptation and specialization of the card sortingknowledge elicitation technique has proved useful in oneimportant application within Hewlett-Packard.

Hewlett-Packard (lIP) is introducing an orderinformation management system which containsembedded knowledge-bases for automated handling ofcustomer requirements. The knowledge bases will becreated and maintained by each lIP entity that uses acopy of the system. This paper will outline how theapplication of a modified card sorting method hasassisted in the creation of such knowledge-bases and howa tool to support the modified technique has beendeveloped to assist every lIP entity involved in such acreation process.

*DFKI, Kaiserslautem© Copyright Hewlett-Packard Company 1992

Internal Accession Date Only

1 INTRODUCTION AND BACKGROUND

Hewlett-Packard is a worldwide manufacturer of electronic, medical, measurement andcomputing equipment. Its product range is broad and each manufacturing entity within itwill be dealing with markets and customers unique to itself. These factors present a consid­erable challenge to designers of information systems for internal use within HP. This is par­ticularly true of a system to handle the reception and management ofcustomer orders. Suchan information system will need to maintain data storage and communication strategies thatare common across the many HP entities that have a stake in the order fulfilment process.Also, such a system should allow individual entities to optimise their own order handlingto best fit their particular market and customer needs.The major tasks of an order handling system are:

• The reception of orders from customers which state product, option, quantities andacceptable delivery dates.

• The matching of these requirements to the factory production plans.• The management and tracking of orders from reception through to fulfilment and

shipment.

HP employs a central organisation to develop and support corporation-wide informationmanagement systems - Product Generation Information Technology (porn. This grouphas developed the latest generation order management system, Apogee. Apogee runs on aHP3000 computer under MPE and is built upon a database design named Orbit that holdsdata in a common format for each entity using the system. Apogee is responsible for anumber of order management functions:

• Data communications between manufacturing sites and sales offices.• Matching of customer requirements to manufacturing capacity.• Presentation and reporting of information for order management staff.

• Tracking and warranty documentation.

• Shipment and logistic information.Apogee contains an embedded knowledge-base for automated handling of customer re­quirements. The knowledge-base will be created and maintained by each HP entity thatuses Apogee.As part of the deployment and testing of the Apogee system, PGrr ran two beta-tests at HPmanufacturing entities. The beta-test involves the creation and running of the new Apogeein parallel with the old information systems within the entity. The rest of this paper will beconcerned with only a small portion of the matching ofcustomer requirements to manufac­turing capacity - the orderbooking cycle - as it was implemented and developed during thebeta-test of Apogee at the HP peripherals manufacturing division, Computer PeripheralsBristol (CPB).The installation and testing of Apogee at CPB showed that the creation of the system'sknowledge-base caused some problems for the order management staff. For that reason theknowledge elicitation technique cardsorting was applied to encode the knowledge of theorder management team in Bristol into Apogee. To support the divisions which are goingto use Apogee, the card sort method was extended. A tool was implemented based on agraphical user interface realizing a special instance of card sorting. The graphical represen­tation enables a transformation of the order managers' knowledge into the formalism usedby the booking system.

2

2 THE APOGEE ORDER MANAGEMENT SYSTEM

Within Apogee, an order may contain requirements for a number of different products indifferent quantities. Orders that arrive at any HP entity will have already been pre-proc­essed and should only contain requirements for products that the entity manufactures.These orders may then befurther divided into partial orders for different quantities of prod­uct at the entity's convenience. The part of Apogee that we will focus on deals with the al­location of these order requirements to the planned capacity of the HP entity. The plannedcapacity is stored by Apogee in what we shall term a "slot plan". This is aDATElQUAN­TITY structure which will be filled by ordered units. The slot plan represents the numberof units of each product type to be built each day/week/month of the current planning ho­rizon.

The Apogee system handles the information flow regarding orders, ordered products,planned capacity for those ordered products and shipment documentation. An overview ofthe Apogee booking process is given in figure 1.

Sort Order Queue

Calculate Order Priority

Prioritized Order Queue

Sequenced Order Queue

Select Top Order

Top Order

Slot Plan

QTYL ...

11M.-DATES

Book

Requirement

Select rules and proceduresfor Booking Requirementsto slot plan

BOLD =Knowledge-based operations.

Figure 1. APOGEE ORDER BOOKING PROCESS

The order queue in figure 1 consists of any orders that remain to be processed. These maybe orders new to the system, or orders which have been returned to the queue because theircircumstances have changed for some reason. Each order in the queue is then scored by thepriority rule set The queue is then sorted on the scores obtained for each order and the orderwith the most points is taken to be allocated into the capacity plan. Once that order has beenallocated to a date (or, possibly, a series of dates if it is an order for multiple units) the next

3

order in the queue is taken for processing. The privilege of higher priority is first choice ofthe remaining production capacity. Once an order has been selected for allocation in to theplanned capacity (the slot plan) the system needs to allocate a booking procedure to it. Abooking procedure describes how the remaining space in the slot plan is to be searched tofind a build date(s) for the requirements of the order. A library of booking procedures canbe defmed by the HP entity and rules may be created to allocate one of these procedures toan order.This paper will focus on the creation of rules for prioritising orders in the order queue anda modification of the card sorting knowledge acquisition technique which was used to ac­quire these expert rules from the personnel within CPB.

3 RULE-BASED ORDER PRIORITISATION

Apogee presents users with a rule language which will automatically order the bookingqueue. The aim of the card sorting activity to be described in this paper is to generate rulesin this given language.The orders within the Apogee queue waiting to be booked are sortedon a weighted total derived from allocating points according to conjunctions of attributesof the orders. The points are allocated by rules which look at the attributes on the order asthey arefound in the database. For example, a rule set for prioritising orders could includethe following:

WT CRITERIA

1300 HOLD-REASON =BLANKORDER-TYPE =A,B,C

Where A,B and C are classifications relating to the customer placing the order and theHOLD-REASON field may contain a flag to signal a temporary halt in the processing ofan order for some reason. This rule can be translated as:

"Add 1300 points to the current priority score of any order that has a blankHOLD-REASON field AND the ORDER-TYPE field contains one of A,B or C"

Given that the definition rules focus on attributes of orders which can be used to contributeto a weighted total, we approached the selection of appropriate order attributes from twodirections:

1. Interviews were held with each member of the order management team to assesswhat attributes of the incoming data they were using in their day-to-day prioriti­sation of the backlog. This produced a list of useful attributes free from consider­ation of whether that data could actually be extracted or derived from the Orbit da­tabase by Apogee.

2. A brain-storming session, held with the supervisor and systems specialist of theorder management function, derived a list of data items in the database whichcould be useful for order prioritisation.

4

The two lists derived from these processes were then screened for those items which couldnot be used or were deemed not to be useful in the prioritisation scheme. Some customisa­tion of the attributes was also possible to more accurately reflect the concerns of the ordermanagement team (e.g. creating special code values in the database to classify particularorders).

The next task in the priority definition process was to derive a set ofmockordersusing theattributes defined above and to have the order managers sort these mock orders into theiridealsequence (i.e. a sequence which could form the target for the eventual automated ruleset). To do this we employed a variation of a knowledge elicitation technique known ascardsorting (Gammack, 1987) whilst encouraging the subjects to talk about the decisionsthey were making in a kind ofprotocolanalysis (Hayes 1989). The attribute/value combi­nations were written on to cards, each card representing a mock order. A card sorting ex­ercise was then undertaken. First the subjects sorted the cards in to groups of similar orders.This process also produced a set of cards that were equivalent to others or contained invalidcombinations of attributes (the original drawing up of the cards had been done without thisparticular knowledge). The subjects then proceeded to sort the mock orders WI1HIN theircategories and then merge categories together. The final outcome was a first pass orderingof the test data set This first pass was then reviewed with the order management supervi­sors, some alterations made to overall priorities of certain types of order, and an ideal se­quenceproduced from this final sorting.Having derived an ideal sequence, the next task was to define a set of rules which couldautomatically generate that ordering when fed with the test set. For this phase of the proc­ess, a small computer-based rule interpretation and order sorting program was written. Thisproved invaluable in the rule creation/debug cycle. The process adopted in this phase canbe summarised as:

----1Sort Orders 1---.1Compare to I~Rules for'------r--..,I~ IIdeal . Apogee

Review EITIIER RulesOR Ideal Sort

Figure 2. Edlt-Run-Debug Cycle

Occasionally, the cycle would lead to an adjustment to the ordering of the data in the idealset and thus a rule set and a ideal sequence for the test data were produced. In order to testthe completeness of its coverage, the rule set was then hand tested with orders not in theoriginal data set to verify that the rules could cope correctly with data not present in the testset. A stable rule set was achieved when both the original data set and any exceptional or­ders that were generated by the implementation team could be placed in a sequence withwhich the order management team was happy.When creating mock orders in this manner there is a continuing problem of completeness.The original set of attributes may not include special cases. The values chosen for thoseattributes may not cover particular types of order. The combinations of attributes and val-

5

ues chosenon each ordercard will not comprise a complete set of cases.The onlycompro­mise that has been achieved with this development cycle has beenrepeatedreviewandex­ceptionalcase testing with experts.To summarise the processfor prioritydefinition:

• Interview order management staff regarding priorities.• Produce a short-listof database attributes whichcould be used to satisfy the pri­

oritisation decisionselicited through interview.• List values that these attributes might take.• Drawupmockordercardseachwithdifferentcombinations of orderattributeval­

ues.• Performa card sort with ordermanagement personnel to elicit such thingsas ille­

gal attributecombinations, ordercategorisations and an ideal sequence for the testdata set.

• Manually generatea rule set which will sequencethe testdata in accordance withthe ideal sequence.

• Review both rule set and ideal sequence in the light of the rule creation/sortingprocess.

• Hand test stable rule set with orderexamples not found in the test set.

4 CRITIQUE OF THE MANUAL CARD SORT METHOD

The card sorting method seemed to be a natural match to the domain in which the ordermanagers workedevery day. This is a domain in which global heuristics are used to dealwiththe massof data with whichordermanagers are confronted. Suchglobalheuristics areof the nature"Orders rated X,Y & Z by the salesorganisation are more important than anyother".The subjectsfound sortingthrough example ordersand placingthemin a sequencerelative to one another preferable to trying to express their priorities and concernsduringan interviewsession. The prioritisation of ordersdepends upona large numberof parame­ters and their combination on a singleorder.The kinds of heuristics broughtto bear on theprioritisation process include:

• Global"rules of thumb"(e.g. category A ordersare very important).• Local exceptions (e.g, if HOLD-REASON has been set, an order's importance is

lowered).• Organisational strategies (e.g. a particularordermay be part of a salespromotion

so it is more important).• Entity strategies (e.g, Marketing are emphasising certain types of shipment this

month).The card sortproduceddata related to each of these typesof heuristics via:

• The initialcategorisation of ordersproducedheuristics operatingat the globallev­el.

• The sequencing of the orders withinthesecategoriesproducedlocalexceptions.• The mergingof the ordercategoriesproduced"leaps" in the relativeplacementof

orders whichwere related to organisational andentity strategies.The experts' knowledge consists of qualitativejudgementson importance. To implementthese judgementsin Apogeerequires the creationof rules with a quantitative value associ-

6

ated with them. Since weights are cumulated during the evaluation process the numericaldifference has a great impact on the sequence of sorted orders. Transforming the qualitativeknowledge about orders into quantitative rules therefore is the question of how much moreimportant are orders of one category compared to another one.

Despite its utility, the manual card sort method has several weak points:

• It is cumbersome to handle a large pile of cards

• Cards and rules are difficult to edit or to change (e.g. changing the set of attributesor allowed values at a later stage of work)

• Documentation overhead is large (e.g. reporting about the application of the rulesand the priorities it produced)

• Storage and retrieval problems due to the nature of real cards• Needs some expertise/support (e.g. transforming qualitative knowledge into quan­

titative rules)

5 REQUIRED FEATURES OF A CARD SORT TOOL

Installers of Apogee face problems right from the beginning. The selection of appropriateorder attributes by interviewing members of the order management team has to be per­formed carefully. A redefinition of attributes or values at any stage of the process shouldbe easy and not cause the work to start allover again. After the selection process editing orcreating, changing and deleting cards, which represent the mockordercards, has to be as­sisted.

A tool to support the card sort technique defined so far would need to be designed to meetthe needs of both the Apogee rule-base builders and experts in the domain. Experts wouldbe expected to interact with the tool when creating the ideal sorting of the mock order cards.The rule-base builder would then edit and debug the rules to automate the production of anequivalent ordering of the test set.The order managers generate the ideal sequence by sorting the cards into groups of similarproperties and sorting again within these groups. At this stage it is useful to have variousand different views onto the cards. Storing and finding these groups, ungrouping and visu­alizing this process are required operations as well.Generating a rule-set from the card sort is the least structured step in the technique. Theorder managers expressed their priorities in a qualitative fashion (e.g. "orders with this at­tribute are slightly more important than these orders"). The assignment of quantitativeweights to these qualitative judgements seemed artificial. Therefore an interface that al­lows qualitative card sorting and weight creation or editing and transforming this qualita­tive information into a quantitative one is essential.The high degree of interdependency between cards and rules forces the user to be very care­ful while debugging the process. To overcome these main problems the storage of cardsand rules in the same database helps to a great deal, since it enables various kinds of cross­referencing between these entity sets.The edit-run-debug cycle ends when the test set and ideal sequences become acceptablysimilar. To assist in the comparison step, the two sequences may be seen side-by-side inthe card sort tool.The other important problem which occurs while using Apogee is the maintenance of theorder or card definitions. Redefinition might be caused by mistakes in the knowledge elic­itation process or by changes in the environment of a division. The latter includes new man-

7

agement strategies or a modified behaviour of the customers, as well as the extension ofproduct lines. Due to the time gap between the installation process of the system and theneed for modification it might be difficult to change or extend the rule set. Understandingthe old set of rules is important when adding new ones and maintenance should provide aneasy way of extending the system without destroying the parts of it which already work.

In order to support all of these uses a tool would need the following features:

• Reldefinition of relevant attributes according to the division's needs• Editing, changing and providing various views on cards and rules

• Grouping and sorting cards to create an ideal sequence. Storing and retrieving thissequence for later comparisons

• Visualizing interdependencies between cards and rules

• Visualizing the comparison between ideal sequence and rule sorted set of cards indifferent ways

• Representation of the qualitative information of the experts' knowledge from thecard sorting process and a facility to use this representation to define the rules.

6 IMPLEMENTATION OF THE CARD SORT TOOL

The architecture of the card sort tool is shown in figure 3. The tool runs under Windows3.0 on a HP Vectra PC. It consists of two main modules:

• A card sort and rule creation interface written in the Hypertext system Toolbook.

• A meta-level interpreter written in Quintec Prolog to score the test set of orderswith a rule set defined in Toolbook.

The two modules communicate via the Windows 3 Dynamic Data Exchange protocol. Thedevelopers of Apogee have provided a rule entry interface which is also implemented inWindows 3. This Defpri software communicates the rule definitions made by the userthrough to the HP3000 computer on which the main Apogee system runs. Users of the sys­tem would use the card sort modules to create cards and rules (in the manner described inthe previous sections) and then transfer the finished rule set to Apogee via the Defpri edit­ing interface.

8

Card+RulesWeightTrace Info

.. '1~~r~~~fif~1~I~1~1~~~~j~~~~

a-!1WeightsTrace Info

Card+RuleDefinitions

illl!'!'!!I:::::·,::·::':·:·· - .ijl~:Card+Rule Creation

iil't~~!:;\1~:;':'~*t*L.,...;;:

Figure 3. I-MRP Architecture

The database managed by Toolbook is object oriented. Every card and rule is an object Theattributes of cards are the object's slots and the values are the slot-fillers. Slots can be filledeither extensionally or functionally. Values are filled in extensionally by typing or by se­lecting default values with the mouse. Values are filled in functionally by a number of soft­ware functions which can automatically calculate appropriate values.The rule interpreter takes rule and card definitions from the Toolbook module, converts thedata into PROLOG evaluable form and runs a meta-interpreter over the rules to weighteach mock order as Apogee would. Since the priority of a card is the total of all weights ofrules which matched or fired on this card, the sequence of applying the rules is of no im­portance. In addition to this, problems like confluence and termination do not arise. A setof rules is confluent, if for every object to which more than one rule can be applied, thesequence of applying these rules has got no influence on the result. Since the card sort tooldeals with cumulative numeric priorities the commutativity and associativity of additionguarantees confluence. A set of rules terminates if only a finite number of rules can be ap­plied to every object Since the role sets of the card sort tool are finite and every role firesat most once on each card, termination is guaranteed.The implementation of a meta-interpreter allows tracing information to be recorded andpassed back to the Toolbook module. If a rules fires on a card, the tuple of their identifiersis stored in the Prolog database. The set of these tuples allows cross-referencing betweenthe set of cards and the set of rules after all the cards have been examined.

7 USING THE CARD SORT TOOL

This section will attempt to give an account of how the user interface in the card sort toolsupports the card sort technique described in section 3. In addition to the provision of helpin using the card sort technique. the method itself is extended by supporting easy and fastaccess to stored cards, various views onto the cards and the rules and visualizing objects'

9

properties, e.g. the weight of rules. The users should have the same feeling as soning "real"cards on a desk, since this seems to be a natural way for humans to solve a soning problem.

Assignment of database attributes to be used in the prioritisation of orders. The workwith the tool starts with raising and defining the set of attributes and their values. The cardsare built with these attributes and slots or text fields to enter values. As outlined above, re­view of the selected attributes and/or values can be made at any time. The attributes addedby the user are entered incrementally on to an order card "template" as labelled fields, Thevalues for each attribute may "pop up" as a menu from which each field may be filled.

Creation of the test set of mock orders. Having decided upon the attributes and allowedvalues for those attributes, a set of cards which will represent the mock order test set is thencreated. This defmition of individual cards is simply the instantiation of instances of thecard template defmed previously. Values are chosen for each attribute field to produce aunique instance of a mock order. Editing the cards is done by filling in the actual valuesinto the text fields using the keyboard or the mouse. Flies of cards can be merged and at­tributes of other flies can be appended. Changing or deleting the values of cards is possibleat any time. Figure 4 shows values being entered into the mock order template to create aninstantiated card for the test set

Order card: 10 Weight: 250

A1TRIBUTE: VALUE: OPTIONS:

I •Panial Priority C A

B

I 0C

Customer Type DE

Order Value I 0

I 0Product Number

Figure 4. An Example Instance of an Order Card

Production of the ideal ordering. Once a test set of mock order cards has been created,the complete set must then be sorted by the order management experts into an ideal se­quence. This sequence will represent the relative ordering of the test set that the order man­agement team feel is appropriate to their current order prioritisation heuristics. In the Ap­ogee rule language, sequencing of orders is achieved through numerical assignments or oneof three symbolic weights:

• NO is set for orders which should not be booked at all.

10

• HIgh and LOw place the cards they fire on respectively on top and at the bottomof the queue.

To avoid clashes, NO overrides both HI and LO. The card sort tool translates between theuser's natural tendency to think in termsof relative placement of orders and Apogee's in­sistence on numeric values, through the manipulation of tokens representing cards on ascaled board. Figure 5 shows the screen representation of this board. Card tokens placed onthis board are allotted a weight based on its position on this board. Thus top-bottom relativeseparation of cards automatically generates numeric distinctions between those cards. Thescale of the board may be defined by the user. The user may type in numeric values for acard's priority and the token will be placed at the appropriate place on the board for thatvalue, but the focus of the task remains the relative placement of cards on the board.At any time, cards can be presented in full detail (as illustrated in figure 4) or a view overthe whole set can be shown. Window's multi-tasking enables the user to see as many cardsas he likes simultaneously. It is possible to start several Toolbook applications all dealingwith the same database and rule-set. The different applications present different pages ofthe card sort tool and modifications in one of them are sent to the others immediately. Afterthe ideal sequence is defined it can be stored for comparing it to the rule sorted set at a laterstage. This ideal sequence may be restored and modified at a later time. The ideal sequenceis a relative ordering among the cards. Therefore only the qualitative judgement is of im­portance. The adjustment of relative order is represented graphically as shown in figure 5.

NO

HI

NUMERICAL

Up-DownMovement

LO

-----.II:~DefinesPriority

Figure 5. Changing Order Priority

Defining rules that will order the test set. After defining the ideal sequence of orders, ap­propriate rules have to be created which will automatically sort the example cards. Eachrule consists of a weight, which is either numerical or one of the special symbols introducedabove, and a criterion. In accordance with the rule language defined by the developers ofApogee, a rule criterion is the conjunction of at most eight items, each of which consists ofa binary predicate and two values applied to it. Valid predicates are: test for equality. mem-

11

bership in a set,membership in an intervaland the negation of all these three.Validvaluesare two attributes or one attribute and a set, which might be a singleton but must not beempty, of values for that attribute. All predicates work on exact patternmatch,except forthe specialsymbolsDATED, whichmatches witheveryvaliddate on a card,and somecol­loquialtermsfordates likeTODAY. A rule flres,and thereby addsits weightto thepriorityof the card, if all its items or conditions evaluate to true. The restrictions concerning thenumberof conditions and predicates are due to the requirements of Defpri.It is possible toadd,changeor delete a rule or a part of it at any time.Figure6 showsthe rule constructionform. On this form, there is an assumedlogical conjunction between each line entered inthe criterion.The primarytask in rule creation is to apply the correct weights to the appropriate ordersin the test set Once again,APOGEE requiresa numeric weightto represent whatis essen­tially a qualitative figure, For example, it may be apparentto the knowledge engineer thatan order withvalue 'a' for attributeX is very important, but the assignment of an absoluteweight to this importance seems artificial. To alleviate this problem, the card sort tool al­lows the knowledge engineer to assign weights to rules via a rule sortingscreensimilar tothemockordersortingscreendescribed earlier.A 'feel' forqualitative differences betweenthe importance of rules can be reflectedin relativepositioning on the rule sortingboard.

Rule No: 1L...7~ _

Criterion:

Weight: 1,,-,;;,,;I00~ _

ORDERVALUE 0

ORDER VALUE 0

IS>

IS <

IS IS NOT

IS IN NOTIN

IS < IS>

200

400

oo

Figure 6. The Rule Entry Screen

Sorting of Mock Orders with Rules and Comparison to Ideal Sequence. Having de­fined the rulesand sortedthe test set with themthrough the meta-interpreter, theirperform­ance must be comparedto the ideal sequence. This can be done either by:

• Comparing a card tokenboard(whichhas beensequenced automatically usingtheexistingrule set) with the token boardfrom the ideal ordering, or

12

• Comparing a list of card identifiers with a list derived from the ideal sort.The ideal sequence is a sequence of sets rather than a total ordering. Within these sets thedifferences between the orders are only marginal. For example, figure 7 shows the way inwhich sortings are compared to the ideal sequence. The discrepancy between the ideal andrule sorted sets may be insignificant A simple algorithm would find a meaningless differ­ence between the two lists. The human ability to compare complex patterns very efficientlyis therefore supported by graphical features. One of these is grouping the mock orders ac­cording to properties they have. Once a group is sorted and stable in terms of applying therules to it, the user can focus his attention to a different group. Figure 7 shows the sortedset/ideal sequence comparison screen.

ICompare Sorting I

Ideal Sequence

5I47236

IPRINT I

Rule Sorted Cards

54I7236

IGOTOCARD I

Rules used by 6

794

IGOTORULE I

Figure 7. Soned vs Ideal set .COmparlson

Rule debugging and Editing. Of crucial importance in the creation of automatic orderingrules is the ability to trace the dependencies between rules and the cards to which they con­tributed their weight For example, if a card is prioritised by the rule set lower than it shouldbe according to the ideal sequence, we would want to increase its weight. This may beachieved either by increasing the weight of one of the contributing rules or by creating anew rule that would apply to this mock order card. In either of these circumstances wewould need to be aware of the consequences of either action on the priority of the card inquestion and the other cards in the test set Card-rule tracing information from the applica­tion of the rule set in the meta-interpretation module is made available throughout the in­terface. Figure 8 shows the interdependencies that exist between various components of thecard sort tool. For example, token number 23 on the "SORT CARDS" screen representscard number 23 as has been previously defined. This existing card has a priority assignedto it via the role interpreter. Its priority includes 1300 points that have been contributed by

13

rule number 4. Rule number 4 has been applied to other cards and is represented on the"SORT RULES" board by token 4. Each of these interdependencies may be traversed bythe user via mouse clicks on the appropriate screen.To summarize how the card son tool presents information:

• Graphical representation of the ideal sequence's relative order

• Grouping and sorting cards according to their properties is supported by colouringor placing cards

• Cross-reference between cards and rules. Different views onto cards and a fastway of switching between these views

• Multi-tasking enables as many views as needed• Redefmition of ideal order, cards or rules at any time

Figure 8. Interdependencies between cards and Rules

8 Discussion

The card sort tool and method presented in this paper represents a specialisation of the tra­ditional knowledge elicitation technique of card sorting. There are two advantages of the

14

traditional card sort technique that the specialisation of the method presented in this paperhas lost Firstly, the traditional method of card sorting allows the expert to add objects intothe stack of cards if he or she feels that the object is relevant to the domain and has beenomitted from the test set This allows for extension of the test set as a move towards com­pleteness. In the method presented, the cards represented combinations of attributes andvalues in the domain. It was difficult for the domain experts in this case to spot incomplete­ness in the coverage of the test set The combinatorics of the attribute/value pairs made thelocation of omissions very difficult Secondly, the traditional card sort method gives directassistance to the knowledge engineer in the creation of a knowledge base which would op­erate in the domain. The card sort method presented here produces a conformance standardto compare the performance of a rule set against, but produces limited amounts of directinformation that would allow the creation of an appropriate knowledge-base. The kind ofinformation produced by the method presented here is much more allied to that producedby protocol analysis, where some interpretation of results needs to be made before a knowl­edge-base can be created.The tool and method did, however, have considerable advantages in the domain whichmade it a useful knowledge acquisition tool. The most important features of the tool are:

• Both experts and knowledge-base builders are allowed to express qualitativejudgements through relative placement of objects on a board. Apogee requires thathuman experts quantify their 'feel' for importance of orders. This judgement, andthe creation of weights to emulate that judgement, is hard to quantify meaningful­ly. The relative placement activity seems a much more natural way for experts toexpress their knowledge.

• Rule editing, testing and tracing are preformed within the same environment al­lowing rapid prototyping of knowledge-bases. Of particular importance here isthat valuable cross-referencing between cards and rules is performed. During therule debugging cycle, manipulation of test sequences needs to be performed withfull knowledge of rule/card interactions if rules are to be edited successfully. Thecard sort tool supports this activity.

• The documentation and storage/retrieval of information are automated.It may be useful to include a rule induction package in the card sort tool at a later date. In­itial investigations have shown that the data present in the domain lends itself to formula­tion as an induction problem:

• The example orders consist of a fixed set of attributes and values.• These attributes are generally qualitative in nature.• The ordering of examples can be couched as a classification task.

Some problems may lie in the interpretation of decision trees produced by the induction al­gorithm by the experts themselves. Further investigation of this may be undertaken.The philosophy behind both the method and the tool was the creation of a process thatwould map naturally on to the needs of both experts in the domain and Apogee rule-basebuilders. Through the provision of a qualitative interface and support for rapid prototypingand documentation, the card sort tool has embodied this philosophy.The method and tool have been used to produce the priority definitions that are currentlyin use at Hewlett-Packard in Bristol, England. The rule set defined by this method has beenoperating successfully and no adjustments have been made since installation in September1991. The card sort method and tool is to be made available by the developers of Apogeeto each future division of Hewlett-Packard that wishes to install the Apogee system.

15

REFERENCESGammack, J (1987)"DifferentTechniques and DifferentAspectson Declarative Knowledge"

in "Knowledge Acquisition for Expert Systems"

Ed. Kidd,A. , Plenum Press

Hayes, Rick Stephan (1989)"Developing a Protocol-based ExpertModelofthe Decision Process ofInternationalBank CreditOfficers"

in "Knowledge Based Management Support Systems"Eds. Doukidis.G, Land.F and Miller.G, Ellis Horwood.

16