A MULTI-LEVEL DYNAMICINSTRUCTIONAL PLANNER

FOR AN INTELLIGENT PHYSIOLOGYTUTORING SYSTEM

0 CHONG W. WOO

GENERATION OF COMPLEX TUTORIAL DIALOGUESMARTHA W. EVENS, PRINCIPAL INVESTIGATOR

COMPUTER SCIENCE DEPARTMENT

ILLINOIS INSTITUTE OF TECHNOLOGYCHICAGO, ILLINOIS 60616 D IC

312-567-5153, csevensOminna.iit.edu LEECTE

SMAR 0 3994 LThis work was supported by the Cognitive Science Program, Office of NavalResearch under Grant No. N00014-89-J-1952, Grant Authority Identifica-tion Number NR4422554, to Illinois Institute of Technology. The contentdoes not reflect the position or policy of the government and no officialendorsement should be inferred.

Reproduction in whole or in part is permittedfor any purpose of the United States Government.

Approved for Public Release; Distribution Unlimited

94-06859• • 6$•I!ii~il#laiclllllllill

BestAvaillable

COPY

Form Approveid

REPORT DOCUMENTATION PAGE FoM No. 070o4-0.

PUIc 'COetl hnq oburden 'or th hillQ e lOn O If ,nform~ tnOrm $ a t.$ m A to .ver.lqe I "our 0 r on'•eo . nclIudlrnq the tine for revwvtnfj in$trUCtiO $. W• rh•nq e h.-tilq data $OU•tci

galth erin And maintaining the •aia ne•dd And O.mbjetjvq and reviewimq the coIIeerOn of ,nformation Send comment% rrqarding this burdenl estimate of any ,3her dlldet Of t-%i

collection Of inormationr. n h•oudin $u qestio•n for rf-d icjnq this iourden to 4V4$irnqtOn s•e6"uarters Services. Oureclorate for inftormation O ratomn$ and RepOrt$. Ii IS JeifeW,.nOawiI't " wa?, Sute 1204 Arlington. v A 12202.4J02 .and to tie Ot fie ,f Manaqemen and 8Bu.dget. PlOerwork Reduction Project (0704-0.18)., 'Vaslqngion, OC 20503

1. AGENCY USE ONLY (Leave blank) 2. REPORT DATE 3. REPORT TYPE AND DATES COVERED

1 1 April, 1992 Interim4. TITLE AND SUBTITLE S. FUNDING NUMBERS

A Multi-Level Dynamic Instructional Plannerfor an Intelligent Physiology Tutoring System G

N00014-89-J-19526. AUTHOR(S) R&T 4422554

Chong W. Woo

7. PERFORMING ORGANIZATION NAME(S) AND ADORESS(ES) 8. PERFORMING ORGANIZATION

Dr. Martha Evens REPORT NUMBER

Computer Science DepartmentIllinois Institute of Technology10 West 31st StreetChicago, IL 60616

9. SPONSORING i MONITORING AGENCY NAME(S) AND ADDRESS(ES) 10 S3ONITURING

Dr. Susan Chipman . P NUMBER

Cognitive Science Program NTIS CRA&JOffice of Naval Research DTIC TAB800 North Quincy Street Unannounced 0Arlington, VA 22217-5000 Justification .......... ..............

11. SUPPLEMENTARY NOTESBy --- ...----- .--- .. . .. ... . .Distribution I

12a. DISTRIBUTION / AVAILABILITY STATEMENT 121 DISTRMU 0NIMOPUMUS

Approved for public release; Avail and/ordistribution unlimited Dist Special

DTicQ

[ sifS A'a Jt2R19@W the design and development of an instructionallanner for an intelligent tutoring system for cardiovascular physiologyhat assists medical students to learn the causal relationships betweenhe parameters of the circulatory system, to understand how a negativeeedback system works, and to solve problems involving disturbances tohe system. The instructional planner is responsible for deciding whato do at each point during a tutoring session. It integratespportunistic control with sophisticated instructional planning,ombining lesson planning with discourse planning. The lesson plannings organized in three levels: goal generation, determination of planningtrategies, and choice of tactics to refine the goal into subgoals. Theixed-initiative discourse planning is implemented using a two levelpproach: pedagogical decision making at the upper level and tacticaliscourse state-based planning at the lower level in its discourseanagement network. It generates plans dynamically based on the student* odel, monitors their execution, repairs plans when necessary, and:eplans when the student asks a question or makes a comment.14. SUBJECT TERMS 15. NUMBER OF PAGES

Planning, Discourse planning, Tutorial dialogues 16.PRICECODE

Text generation

17. SECURITY CLASSIFICATION 18. SECURITY CLASSIFICATION 19. SECURITY CLASSIFICATION 20. LIMITATION OF ABSTRACTOF REPORT OF THIS PAGE OF ABSTRACT

Unclassified Unclassified Unclassified UL

NSN 7540-01-280-5500 Standard Form 298 (Rev 2-89)PrecriCbed by ANSI Sid 139.18298-102

A MULTI LEVEL DYNAMIC INSTRUCTIONAL PLANNER FOR ANINTELLIGENT PHYSIOLOGY TUTORING SYSTEM

Chong WooDepartment of Computer Science

Illinois Institute of TechnologyChicago, Illinois 60616

Phone: (312) 567-5153E-Mail: csevens@minna.iit.edu

I.NAn instructional planner in an intelligent tutoring system (ITS) is

responsible for deciding what to do next at each step during the tutoringsession. The planner has to decide what subject matter to focus on, how topresent it to the student and when to interrupt the student's problem-solvingactivity (Dede, 1986; Kearsley, 1987]. This pedagogical decision making isvery complex and there is no one correct choice due to the dynamic changes inthe student's learning state. Hence, the decision must be based on manydifferent knowledge sources, such as knowledge about the domain, knowledgeabout the student, and pedagogical knowledge about tutoring.

Recent approaches to designing tutoring systems view the decision makingprocess as a planning problem (Peachey and McCalla, 1986; Macmillan et al.,1988; Brecht et al., 1989; Murray, 1990]. Adaptive planning techniques in thetutoring domain enable the generation of customized plans for individualizedinstruction. Among the recent research systems, MENO-TUTOR [Woolf, 1984]represents an important attempt at planning the discourse strategies observedin human tutors, but it lacks global lesson goals [Murray, 1988]. IDE-INTERPRETER (Russell, 1988] is another attempt at planning the lesson goals atvarious levels of abstraction, but this system lacks power at the localdiagnostic level. Thus, there is a need to build an instructional planner thatcombines globally coherent lesson goals with flexible local discourse plans.

In this research, I am building a planner that integrates opportunisticcontrol with a sophisticated instructional planning methods; combiningcapabilities of lesson planning with discourse planning. This planner is- adynamic instructional planner that supports customized, globally coherentinstruction, carries out a mixed initiative strategy. It monitors currentplans in progress, repairs those plans, or replans as needed. This hasrequired the invention of multi-level instructional planning.

The goal of this research is to develop an ITS, CIRCSIM-TUTOR, thatassists first year medical students to learn the behavior of thecardiovascular reflex system that stabilizes blood pressure. Since thestudents have already attended lectures about the domain, CIRCSIM-TUTORassumes prerequisite knowledge and assists them to correct theirmisconceptions in a problem solving environment. This system is beingdeveloped as a joint project of Rush Medical College and Illinois Institute ofTechnology.

Eot g Comnuter-Based • a

Computer Aided Instruction (CAI) in the cardiovascular domain at RushMedical college has evolved from HEARTSIM [Rovick and Brenner, 1983], toCIRCSIM (Rovick and Michael, 1986], to the CIRCSIM-TUTOR prototype (Kim etal., 1989] and finally to CIRCSIM-TUTOR over the last ten years.

HEARTSIM was a Plato program and CIRCSIM is a stand-alone Basic program.The CIRCSIM-TUTOR prototype is a Prolog prototype of our ITS designed andimplemented by Kim [1989]. Its design is based on major ITS architecture.However, the prototype system still does not possess all of the capabilitiesneeded for an ITS. It lacks natural language capabilities, it does not analyzethe student's misconceptions, and the instructional planner is very primitive;

2

a discourse planner could not be implemented since complete discoursestrategies for all the primitive actions had not been developed, planningknowledge is not explicitly represented as a separate module, and there was noreplanning capability so that the system could not respond to studentinitiatives. CIRCSIM-TUTOR uses the same architecture as Kim's prototype butincludes complete student modelling, instructional planning, and naturallanguage understanding and generation facilities.

j*Organization

Section 2 describes the environment in which the system runs. Thesubject area of CIRCSIM-TUTOR is cardiovascular physiology and the systemassists students to understand the behavior of the complex negative feedbacksystem. Section 3 begins with a brief introduction to ITS: the generalstructure and the issues involved in each module of the ITS. Then eachcomponent of CIRCSIM-TUTOR will be briefly introduced. Section 4 presentsdesign issues for building the planner: levels of planning and tutoringstrategies. A short tutoring excerpt is displayed, from a transcript of humantutor and student interaction. And then a short scenario shows how the systemworks. It concludes with a discussion of the overall organization of theplanner: lesson planning, discourse planning, and plan monitoring. Section 5explains the generation of the content of lesson plans in detail. It firstdiscusses the main features of the planner: goal generation and plangeneration. And then it describes its own lesson planning rules: goalgeneration rules and plan expansion rules. Section 6 discusses the discourseplanner. The structure of the planner is a two level discourse managementnetwork, which consists of a set of states that represent tutorial actions.The control mechanism is separated into default and meta-rule transitions. Thepaper concludes in section 7 with a discussion of the significance of theplanner, describes some of its limitations, and gives suggestions for futureresearch.

Qualitative reasoning or simulation [deKleer and Brown, 1984; Forbus,1984; Kuipers, 1984] is an approach to problem solving that reasons about thecausal relationships that structure our world. Anderson [1988] argues thatqualitative reasoning is the most demanding approach and essential to producea high performance tutoring system. He states that qualitative modelling canmaximize the pedagogical effectiveness since it is human-like reasoning,although the implementation effort is much larger than that required for thetraditional black box models or glass box models. CIRCSIM-TUTOR is an approachto qualitative simulation in cardiovascular physiology [Michael et al., 1990].

The cardiovascular system consists of many mutually interactingcomponents, and the student must understand the cause and effect relationshipsfor each individual component of the system. Figure 1 shows a causal model ofCIRCSIM-TUTOR, called the "Concept Map," designed by Michael and Rovick (Kimet al., 1989]. Each box in the map represents a physiological variable, suchas SV for Stroke Volume and RAP for Right Atrial Pressure. An arrow with a "+"or "-" sign between two boxes tells the direction of the causal effects andwhether the causal relationship between the connected variables is direct orinverse. For example, a qualitative change in one component of the system, adecrease in RAP, directly causes a decrease in SV. This qualitative changepropagates to other adjacent components of the system according to thepropagation rule.

There are three stages in the human body's response to a perturbation inthe system that controls blood pressure. The first stage is the DirectResponse (DR) in which a perturbation in the system will physically affect

3

many other parameters. The second stage is the Reflex Response (RR), in whichother parameters are affected by the negative feedback mechanism to stabilizethe blood pressure. The final stage is the Steady State (SS), which isachieved as a balance between the changes directly caused by the initialperturbation and the further changes induced by negative feedback.

[RV + ]PIT[

;+ + -- N

Figure 1. The Concept Map

jjOrganizationCIRCSIM-TUTOR begins with a brief introductory message and then asks the

student to choose any procedure from the curriculum list. The curriculum(Figure 2) is stored as a set of four different experimental proceduresdesigned by our expert human tutors (JAM and AAR).

List of Available Procedures

1. Hemorrhage: Remove 1.0 Liter of Blood.

2. Decrease Cardiac Contractility (CC) to 50% of Normal.

3. Increase Venous Resistance (RV) to 200% of Normal.

4. Increase Intrathoracic Pressure (PIT) to 2 mmHg.

5. Quit.

Figure 2. List of Available Procedures

4

Each procedure begins by describing a perturbation of the cardiovascularsystem, and asking the student to predict how the system variables willrespond to the perturbation by making qualitative entries in the PredictionTable (see Figure 3); using a "+" sign to represent an increase, a "-" for adecrease, and "0" to indicate no change. The first column of the table is usedto predict the Direct Response (DR) of each variable to the perturbation, thesecond is used for the Reflex Responses (RR), and the third for the SteadyState (SS).

Parameters DR RR SS

Cardiac Contractility 0

Right Atrial Pressure

Stroke Volume

Heart Rate 0

Cardiac Output _

Total Peripheral Resistance 0

Mean Arterial Pressure

Figure 3. The Prediction Table

When the student finishes predicting all four parameters in one columnof the table, for example the DR stage, the student's answers are comparedwith the correct answers. If the student has made any errors, a naturallanguage tutoring session will begin, based on the result of this evaluationin order to correct the student's misconceptions.

2.3 Syste Constraints

There are some system variables that need to be described; the procedurevariable is the variable changed by the perturbation; the primary variable isthe first variable in the Prediction Table affected by the procedure variable,(in some cases the procedure variable is the primary variable); the neuralvariables are the variables directly under nervous system control. The rest ofthe variables we call physical variables. The students are not allowed topredict the variables in any arbitrary order, since there are some constraintsthat they must follow. For example, the constraints for DR are fairly complex:

Constraint DR1: The student must predict the primaryvariable first, and the value must becorrect.

Constraint DR2: The student must predict the physicalvariables in the correct causal sequence.

Constraint DR3: The student may predict the neuralvariables at any time and in any order.

5

The student receives a canned error message, when either of the firsttwo constraints is violated, and is told what to do next. The purpose offorcing the student into the correct sequence is to make sure the causalbehavior of the system is followed correctly. Neural variables can be enteredat any time since neural variables do not change during the DR period exceptwhen one is a primary variable. The constraints for the RR stage are designedto teach the students about the effect of the baroreceptor reflex:

Constraint RRl: The student must predict either theneural variables or MAP first.

Constraint RR2: The student must finish predicting allthe neural variables before predictingother physical variables.

Constraint RR3: The student must predict the physicalvariables in the correct causal sequence.

Finally, when predicting the SS stage, the student is allowed to enterpredictions in any arbitrary order since there are no specific constraints forthis stage.

2,.A Mlinlei simultaneous InutIn a mixed-initiative type of ITS, tutor begins by posing a question and

the student either responds to the question or takes the initiative. Sometimesthis style of tutoring leaves students confused and frustrated if they do nothave enough background in the domain knowledge. Rather than blindly walkingthrough the domain, it would be much more effective if the tutor provides asimulated problem situation in the domain for the student before the actualinteractive tutoring begins.

CIRCSIM-TUTOR begins with a Prediction Table, in which the student isasked to make qualitative predictions about the behavior of the system given aparticular perturbation. After the student finishes all the predictions, thetutor analyzes the student's answers and shows what errors were made if any.Based on a careful analysis of these errors, the tutor can generate a globallesson plan, and interactive tutoring begins by using a mixed-initiativeSocratic strategy in natural language. Thus, the Prediction Table provides aqualitative simulation environment for the student by requiring multiplesimultaneous inputs (multiple responses to different aspects of a problemprovided by the student in a single uninterrupted turn) before interactivetutoring begins.

There are several benefits of adapting this kind of design strategy.First, the tutor receives enough initial knowledge about the student so thatit can narrow the focus for tutoring. It can also detect some common studentmisconceptions [Michael et al., 1991] or bugs. Second, the students can see asimple mental model of the entire domain at the start, which prevents thestudents from getting too far off the track [Reiser, 1989]. Elsom-Cook [1988]argues that using multiple pedagogic strategies can provide a very powerfullearning environment. CIRCSIM-TUTOR begins with a coach-like environmentduring the Prediction Table entry, and then moves to Socratic tutoring for theinteractive tutoring session. This flexibility in adapting to the student'sneeds at different stages provides another benefit.

OR.ANZATION QE ICIMTTMost of ICAI systems have been separated into four major components

(Carr and Goldstein, 1977; Sleeman and Brown, 1982; Barr and Feigenbaum,1982]: the domain knowledge base, a collection of instructional strategies andan algorithm for applying them, a student modeler, and an interface. Since amajor goal of CIRCSIM-TUTOR is to carry on a natural language dialogue, wehave divided the interface module into three pieces, an input understander, a

6

text generator, and a screen manager. As a result, CIRCSIM-TUTOR has sevensubmodules: a domain knowledge base, a problem solver, a student modeler, aninstructional planner, an input understander, a text generator, and a screenmanager. Figure 5 shows the overall architecture of CIRCSIM-TUTOR.

Domain Knowledge Base. Anderson [1988] describes three differentcategories of knowledge encoding: the black box model, the glass box model,and the cognitive model. The cognitive model is the approach that CIRCSIM-TUTOR is attempting to implement. The domain knowledge is decomposed intomeaningful, human-like components and a causal reasoning mechanism is appliedto it, so that the system can teach the student to solve problems in a human-like manner. For a detailed discussion of this problem see Wielinga andBreuker [1990].

Domain knowledge can be divided into three different types of knowledgeto be tutored: declarative knowledge, procedural knowledge, and knowledge oftutoring heuristics. Declarative knowledge includes domain concepts and causalrelationships between them. Procedural knowledge involves the rules for usingthe concepts in solving problems. For example, in CIRCSIM-TUTOR, a rule thatfigures out the actual determinant of SV is if the primary variable is RAP,then RAP is the actual determinant of SV. Knowledge of tutoring heuristicsmust be extracted from the experience of domain experts; it involves ways ofteaching the student about the particularly difficult points in the domain.

We have built a small domain knowledge base encoded as a network offrames (see Figure 4). Each frame represents domain concepts and how theyrelate to each other causally. There are three conceptual levels in the domainknowledge; level 0 consists of the definitions and static facts, level 1consists of the cause-effect relationships between the parameters of thecardiovascular system, and level 2 contains a deeper knowledge of underlyingphysiology. The level 2 knowledge is used when the tutor needs to give a hintto the student. Currently, the level 2 knowledge is under refinement anddevelopment. Hence, in the present program the domain knowledge base isconstructed as a set of components that is used for both problem solving andcausal explanation. This is the most important and the basic knowledge thatconstitutes the domain expertise.

(frame SV

(frame-type variablevar-type physically-affectedframe-name SVclass instanceinstance-of variablename Stroke Volumedefinition volume of blood ejected each

heart beatpart-of heartanatomy ventriclecausal-relation-in causal-RAP-SV causal-CC-SVcausal-relation-out causal-SV-CO))

Figure 4. A Frame from the Domain Knowledge Base

Problem Solver. The intelligence of an ITS comes from its ability tosolve the problems (Clancey, 1987]. The problem solver solves the problems

7

presented to the student or asked by the student. If the problem solver solvesthe problems but can not explain how it solves them, it may just as wellretrieve stored answers. The ability to solve the problem, using the expert'sproblem solving behavior, can be used to identify the student'smisconceptions, to give an explanation, and to provide a basis for tutoringstrategies.

Problem solving in CIRCSIM-TUTOR is carried out by two problem solvers;the main problem solver and the subproblem solver. The main problem solversolves the problem, generates correct answers, and produces the same problemsolving path as an expert in the domain. This solution path can be used tomonitor the student's problem solving behavior while the student is makingentries in the predictions table. The subproblem solver solves currentproblems generated by the planner, such as determinant of X, relationshipbetween X and Y, and also problems coming from the student questions. Theother modules of the system may consult these problem solvers to get anyinformation they need.

.2Z Student MoeeThe student modeler is responsible for representing the student's

understanding of the subject by building a student model [VanLehn, 1988]. Thestudent model is a data structure that represents the student's current stateof knowledge; what the student knows, what the student does not know, and whatmisconceptions he or she may have. Based on this information, the tutor cangive individualized instruction to the student. There are two major approachesfor student modeling. One approach, the overlay model [Carr and Goldstein,1977], is designed to represent the student's knowledge state as a subset ofan expert's knowledge state. Another approach, the buggy model [Brown andBurton 1978], represents the student's misconceptions not as subsets of theexpert's knowledge, but as variants of the expert's knowledge. In CIRCSIM-TUTOR, the student modeler integrates overlay and buggy strategies into one[Shim et al., 1991).

The student modeler begins analyzing the student's entries in thePrediction Table. Based on this analysis, the planner generates a lesson plan.During the tutoring session, the planner sends the student's answer to themodeler and the modeler analyzes it and returns the result. Based on thisinformation, the planner can decide the next instruction. currently only theoverlay information is used for choosing the next tutoring strategy.

L.aJ TIu UnderstanderThe input understander is responsible for understanding the student's

natural language input. It handles not only well-formed but also ill-formedstudent inputs [Lee et al., 1990; Lee, 1990]. The student input may be eitheran answer to the tutor's question, or a question from the student. If thestudent's answer is The actual determinant of SV is RAP, then the planner willpass the sentence to the understander along with the current lesson topic inlogical form, (actual-determinant SV). Then the input understander parses thesentence, checks its coherence with the current topic, and returns the logicform, (answer (actual-determinant SV (RAP)). Then the planner extracts thestudent answer, RAP, and passes it to the student modeler to diagnose thestudent answer.

The input understander must also understand student initiatives; whetherthe student is asking for an explanation, or referring to the previous remarksof the tutor, or wants to stop the session. For example, if the studentinitiative is I don't understand about SV, then the input understander returnsthe logical form (question (explain SV)). This process needs to be studied indetail and we are currently investigating the student initiatives by analyzingtranscripts.

8

JA Text GeneratorThe text generator is responsible for turning the tutor's output into a

natural language sentence. It receives necessary information as a logical formfrom the planner and generates a natural language sentence or sequence ofsentences [Zhang and Evens, 1990]. This information includes the current topicand text styles: question, hint, answer, etc. For example, the text generatoris given a logic form from the planner, (question (affected-by SV ?)), then itproduces the English sentence, *What are the determinants of SV?= The textgenerator can handle this kind of simple question, explanation, oracknowledgement. The current version of the text generator only receives thenecessary information from the planner, not from all the other modules, sothat its behavior is somewhat passive.

1.a Screen ManagerThe screen manager takes care of the interaction between the student and

the system. First, the screen manager displays system messages through theintroductory windows. Then it displays the list of procedures that the studentcan select. When the student selects the problem, it displays the predictiontable with instructions about how to use the mouse and how to make entriesinto the table. Then it receives qualitative answers, (+, -, 0), from theprediction table one by one from the clicking of the mouse and passes them tothe planner. It also handles two other windows, the student window and thetutor window. From the student window, it receives the student's naturallanguage input in English sentences. In the tutor window, it displays naturallanguage sentences created and passed to it from the text generator.

J Instructional PlanerThe instructional planner is responsible for determining what to do next

at each point during a tutoring session. It interacts with the inputunderstander, the text generator, the student modeler, and the screen manager,in order to carry out tutorial activities. Although the design of the plannermay vary depending on the purpose of the ITS, several researchers haverecently proposed combining opportunistic control with a plan-based approach[Derry et al., 1988; Murray, 1990; Macmillan and Sleeman. 1987]. For instance,Murray (1990] suggests that the way to provide opportunistic control withglobal lesson plans is to implement a dynamic instructional planner. ForCIRCSIM-TUTOR, the planner needs to generate the global lesson plan and takecare of the discourse control as well [Woo et al., 1991a, 1991b].

PLANNINGj INSTRUCTIOMThe instructional planner is the central component of the ITS; it is

responsible for selecting or generating instructional goals, deciding how toteach the selected goals, monitoring and critiquing the student's behavior,and determining what to do next at each point during a tutoring session. Thatis, the planner makes two different types of important decisions during thetutoring session, decisions about the content of the lesson and decisionsabout its presentation strategy. Although the early ITSs largely focused onthe delivery strategy of the planner, some recent planning research shows theintegration of both aspects in building the planner [Macmillan et al., 1988;Derry et al., 1988; Murray, 1990].

The planning component of CIRCSIM-TUTOR must carry out both functions,since it needs to provide a global lesson plan, and it needs to carry on anatural language exchange with the student. This section discusses generaldesign issues of the planner with the goal of providing the most effectiveinstruction possible to the student, a sample dialogue extracted from thetranscript of an actual human tutor-student interaction and a scenarioimplementing that dialogue, and a description of the overall organization ofthe planner.

9

S TUDENT MODELLER 1,Epr'Beair PROBLEM SOLVERCorrect Answers

Student' sAnswers Dverlay

[KNOWLEDGE BASE ,

rurriculum

Student 'sCurrent State

Cu0 SINSTRUCTIONAL PLANNER

Student' sInput Dialogue

INPUT NDER rTEXT GENERATOR

SCREEN MANAGER

STUDENT

Figure 5. The Structure of CIRCSIM-TUTOR

- fesin •*maCapabilities of the Planner. Most machine planning systems, like STRIPS(Fikes and Nilsson, 1971], HACKER [Sussman, 1975], and NOAH (Sacerdoti, 1977],

deal with the observable physical world, whereas instructional planningsystems deal with unpredictable dynamic changes in the student's knowledge.The student's current learning status can never be observed directly. It canonly be guessed; the results of this guesswork are stored in the form of thestudent model (VanLehn, 1988]. Thus, the planner must possess uniquecapabilities for handling unpredictable situations as an expert human tutordoes.

The planner must plan at different levels of the hierarchy; ahierarchical planning technique can reduce the complexity of the planningprocess. The plan is first developed at a higher level and the details aredeveloped later; this technique prevents development of unnecessary plans in

10

advance. The planner must plan at a global level; when the planner generatesthe next instruction, it must consider the past plan and the student'sresponses to provide continuity of instruction. The planner must replan whenthe current plan fails or a request is made by the student. The planner mustbe able to monitor the plan to identify the need for replanning. The plannerof CIRCSIM-TUTOR provides all these capabilities.

Levels of Planning. Research by Leinhardt and Greeno [1986, cited inDerry et al., 1988] has shown that experienced teachers employ levels ofplanning in accomplishing their goals; planning instructional goals occurs atthe most global level, planning actions and decision making occur at a lessglobal level. Inspired by this research, Derry et al. [1988] designed theirTAPS system with three levels of instructional activity: curriculum planning(the agenda), lesson planning (instructional actions), and on-line tutorialintervention. Murray [1988] also distinguished three levels of instructionalplanning; curriculum planning (planning a sequence of lessons), lessonplanning (determining the subject matter in a single lesson), and discourseplanning (planning communicative actions between the tutor and the student).He argues that at least two levels of planning, lesson planning and discourseplanning, must exist in an ITS to deliver more effective and flexibleinstruction.

CIRCSIM-TUTOR is capable of both lesson planning and discourse planning.It can be set up so that the student can select a problem from a list of fourexperimental procedures or it can do complex curriculum planning. The numberand types of procedures will be extended further in future versions of thesystem.

_4.,Z ainaiA Sample Tutoring Session. We have recorded a number of tutoring

sessions with our experts, Joel Michael and Allen Rovick, who are Professorsof Physiology at Rush Medical College, and some of their first year medicalstudents. After careful studies of the recorded transcripts, we extracted somepossible tutorial strategies and tactics that provided us with the frameworkfor building the instructional planner and the overall system. It is assumedthat students have already learned much of the domain knowledge, hence thesystem will mainly assist the students to correct their misconceptions and tosolve problems. Our current system can handle dialogues like the following.

Example 1:Tutor> What are the determinants of SV?Student> SV is determined by RAP and CO.Tutor> RAP is correct, but CO is not a determinant of

SV. Remember. SV is the amount of blood pumpedper beat. what is the other determinant of SV?

One important point about the above tutor-student interaction is thecontent of the questions posed by the tutor. For example, on the first line ofthe excerpt, the tutor is asking the student about the determinants of strokevolume. Asking a question about determinants is the first part of the planthat the tutor is using to teach the student about the causal relationshipsbetween two variables, RAP and SV. Thus the content of the question has to begenerated by the lesson planner before the tutoring begins. Another importantaspect is how to present the selected topic. From the above short excerpt, wecan see four different kinds of delivery modes: a direct question (line 1),positive and negative acknowledgements (line 3), hints (line 4), and follow upquestions after hints (line 5). Thus, the planner (discourse planner) needs toplan how to present the selected content to the student effectively.

Example 2:Tutor> By what mechanism is TPR controlled?Student> Nervous System.Tutor> Correct, TPR is controlled by the nervous system.

11

Then what is the correct prediction of TPR?Student> No change.

Example 2 is an another tutoring situation that focusses on one of theneurally controlled variables, TPR. The tutor first asks the student about itscontrol mechanism in line 1. This control mechanism is the first strategy toteach the student about the neurally controlled variables. Since the studentanswered correctly, the tutor gives a positive acknowledgement and then usesits second strategy, asking for a prediction, in line 4. We have extractedthis kind of tutoring strategy from the transcripts and designed explicitlesson planning rules.

From the above examples of tutor-student interaction, we candistinguish between the subject matter and its presentation formats. Ohlsson[1986, p. 217] argues that an effective ITS should be able to generatedifferent presentations of each piece of subject matter in order to provideadaptive instruction to the student. The content of the questions posed by thetutor and its delivery modes lead to the development of two different kinds ofinstructional planning, lesson planning and discourse planning, because thesubject to be taught has to be generated adaptively, and also its presentationform can vary according to the situation.

Implementation of the Scenario. Assume that the current lesson goal isto tutor the causal relationships between two parameters, RAP and SV. Thisgoal gets refined into a set of hierarchical subgoals by using strategic andtactical rules. The subgoals generated at the tactical level, such asdeterminants, actual determinant, relation, and value, are kept in a stack,which is used by the discourse planner to pick the next topic.

The following scenario describes what each component of the system does,what kind of information it needs, and what is the result after each step. Thesteps are numbered to show the execution sequence. This tutorial interactionbegins after the lesson planning is done. So that the discourse planner beginswith the first topic in the stack, the determinants, and when that topic iscompleted, continues with the next topic, the actual determinant, and so on.

1. Planner: Picks the current topic from the stack,selects the discourse tactic, and passes it to thetext generator as an internal logical form.

current topic: (determinant SV),discourse tactic: question.call Text Generator: (question (determinant SV))

2. Text Generator: Generates a sentence,"What are the determinants of SV?"

3. Screen Manager: Displays the sentence in the window.

4. STUDENT: "SV is determined by RAP and CO."

5. Planner: Passes the student's input with the currentlesson topic to the input understander.

(question (determinant Sv)(SV is determined by RAP and CO))

6. Input Understander: Parses the student's answer, checksits coherence with the dialog history, and returnsthe answer to the planner in logic form.

call planner: (answer ((determinant SV)(RAP CO)))

12

7. Planner: Passes the current topic and student answerto the the student modeler in logic form.

current topic: (determinant SV),student answer: (RAP, CO),call Student Modeler: ((determinant SV) (RAP, CO))

8. Student Modeler: Calls the problem solver, gets thecorrect answer: (RAP, CC), compares the correctanswer with the student answer, and updates thestudent model.

In step 1, the discourse planner picks the topic, determinant, from thesubgoal stack, selects the discourse tactic, question, binds these twotogether with the current variable, SV, into a logical form, (question(determinant SV)), which is passed to the text generator to generate a naturallanguage sentence. After receiving the logical form from the planner, the textgenerator generates a sentence like the one in step 2. In step 3, the screenmanager displays the sentence on the student window, and the student respondswith the answer in step 4. So the current dialogue is:

Tutor> What are the determinants of SV?Student> RAP and CO.

In step 5, the planner passes the student's input along with the currenttopic. The input understander has to recognize the student's answer; parse theanswer, check its coherence with the dialogue history, and return the answerto the planner in its logical form. Then the planner sends the current topicwith the student's answer to the student modeler in step 7. Finally, thestudent modeler analyzes the student's answer, and records the result in thestudent model. The next step will start with the planner checking the studentmodel, and then deciding what to do. Since one of the student's answers iswrong, the planner consults its tutoring rules and decides to give someacknowledgement first:

Tutor> RAP is correct, but CO is not a determinant of SV.

At this point the tutor has two choices, either give a hint or just :ivean answer and continue with the next topic. since this is the first trial, Chetutor decides to "give a hint" and then ask a question to complete '.heprevious answer. So a possible response would be:

Tutor> Remember. SV is the amount of blood pumped per beat.What is the other determinant of SV?

A different tutoring rule will be applied if the student again makes anerror after receiving a hint; the student will be given a direct answer forthe second question. Our current tutoring rules vary according to the topicand the student's responses (i.e., the tutor gives different responses indifferent situations). The question may be about neural variables or causalrelationships; in each case the tutoring rules are different. Also we havedifferent rules for each stage, DR, RR and SS.

43Organization 21 the Inatructional la~nnayThe instructional planner of CIRCSIM-TUTOR consists of two parts (see

Figure 6); the lesson planner and the discourse planner. The plan controllermonitors the execution of the current plan. The planner can be thought of as asmall expert system, which consists of two main parts: a knowledge base and an

13

inference engine (Harmon, 1987]. Thus, the lesson planner consists of threesets of lesson planning rules, and an inference mechanism. The discourseplanner consists of four sets of discourse planning rules and an inferencemechanism. This section introduces the organization and the main features ofthe instructional planner briefly.

INSTRUCTIONAL PLANNER

LESSONPLANNER:

Level 1: Goal GenerationJGal

Plan GenerationPLAN

Level 2: StrategyCONTROLLERLevel 3: Tactic

I Lesson Plan

DISCOURSEPLANNER: DISCOURSE NETWORK

Level 4: DiscoursePedagogic State

[ Level 5: DiscourseTactical State

STUDENT

Figure 6. Instructional Planner

Lesson Planning. Lesson planning determines the content and sequence ofthe subject matter to be taught in a single lesson [Murray, 1988; Brecht etal., 1989; Russell, 1988]. The lesson planning in CIRCSIM-TUTOR consists oftwo phases: goal generation and plan generation. The generation of the lesson

14

goals is guided by a set of explicit domain-dependent heuristics (goalgeneration rules), and the lesson plans are determined by applying two set ofrules, rules for selecting strategies and rules for selecting tactics. As aresult the lesson planner does hierarchical lesson planning with its threesets of rules; at the topmost level it generates lesson goals, and then itexpands one of the goals into a set of subgoals (a plan) at the next level.The generated goals will be saved in the goal stack and the subgoals in thesubgoal stack. The lesson planner must update the goals dynamically as thestudent model changes.

Discourse Planning. Discourse planning is a mechanism for planningcommunicative actions between the tutor and the student within a lesson[Woolf, 1984; Winkels et al., 1988]. CIRCSIM-TUTOR communicates with thestudent in natural language. Thus, the discourse planner must interact withthe student modeler, the screen manager, the input understander, and the textgenerator using a flexible control mechanism. This control mechanism residesin its discourse network.

The network consists of two levels; the top level of the networkspecifies pedagogic decisions and the lower level consists of a set ofdiscourse tactical states, the execution of which causes text generation,student model updates, and moves to the other states. It represents thediscourse planning rules and the control mechanism in explicit form. The rulesinclude all the necessary information to carry out the discourse with thestudent, and the control mechanism is also specified within the rules; twosets of default rules manage the fixed control flow, and two sets of metarules handle dynamic control flow.

Plan Monitoring. AI research on planning emphasizes that execution of aplan requires some monitoring (Charniak and McDermott, 1986]. In the rece-trobot planning systems (Wilkins, 1988; Swartout, 1988], the plan monitoring isdone by inserting monitoring steps in the plan, which behaves like a studentmodel in instructional planning. In an ITS, since the student's learningstatus is unpredictable, the planner also needs to monitor the execution ofthe plan and revise the plan if necessary. As a result, plan monitoring shouldoccur whenever there is a change in the student model. Plan revision may occurwhen the current plan is completed or when the student takes the initiative.

For the current version of CIRCSIM-TUTOR, the planner monitors thestudent problem solving in two different places. First, when the studententers predictions in the prediction table, the planner monitors the student'sentries in the table and interrupts with some warning messages if the studentviolates the system constraints. The messages are designed by the experts, tohelp the students in their problem solving. The system gives differentmessages depending on the procedure, the variables, and the stages. Second,the planner monitors the student answer at each step during the tutoringsession, by referring to the student model, and then decides what to say next;give a hint, give an answer, or continue with the next topic, etc. When thestudent takes control by asking a question during the tutoring session, theplanner suspends the current plan, carries out the student's request and thenresumes the suspended plan.

= TLESSON ZLHThe lesson planner decides on the contents of a lesson, based on the

student's current knowledge about the domain. The planner has to generatelesson goals, sequence the goals, and select the appropriate planningstrategies to create a plan for the current lesson goal. Figure 7 shows thearchitecture of the lesson planner including the necessary planning steps,student model, and lesson planning rules. The result of the lesson planning isa set of subgoals (a plan), each of which will be the topic for a dialoguewith the student.

15

The lesson planning mechanism is an essential component of theinstructional planner, since the system must generate globally coherent andconsisten instruction for the student (Macmillan et al., 1988; Murray, 1990],in such a way that the topics are logically connected wit each other, andsequenced and presented in a manner sensitive to the tutorial goals and thestudent's needs. This section describes the lesson planner: lesson planningrules, an architecture, and its two main mechanisms (i.e., goal generation andplan generation).

LESSON PLANNER

Plan GenerationGoal Generation

Goal Generation Strategy 4-Student Model

Rules Goal 'IL-Strategic Rules

Student Model Tactics 4-Tactical Rules

aLesson Plan

Figure 7. Structure of the Lesson Planner

5L_ Lesson Plan g RulesThe contents of the tutoring strategies are extracted from the

transcripts of the human tutor and student interaction, and we need to encodethem explicitly in the program as rules. I designed this part as a productionsystem, which consists of a rule interpreter and a set of rules. This is themost common approach used in expert systems (Hasemer and Dominque, 1989]. Thissubsection, describes the design of the rules and the implementation of therule interpreter to parse the rules.

The Rule Interpreter. The rule interpreter consists of three main parts:its main loop, its working memory, and its pattern matcher. The working memoryis crucial to the operation of the rule interpreter, because the workingmemory holds an initial representation of the problem that the system istrying to solve. Each time around the loop, the contents of the working memorywill be compared to the antecedent of the rules, and then will fire only onerule if it matches. If an antecedent matches with the working memory, theconsequent will be executed, and the content of the working memory will bechanged for the next inference. The matching cycle will continue until norules match. At this point the interpreter halts, and the content of theworking memory is the desired result for the given problem.

The interpreter is built using LISP macro functions, which understandand interpret the rules for the system. The rule format consists of threeparts: the name part of the rule, the antecedent part, and the consequentpart. For example, (Rulename: (antecedent) => (consequent)). This approachmakes the system efficient in representing the rules explicitly.

How to Encode the Lesson Planning Rules. The lesson planner uses threesets of lesson planning rules (goal generation rules, strategy rules, andtactical rules). The general form in which the rules are written is if X thenY. Here X is the antecedent or left-hand side of the rule and Y the consequentpart or right-hand side of the rule. Both the antecedent and the consequentmay contain one or more terms.

For example, assume that the student made an error in predicting thevariable TPR. One of the goal generation rules applies; if the student doesnot know TPR, then build the lesson goal, tutor TPR about the neural control.

16

This rule can be expressed as (GRulel: ((do-not-know TPR) => (neural-controlTPR))). If the current lesson goal is teach the causal relationship betweenRAP and SV, and the student does not know the direction, then this rule can bewritten as (SRulel: ((causal-relation)(do-not-know direction)) => (tutor-causality))). This is the strategy rule for dealing with non-neural variables.If the strategy rule is tutor-causality, then the corresponding tactical ruleis to teach determinants, actual-determinant, relation, and value. This rulecan be written as (T_Rulel: ((tutor-causality) => (determinants) (actual-determinant) (relation) (value))).

Currently, there are about 50 goal generation rules, 20 strategy rules,and 20 tactical rules that handle DR, RR and SS phases, and for procedures 4,6 and 7. The rules may need to be extended to handle the other procedures.

5,Z Lesson PlnnnInstructional planning centers around instructional goals. The lesson

planning generates the lesson goals, the knowledge that the system intends thestudent to acquire through the tutoring session. This subsection describes howto generate the lesson goals, and how to develop a lesson plan for the each ofthe goals. The two main mechanisms of the lesson planning process, goalgeneration and plan generation, are explained below.

Goal Generation. CIRCSIM-TUTOR generates instructional goals based onthe student's knowledge demonstrated as entries in the Prediction Table. Thegeneration of the goals is guided by a set of explicit goal generation rulesdesigned by our experts (Joel Michael and Allen Rovick), which ensures thatthe most serious misconception is selected and tutored first.

Goal Generation Rules

1. IF Current Primary Variable is CC and

Student Answer is not NOCHANGE for TPR

Then Build Lesson Goal (NEURAL-CONTROL (TPR))

2. IF Current Primary Variable is RAP and

Student does not know the CAUSAL-RELATIONSHIP

between RAP and SV

Then Build Lesson Goal (CAUSAL-RELATION (RAP, SV))

3. IF Current Primary Variable is RAP and

Student does not know the CAUSAL-RELATIONSHIP

between SV and CO

Then Build Lesson Goal (CAUSAL-RELATION (SV, CO))

Figure 8. Goal Generation Rules

For example, suppose the student made wrong predictions in the table forthe variables, TPR and SV. The student modeler has determined, from itsanalysis, that the student is confused about the mechanism controlling TPR and

17

the causal relationships between RAP and SV and SV and CO. So the lessonplanner retrieves the information from the student model, applies the goalgeneration rules (see Figure 8), and generates the lesson goals dynamically.The result is a set of lesson goals in the goal stack (see Figure 9).

Order Lesson Goals

1. NEURAL-CONTROL (TPR)

2. CAUSAL-RELATION (RAP,SV)

3. CAUSAL-RELATION (SV, CO)

Figure 9. Generated Lesson Goals in the Goalstack

The goal generation is significant in many ways; the goals are generateddynamically and adaptively; the goals are sequenced in the order that theexpert tutors this material; the goals provide a global context that remainscoherent and consistent throughout the tutoring session, unless the goals arerevised. New goals can also be generated, which tutor the student about acommon misconception (a bug), if the student modeler detects such amisconception. The goals remain in force until they are changed by the plannerdynamically.

Strategic Rule

1. If the Goal = CAUSAL-RELATION and

Student does not know and

direction is incorrect

Then Strategy = TUTOR-CAUSALITY

2. If the Goal = CAUSAL-RELATION and

Student does not know and

direction is correct

Then Strategy = REMIND-RELATION

3. If the Goal = NEURAL-CONTROL and

this is the first procedure

Then Strategy = DEFINE-TUTOR-NEURAL

Figure 10. The Strategy Rules

Plan Generation. The second stage of the lesson planning is the plangeneration mechanism, which creates the instructional plan by applying twosets of rules, rules for selecting tutorial strategies to achieve the selectedgoal and rules for selecting pedagogic tactics to execute those strategies.

18

Strategy rules (Figure 10) describe the tutorial approach from a domain-independent point of view. These include tutoring prerequisites before thematerial they underlie, reminding the student about relations between twoparameters, explaining the definition before tutoring about it, and so on.Tactical rules (Figure 11) also represent a domain-independent tutorialapproach; they involve asking about concepts and relations between theconcepts.

Tactical Rule

1. If Strategy = TUTOR-CAUSALITY

Then Tactic = DETERMINANTS,ACTUAL-DETERMINANT,RELATIONSHIP,VALUE

2. If Strategy = TUTOR-NEURAL-CONTROL

Then Tactic = MECHANISM,VALUE

3. If Strategy = TUTOR-SS-PHYSICAL-VARIABLE

Then Tactic = VALUE-DR,VALUE-RR,VALUE-SS

Figure 11. The Tactical Rules

For instance, if the goal is teach the causal relationship between thetwo parameters, then the fired strategy rule is tutor the causality, and thisthen fires the pedagogic tactical rule: ask about: determinants, actualdeterminant, relationship, and correct value. The result is a hierarchicalgoal tree (Figure 12).

Goal: CAUSAL-RELATION (RAP, SV)

Strategy: Tutor Causality

1 24

Tactic: Determinants Actual Relationship ValueDeterminant

Figure 12. Generated Plan for "causal relation(RAP,SV)"

19

Thus the current goal is ultimately refined into four subgoals by two-step goal transformations. In order to solve the current goal, all thesubgoals must be solved. This is the well-known AI problem-reductiontechnique, which transforms a goal into a set of immediate subproblems by asequence of transformations (Barr and Feigenbaum, 1982]. The four subgoalsgenerated at the tactical level are the current plan for the goal. These arekept in a subgoal stack (Figure 13), which is used by the discourse planner topick the next topic.

Order Subgoals

1. Determinants

2. Actual-determinant

3. Relation

4. Value

Figure 13. The Subgoal Stack

Figure 14 shows an example of the lesson planning process for thecausal-relationship between RAP and SV. From the top of the Figure, the goalgeneration step is described with its other information: student model, rulesused, goal stack, and current goal. Then the plan generation step is describedin two steps, the strategic and the tactical steps. The lesson planner waitsfor the discourse planner to complete the current lesson plan, and when theplan controller sends a wake-up signal, then the planner gets reactivated andcontinues with the next goal in the goal stack.

=. 2LjCOU" PLANNPRThe discourse planner is responsible for controlling interactions

between the tutor and the student. It needs to decide how the tutor shouldrespond to a student with a given problem (Woolf, 1984; Winkels et al., 1988].This discourse strategy must be planned explicitly by the discourse planner,so that the system can enter into flexible and coherent interactions. InCIRCSIM-TUTOR, the discourse planner is combined with the lesson planner, sothat the discourse planner receives a global lesson plan from the lessonplanner. The plan controller monitors the execution of the plan and forces thediscourse planner to suspend or resume the current plan when the student takescontrol. The planner consists of sets of discourse planning rules and a twolevel discourse network.

j Architecture 2o the 2jag2= PlannerFlow Chart Approach. Meta knowledge is knowledge about knowledge (Davis

and Buchanan, 1987]; what you know and don't know (operational metaknowledge), and how you do things (control meta knowledge). The operationalmeta knowledge is needed to recognize a question outside the limits of thesystem. It can be ignored in the discourse planner, since the inputunderstander receives such a question or answer and responds with I don'tunderstand, please rephrase. The control meta knowledge determines how thesystem interacts with the student; it is based on our observations about howthe human expert tutors the student. The integration of this knowledge intothe system ensures that it appears to ask questions in a logical order.

20

Goal Generation

Rules Used:

Student Model: do-not-know (SV) DR_G_Rule8

Goal Stack: Causal-relation (RAP, SV)Causal-relation (SV, CO)

Current Goal: Causal-relation (RAP, SV)

Plan GenerationRules Used:

Strategy: Tutor-causality DR_S_Rulel

Tactics: (determinants) DRTRule6(actual-determinant)(relation)(value)

Subgoal Stack: (determinants)(actual-determinant)(Plan) (relation)(value)

Discourse Planner

executes "determinants of SV"

Plan Monitoring: Waits for the student response

Figure 14. An Example of Lesson Planning

The basic representation of the control meta knowledge in CIRCSIX-TUTORis the flow chart. This is a model of what the expert does and when he doesit. For our system, Allen Rovick designed several flow charts (see Figure 15),each of which is used for tutoring the student in a different situation. Weneed different tutoring strategies for handling different variables anddifferent phases (DR/RR/SS).

21

Tutoring DR-Non-Neural Variables

N So Give hint from lev 1 2

(1 determinant) 2nd N -. Give answer {

What are the ideterminants4- 1(2 determinantsy

1Y,1iN answer iSelGive hint f rom J(2 det rmnns\level 2

Yes 2N answers ------- oGive answer

N If no equation, give hintV 1-f rom level 2

What is 4 4relationshi N If equation, state in words

Yes 2nd N----*Give relationship---.

4

Predict again '**bN -m*Gi entire level 2

YesStill N---Sosend to textbook

Next error

Figure 15. The Flow Chart for Tutoring Non-Neural Variablesin DR

Figure 15 is the one that tutors the non-neural variables in DR. The contentof the questions is determined by the lesson planner and passed on to thediscourse planner, which must then decide how to express this content,determining whether to ask a question, give an answer, and so on. After thechart was created, I encoded this information as discourse planning rules. The

22

next step is to create a sophisticated inference mechanism that can utilizethese rules.

Discourse Network. The network is the main knowledge structure of thediscourse planner. It consists of states, links, and arcs (see Figure 16). Thestates represent tutorial actions, the arcs imply state transitions, and thelinks indicate hierarchical dependencies; a state at the tactical levelrepresents the refinements of the level above. Three important mechanisms needto be discussed: levels of planning, representation of the tutorial states,and control structures.

Begin Finish

INTRODUCE TT MLT

emind romplete Topiq

Fxplanation ivAns

sk_Question ive CorrectA-k

equestion UGiveIncorrectAcvaluate_Inpu

ive Half Correct Ack

Figure 16. The Discourse Network

A. Levels of Planning. The discourse planner is divided into twoplanning levels: pedagogical and tactical. The pedagogical level makesdecisions about the style of tutoring; introduces a topic, remediates thestudent's misconceptions, and completes a topic. The discourse action beginswith the pedagogical level, introduce state, and then it traverses the networkand finishes one topic as it reaches the complete state. The tactical levelchooses an expository style to implement the pedagogy; question the student,give acknowledgement, or give an answer. The states at this level arerefinements of the states at the pedagogic level.

B. Representation of Discourse Strategies. The second importantmechanism is the representation of the tutorial strategies in the form ofstates. The discourse strategies were then extracted from the flow chart andexpressed as discourse rules. The rules are written as a frame-like structureusing Lisp macro functions, which represent the states in the network. Thestates are divided into default states and meta states, and each state isfurther divided into pedagogic and tactical states. Each state consists of astate name and slots. The slots in the default state contain information about

23

tutoring strategy, text style, and explicit control. The meta states mainlyinclude explicit control mechanism and preconditions. In Figure 19, theexecution of the AskQuestion state will cause the text generator to generatea question, and then move on to the next default state, Eval_Input. The slotsalso contain a register to keep track of the completion of the topic, and aflag to update the student model.

C. Control Structure. The discourse control in the network can bedivided into a default control structure and a meta control structure. Thedefault control is specified in the default states, so that the tutor movesfrom one state to another according to a pre-determined path. The meta controlabandons the default path and moves to the state that is specified in themeta-rule. The system checks the meta-rules first and if none of the meta-rules fire, then the control flow will follow the default path. This controlpath is hidden in Figure 16, because the exceptional behavior by the meta-rules can not be predicted in advance. For example, the Eval Input state willbe selected right after the Ask Question state as a default path, but the nextstate is unpredictable, since the student answer could be correct, wrong, orpartially correct. This mechanism enables the dynamic behavior of thediscourse planner.

Pedagogic Default Rule

(Pedagogicdefault *introduce*(subgoal current-taskupdate topic-completednext-state *tutor*))

(Pedagogicdefault *tutor*(subgoal current-taskupdate topic-completednext-state *complete*))

Figure 17. The Pedagogic Default Rule

Pedagogic Meta Rule

(Pedagogicmeta *m tutor*(precondition topic-completedprior-state *tutor*next-state *introduce*))

(Pedagogic meta *m-complete*(precondition no-more-topicsprior-state (*introduce* *complete*)next-state *stop*))

Figure 18. Some Pedagogical Meta Rules

The main disadvantage of earlier discourse management networks (Woolf,1984; Clancey, 1982] is that they needed to be coupled with some other controlmechanism, such as an agenda and an external memory to provide a topic. In

24

CIRCSIM-TUTOR, since the lesson planner provides a globally coherent lessonplan, the network itself can function solely for delivery purposes whilekeeping all the advantages of the discourse management network, such asflexible discourse control and explicit representation of discoursestrategies.

Tactical Default Rule

(Tactical default *ask_question*(text-style questioncontent current-taskupdate nilnext-state *eval-input*))

(Tactical default *giveanswer*(text-style give-answercontent (current-task correct-answer)update student-modelnext-state *complete-topic*))

Figure 19. Tactical Default Rules

Tactical Meta Rule

(Tacticalmeta *m correct*(precondition correct-responseprior-state *eval-input*next-state *correct-ack*))

(Tacticalmeta *m incorrect*(precondition incorrect-responseprior-state *eval-input*next-state *incorrect-ack*))

Figure 20. Tactical Meta Rules

Discourse planning in CIRCSIM-TUTOR is managed by a simple algorithm. Ititerates through the states until a topic becomes complete. Either the studentresponds with a correct answer or the tutor gives the answer. This sectiondescribes important features of the discourse planning.

The Discourse Goal. The discourse planner needs a goal to tutor thestudent. This goal can be found in the subgoal stack, which the lesson plannerhas produced. In Figure 13, the subgoals are sequenced by number, so that thediscourse planner can carry them out in that order. When the planner finishescarrying out one of the subgoals, it will be removed from the stack, and theplanner picks the next one. This cycle continues until the stack is empty, or

25

is suspended by the plan controller in favor of a student initiative (Woo,1991c].

Generating Natural Language Sentence. The tactical default states haveslots containing information for the text generator. When the plannerprocesses the states, the text-style and content slots will be extracted fromthe current state. For example, assume that the planner is processing the*askquestion* state (Figure 19), while the text-style slot contains questionand the content slot contains the current-task, such as determinant (SV).Binding these two slot values provides us with a logic form, (question(determinant (SV))), which will be passed to the text generator, whichgenerates the sentence, What are the determinants of SV? Then the screenmanager will display the sentence in the tutor window.

The logic form may need to be extended to generate richer sentences,since this kind of the logic form only contains information about a particulartask or the solution of a problem. The text generator may need to collect moreinformation from many other sources, the domain knowledge base, the studentmodel, the dialog history, and so on.

How to Recognize a Student Initiative. CIRCSIM-TUTOR allows studentinitiatives during the tutoring session. So the planner must understandwhether the student response is a question or an answer by checking the inputlogic form, which is being passed from the input understander. For example, ifthe input understander passes a logic form, (answer (determinant SV)(RAP CO)),the first item of the list, answer, indicates that this is an answer. Thesecond item of the list, (determinant SV), is the current topic, and the thirditem, (RAP CO), is the student answer. Let's assume that the tutor asks thequestion, What are the determinants of SV? and the student responds with Idon't know about SV. Then the input understander recognizes this as animplicit question and returns a logic form, (question (do-not-know) (SV)). Theplanner receives the logic form and recognizes that this is a studentinitiative, so it suspends the current plan and carries out the studentrequest; asks the problem solver to get the definition of SV from theknowledge base, and then asks the screen manager to display it.

§j aTrace 2o jgaitiolFigure 21 shows a short trace of a sequence of discourse transitions.

The short arrows represent the pedagogic level transitions; the long arrowsrepresent the tactical level transitions; and the double arrows represent themeta level transitions. The left side of the figure shows the processing ofstates, and the right side of the figure shows the discourse actions resultingfrom visiting the states.

The tutor begins by asking a question, then it moves to the evaluatestate by the default control rule. At this time, the student responds with ahalf correct answer, which is recognized by the meta tactical rule3, whichforces a move to the half-correct state. This state produces anacknowledgement and then another meta rule fires, which recognizes that thisis the first try. So the meta rule forces a move to the give-hint state, whichproduces a hint. Since there is no default and a meta rule applies, thecontrol pops up to the upper level and checks whether the topic has beencompleted. If not, then control goes back to the introduce state again, andmoves down to the tactical level. This time the requestion state is selected,since this is the second try on the same topic.

7. CONCLUSION

7.1 Significant FeaturesThis paper describes the design and development of an instructional

planner for a Physiology ITS, CIRCSIM-TUTOR. The planner has severalsignificant features.

26

Current Topic: Determinants of SV

Pedagogic Level

-- >, ==>: Tactical Level ( -> Default , => Meta

Discourse States Discourse Action

-> INTRODUCE

-- > Ask-question Tutor: What are the determinantsof SV?

-- > Eval-input Student: RAP and CO

==> Meta-tactic3(Incorrect-one)

-- > Half-Correct Tutor: RAP is correct, but CO isnot a determinant of SV.

==> Meta-tactic6(First-try)

-- > Give-hint Tutor: Remember. SV is the amountof blood pumped per beat.

-> TUTOR

=> Meta-pedagogic(Not-completed)

-> INTRODUCE

-- > Requestion Tutor: What is the otherdeterminant of SV?

Figure 21. Trace of the Discourse Transition Process

First, the planner combines two different instructional planningapproaches: lesson planning and discourse planning. Lesson planning producesglobal lesson plans, which will be carried out during the discourse planningstage. Second, the planner plans dynamically based on the inferred studentmodel; it generates plans, monitors the execution of the plans, and replanawhen the student interrupts with a question during the tutoring session.Third, the pedagogical knowledge is extracted from the experts and representedexplicitly as rules, lesson planning rules and discourse planning rules. Therules are used to generate lesson plans and to control discourse strategies.The system interprets the rules and builds the lesson plans or returns anappropriate discourse action. Fourth, the planner plans at different levels ofthe hierarchy; the higher level is a simplification or abstraction of the plan(lesson goals) and the lower level is a detailed plan (subgoals), sufficient

27

to solve the problem. Fifth, the planner allows minimal student initiativesduring the tutoring session. If the student asks a question the plannersuspends the current plan, carries out the student request, and then resumesthe suspended plan.

Since one of the main goals of CIRCSIi-TUTOR is to provide a naturallanguage interface, the discourse planner is designed not only to providesophisticated discourse control, but also to create the internal logic formsfor the text generator to generate the sentence. A short tutoring scenario isintroduced, which came from a transcript of human tutor and studentinteraction, to explain the internal process of the system.

7.2 Future ResearchThe current version of the student model is limited to the overlay

strategy, so the planner can support tutoring on the overlay errors only, notthe bugs. The tutoring strategy for the bug library has not been developedyet, so the system cannot tutor the student about bugs at the moment.

Another important tutoring strategy is giving a more detailed level hintduring the tutoring session. Also it needs more anticipation from othermodules; the domain knowledge base needs much more detailed knowledge, theinput understander and the text generator need to expand their lexicon andlogic forms to contain all the variables at the detailed level, the problemsolver needs to be able to access the knowledge base and extract a hint, andthe planner needs to have a general strategy for deciding the content of thehint for every situation during the tutoring session.

CIRCSIM-TUTOR supports four pre-determined problems as a curriculum, sothat it does not really require curriculum planning. Our expert tutors aredeveloping many more procedures for the system, which may requiresophisticated curriculum planning in future versions of the system.

L& R.L0MA

Anderson, J.R. 1988. The Expert Module. In: Polson, M. C. and Richardson,J.J., Eds., Foundations of Intelligent Tutoring Systems, LawrenceErlbaum Associates Publishers, Hillsdale, New Jersey, 21-53.

Barr, A. and Feigenbaum, E.A. 1982. The Handbook of Artificial Intelligence.William Kaufmann, Inc., Los Altos, California, 2:225-294.

Brecht, B., McCalla, G., Greer, J., and Jones, M. 1989. Planning the Contentof Instruction. Proceedings of the 4th International Conference on Aland Education, Amsterdam, Netherlands, 32-41.

Brown, J.S. and Burton, R.R. 1978. Diagnostic .Models for Procedural Bugs inBasic Mathematical skills. Cognitive Science, 2:155-191.

Carr, B. and Goldstein, I.P. 1977. Overlays: a Theory of Modelling forComputer Aided Instruction. AI Memo 406, AI Laboratory, MassachusettsInstitute of Technology.

Charniak, E. and McDermott, D. 1986. Introduction to Artificial Intelligence,Addison-Wesley, Reading, Massachusetts.

Clancey, W.J. 1982. Tutoring Rules for Guiding a Case Methods Dialogue. In:Sleeman, D.H. and Brown, J.S., Eds., Intelligent Tutoring Systems,Academic Press, New York, 201-225.

Clancey, W.J. 1987. Intelligent Tutoring Systems: A Tutorial Survey. ReportNo. STAN-CS-87-1174.

Davis, R. and Buchanan, B. G. 1985. Meta-Level Knowledge. In: Buchanan, B. G.and Shortliffe, E. H., Eds., Rule-Based Expert Systems. Addison-Wesley,Reading, Massachusetts, 507-530.

deKleer, J. and Brown, J.S. 1984. A Physics Based on Confluences. ArtificialIntelligence, 24:7-83.

28

Dede, C.J. 1986. A Review and Synthesis of Recent Research in IntelligentComputer-Assisted Instruction. International Journal of Man-MachineStudies, 24:329-353.

Derry, S.J., Hawkes, L.W. and Ziegler,U. 1988. A Plan-Based OpportunisticArchitecture for Intelligent Tutoring. Proceedings of IntelligentTutoring Systems. ITS-88, Montreal, Quebec, Canada, 116-123.

Elsom-Cook, M. 1988. Using Multiple Teaching Strategies in an ITS. Proceedingsof Intelligent Tutoring Systems: ITS-88, Montreal, Quebec, Canada, 286-290.

Fikes, R. and Nilsson, N.J. 1971. STRIPS: A New Approach to the Application ofTheorem Proving to Problem Solving. Artificial Intelligence, 2:189-208.

Forbus, K.D. 1984. Qualitative Process Theory. Artificial Intelligence, 24:85-168.

Harmon, P. 1987. Intelligent Job Aids: How AI will Change Training in the NextFive Years. In: Kearsley, G.P., Ed., Artificial Intelligence andInstruction, Addison-Wesley, Reading, Massachusetts, 165-190.

Hasemer, T. and Dominque, J. 1989. Common Lisp Programming for ArtificialIntelligence, Addison-Wesley, Reading, Massachusetts.

Kearsley, G. 1987. Artificial Intelligence and Instruction: Application andMethods, Addison-Wesley, Reading, Massachusetts.

Kim, N., Evens, M., Michael, J.A., and Rovick, A.A. 1989. CIRCSIM-TUTOR: AnIntelligent Tutoring System for Circulatory Physiology. In: Mauer, H.,Ed., Computer Assisted Learning, Proceedings of the InternationalConference on Computer Assisted Learning, Dallas, Texas, Springer-Verlag, Berlin:254-266.

Kim, N. 1989. CIRCSIM-TUTOR: An Intelligent Tutoring System for CirculatoryPhysiology. PhD. Dissertation, Department of Computer Science, IllinoisInstitute of Technology, Chicago, Illinois.

Kuipers, B. 1984. Commonsense Reasoning about Causality: Deriving Behaviorfrom Structure. Artificial Intelligence,_24:169-203.

Lee, Y., Evens, M., Michael, J.A., and Rovick, A.A. 1990.IFIHS: Ill-FormedInput Handling System. Proceedings of the Second Midwest ArtificialIntelligence and Cognitive Science Conference. Carbondale, IL, 93-97.

Lee, Y. 1990. Handling Ill-Formed Natural Language Input for an IntelligentTutoring System. Ph.D. Dissertation, Computer Science Department,Illinois Institute of Technology, Chicago, Illinois.

Leinhardt, G. and Greeno, J.G. 1986. The Cognitive Skill of Teaching. Journalof Educational Psychology, 78:75-95.

Macmillan, S.A. and Sleeman, D.H. 1987. An Architecture for a Self-ImprovingInstructional Planner for Intelligent Tutoring Systems. ComputationalIntelligence, 3:17-27.

Macmillan, S.A., Emme, D. and Berkowitz, M. 1988. InstructionalPlanners:Lessons Learned, In: Psotka, J., Massey, L.D., and Mutter,S.A., Eds., Intelligent Tutoring Systems: Lessons Learned, LawrenceErlbaum Publishers, Hillsdale, New Jersey, 229-256.

Michael, J. A., Rovick, A. A., Evens, M., and Kim, N. 1990. A Smart TutorBased on a Qualitative Causal Model. Proceedings of the AAAI SpringSymposium on Knowledge-Based Environments for Learning and Teaching,Stanford, March 27-29, 112-117.

Michael, J.A., Rovick, A.A., Evens, M., Shim, L., Woo, C., and Kim, N. 1991.The Uses of Multiple Student Inputs in Modeling and Lesson Planning inCAI and ICAI Programs. Submitted to the International Conference onComputer Assisted Learning, Nova Scotia, CANADA.

Murray, W.R. 1988. Control for Intelligent Tutoring Systems: A Comparison ofBlackboard Architecture and Discourse Management Networks. ResearchReport R-6267, FMC Corporation, Santa Clara, California.

Murray, W.R. 1990. A Blackboard-based Dynamic Instructional Planner. ResearchReport No. R-6376, FiC Corporation, Santa Clara, California.

29

Ohlsson, S. 1986. Some Principles of Intelligent tutoring. In: Lawler, R. andYazdani, M., Eds., Artificial Intelligence and Education: LearningEnvironment and Tutoring Systems,_Ablex Publishing, Norwood, New Jersey,204-237.

Peachey, D.R. and McCalla, G.I. 1986. Using Planning Techniques in IntelligentTutoring Systems. International Journal of Man-Machine Studies, 24:77-88.

Rovick, A. A. and Brenner, L. 1983. HEARTSIM: A Cardiovascular Simulation withDidactic Feedback. The Physiologist, 26:236-239.

Rovick, A.A. and Michael, J.A. 1986. CIRCSIM: An IBM PC Computer TeachingExercise on Blood Pressure Regulation. XXX IUPS Congress, Vancouver,Canada.

Russell, D.M. 1988. IDE: The Interpreter. In: Psotka, J., Massey, L.D., andMutter, S.A., Eds., Intelligent Tutoring Systems: Lessons Learned,Lawrence Erlbaum Publishers, Hillsdale, New Jersey, 323-349.

Sacerdoti, E.D., 1977. A Structure for Plans and Behavior. Elsevier-NorthHolland, Amsterdam, the Netherlands.

Shim, L., Evens, M., Rovick, A.A., and Michael, J.A. 1991. Effective CognitiveModeling in an Intelligent Tutoring System for CardiovascularPhysiology. Proceedings of Fourth IEEE Symposium on Computer-BasedMedical Systems. Baltimore, Maryland, 338-345.

Sleeman, D.H. and Brown, J.S. 1982. Intelligent Tutoring Systems. AcademicPress, New York.

Sussman, G.A. 1975. A Computational Model of Skill Acquisition, Elsevier-NorthHolland, New York.

Swartout, W. 1988. DARPA Santa Cruz Workshop on Planning. AX Magazine, Summer,115-131.

VanLehn, K. 1988. Student Modelling. In: Polson, M. C. and Richardson, J.J.,Eds., Foundations of Intelligent Tutoring Systems, Lawrence ErlbaumAssociates Publishers, Hillsdale, New Jersey: 55-78.

Wielinga, B.J. and Breuker, J.A. 1990. Models of Expertise. InternationalJournal of Intelligent Systems, 5(5):497-509.

Winkels, R., Breuker, J. and Sandberg, J. 1988. Didactic Discourse inIntelligent Help Systems. Proceedings of Intelligent Tutoring Systems:ITS-88, Montreal, Quebec, Canada, 279-285.

Wilkins, D. E. 1988. Practical Planning. Morgan Kaufmann, Los Altos,California.

Woo, C., Evens, M., Michael, J.A. and Rovick, A.A. 1991a. DynamicInstructional Planning for an Intelligent Physiology Tutoring System.Proceedings of Fourth IEEE Symposium on Computer-Based Medical Systems,Baltimore, MD:226-213.

Woo, C., Evens, M., Michael, J.A., and Rovick, A.A. 1991b. InstructionalPlanning for an Intelligent Medical Tutoring System. Proceedings of theThird Midwest Artificial Intelligence and Cognitive Science Conference.Carbondale, IL, 31-35.

Woo, C., Evens, M., Michael, J.A. and Rovick, A.A. 1991c. Planning in anintelligent Tutoring System. Poster Session, International Conference onthe Learning Sciences, Evanston, Illinois, U.S.A.

Woolf, B. 1984. Context Dependent Planning in a Machine Tutor.Ph.D.Dissertation, University of Massachusetts, Amherst, Massachusetts.

Zhang, Y. and Evens, M., 1990. Extending a Knowledge Base to SupportExplanations. Proceedings of Third Annual IEEE Symposium on Computer-Based Medical Systems, 259-266.

gDUawtL 4 WAS n Phinduiuin LAW"bo0w M-A.l varga.v wwMhIAT Lorec

Di. PIl 9 L Ainm P Woo 139 So."Upomo id he Now vow. NV MUia Clude "Go mino -dDiymmmpe.1F hiW Tm..g to-it75 amm Appe Road Dr. Dumb Shaaf NAS Pwomb. FL 3010"MI sho Nope Dwoo dp Cllg

Upopw N"3 Phi aaw mw DL Fu UoMpDI.seql "m siope Phd.. s hoem ... pil

Oqmm .1 ?1dbi Aiao U.msa IM am n bolueg

SWAP" U&...u" Di. 2. c. ohaCaaNim IIIas mamd- goome Lob On. .4 Ndb Raw&"&

N.W..I Smmbd Cab 11WDL. Good L. MAlphl Cols 134 sometime ad Temsemiw MAO QImpso"N". Pompan "ND C... 0pspmpa MD nmAdlgss VA nn1 I=Sap Dool,. CA MUI

DL Tavy Aftd Cmi. MS Tiu 1p SP. D.P...Code 1111C Ndb1m -mk lpo Colm Cob tiNon" .9 ou Nh." Kumilo 0m.L 3MxM. ".. Fapoi R&D C....

(M.mQ oft sms ank CA urnnVam VA ZM.$M DL. Am Ofa

Oumdes Speedd A dmou. Li.Wp. coad.DL ,- I Usenaft. 44Campeau N"P Iie R&D Co...Cooper So obe Soo~ P.1..&"u Ma4U Teloas Woli an cook. CA anstsib174 chaiai... Ann C..W sopisli CA "MI51 Oftsy DPW Meow Tow" 1peo.. C...CMm..gs. u. 61M Dr. Jobe . bmvo Ame Do. 11.6of blp% Cmi. 362

Jone L. McDonnel Popoeple 1M 116.00 Fmulpy0D. Spolie L Aoas6% hoom Sol" mw pipmed. FL im~iflCover .4 liaborpos,. sappap 106 las. SFSmmed Bk&iGould umwmuy SL Laumw No 8117 PAD Coadno Ama Us. HoaftImoldeplips PA SOM on". W of Dolq Ml.rr)p.*I XI

Do. Fissim &A Dslopopo of on no.% A^AWl7DL 0 --1 ApMug Cooer Sir Sm *80 ad Womm.... Oc Wmum

l3ipsq' W4 culkNis Meow Ra" LabommvpDr. Miubd S Amm L" MppG.. CA am Cede SOPNYHUS Wappeppe Dc UW1~iAl L .bmI Do. 3m...s Copper

Wines..@* Mm. 3s"1 AU.ý book MW Do. Jobe. M. CmmW~ha nowm MY lam 1m0 up. CC 3M Doll-a . Feiuh

Di. Pminvivi Pole D i Cospospe 1wo Uuiumps"Sia.41 loom"u CAMInueu.s Ud..mas Nm Calm%. A MIS

Umirmrp49 Mabsp lmi* PA u3s Do. nbIad CpAn Ameeo d M1 N I& Cobu 342

DL fi f .Capsule, somummsed P&D CooperDL. Maple L. hallo aussid Taftg SNOWSmDm. ArnZsNumy Pennmd PAD Cip..o Rudiub newdso Diego CA iOtDafs Pdoem Nila OL Swod Lm PCoup

-seep SomaAm.. .P4. am sives 0. son De Ra I. ClOimbe P.Iun so 1tUsml a mum a cc= NNW" Has soft Dmbow CA 211110-moipp aroa1o Cm..& bidb Uakm@MyUuu. , 41 F"ubý PA ID3W CbUmokepHi La...7

MFALY DL Po CLG Clog Mmmsý CA 0Mn.""3

Mr Phises ma~son Nuke Sing. Puadmmdns Dr. PUS Colma10M der Uiummens go "ON D rn UNIS iss. Copper llr e' , So

D4$04 Diulidu 16 WO GERMANY Di. hod PL 0kuf. 1111011 IludsioaFEDEAL PairUmc OF GERMANY ?moves"u HIS Two Pat P..

ur1 Numb Pu. afo DL ColluallooON 43fl-11uuDL. UIMpa C.q $".,.MmDhsW.. LAu mad Adfblew VA WRDL Clalp L. DudBWAMumNMui 1.4m aftime TOi@@ Smiums

ASP~QU . Mi, 0ig 410 O~opcios osop I.?111011WI AF36 DC MOMM Cupu sou Pims.am P, wA amt

DL. Tiou;M. 0.no Wu. nook aim" S. -olo Dv% LI&-Dupm etmg I4 PI pm AAmpU% VA =?.SU Appliedm..hoo Anum..

Rh... 2bm DL Peruedmm Ciea IN&%a PA tNUlad....NY4 myfe UMU tMP Spoon A~mer

MALMU.1101 DL Doe, o

an WKS same" a Toobm% FIL 32"

MNSOON To VVID Dr. &. Soma Dee

OL amdmalum nzUwe NoTema Nem Vmmummno.11C CANADA VET IZO1

OL Jobe a"n tldsopky D.4muTedbsmTeo..ss Colins, an a munghm lm mý Come

D~flCVOA.l v La 06 rkd obOXi30C..sma ý awe. Iu S&W W Na bu A-4-~ Umrrm KU60O0M

kMaab~e VA IWtd M O. DUN""pu s(4 C.Pu Ahomn VA Wit ft. S IL Oadý

P I IC.So an 45ft. Piu Dsp.P& Dr. lfds Re VsWadN60 ss

Dim m.L Eml m Camp."Wmsb Umam NON" INCesue.sd O..-qme Dwpuma.4d goo&

m.r Modwrow fftai I^ IA(m) Ms. Illo aGsl?111 PARE V uI Id DCFRMIKX Dr. Cod I. resdm Dwpaaa Coad GOVm.""

DOPe.. ad ouftw radh af 4% Re.. aDL. Raisd Duft M"U U&M"n 111 Ceammes Alma.% N.W.Osndsaf Iowa ad Emma.. ino USTesAb Da Wmh%"DC amUww" .1 C*Mb... MWN OrN6

UMe. EmaLCA so"d CANSA" A IMM Oi SIM.A 1mG~APUW

DLi.M Idei 016 Di um Pmbmt* 1mk AFID YX IN-D&me" Fauseas 3* C4aeW afneo To"e SuinCads Is (IM) DL T. OeASNOulSa bmw. CA 9=524MU ri III on"4 Os0p i .4

DL .ft Sawp e DL AMad L. Pap ibd Ad .4 owoVUu.saupad CAddesam AIolmo. W&~ asa s5pa -

be4 Dia Soft$ AMI DC 31114644 AdlW. GA IWN13110.pmam ad Uspeam. C-La look CA 9mE DiL AN" Fawl Dr. MagUp L. Go-is

Di*.r ad pqdaieg Offia of Pumead R*0EMIC Fely-Aaqs-daa U4Mmt4dAMbfbS Is. f. NW. Rea AUw1I0 ran" Ona. S." es ~afne. ASS.. de 0 . W Potaase M~pame"-Aaa'adS. mo lO.&41 CAXAaA 760 3111 WN&WO DC MISE

mi me" berno DL. bkol P. 0~~ Di. Mb C. GmmaDops. 4d Caspa ea Dimbad wCfpSdnw ben Osee adio IyasIN.. lasrn fd Tsdaief Dspmosoa haal , A R IWAea III Uampho Sm. Uam.ssq10 Wam Jim. gues Uomw ad Aaý Mempb. TN 3032Cesmw. IL A0 Tonmk 7fl

DL W"- ." aDr. Lama 0. OP& *bl Dspurnmna 49 n S."e4soa emmssus Odke ad Pwao-i MaOaMP.0 CeiMs8W 5dmw Famo uw.sO111.11 Pavma DAMOM "d Om Name Uwm"u Us WeNdf 4 uem

Dsmdupus Cdpam. take VA Sm New Ya.6 My SWOSm st. NWWmhubba. DC MI3 DLMin Rd. I*s DL. iamu 0. Os"s

no Limb Lab. CAds IllS beal o dwDameDi. isf.cbs& rakmp 415 ONIseb A-.66% L W. smatesi Um"OssssI~ar VAmin Urns. Wsa. DC amIsmi Rem Iihadaasemefe A EsbedAmil dm.Ihd CA 0"n

Ussemep .4 cbidmas ms Pulp a"SInna CA EYA? AARI-1a Oasis. DLI. Ssbs nms

ATM PEA.I. am er Aspw SpseusDr. Uwma L Psm Fed &%^3 aw. AUS e. 3eAmp wuadi lmbf III n a brPEME i Uin ium Emfa Uma

301141 860b~ Sam &am%. MA owsAboldms. VA nMi Taduec~asp

Ofto Uim - DL. Mibiad Mob"OF. Odmap" J. Pu, was 1219 ESON DownRNMOMOPsNv.3e C& Neow Yea4. NY NE? PAX Em 143= MO Use. Vas.ome. 067M. "' I.h Iqdpa VA ZMl Dr. Dow nin WOTOERMAJEY

aemted" Tud" ESsadsDr. PA.Padss~s usdsse 4034 DL. Nsamy X. N.

Cads UE How Rusa. Ie"NPADC N. Lee o QI0141 33d Read. Neebe me@% CA EniE JIB Do*d Asas Aqpa VA mi?

LSR68 VA 32M

CaP-is bemi Laboueq 016 PAdam Gins dmi ut IeP, umeamamss Ila PAm" Efmabses Toni" bSas211 Nmý bees & Osasispu Camu Cs.M sad EAMaIS Reeds

Dr. R~bdm L Pseps Plmt IPA IS=A-nmm Celp T-"qPAX EmIsm a5?. IA WO Mb. IL Hemlaup

DL. Made Oa G Ossp Mama Usiwussmi tsamrnsd 7mfm Fskib VA nM

DLI" irwW Fu Idw% my 1450D-amarna FpI asa Dr. fpbs J. Nussmoss-uyC~sig p DbOkea*"g L--*4 Ks.ad"Redoeas. No VIM Dqs d PspsIhdsa Aeqimee Arnesu

Dr WWF~p dwm Udm 01410m cwpRmd Butw

Cods 14 F~ia NII a"eNPROC Fred iesapb Gaps.Sae 0.8k CA 223=01dil "NIX thidof od"s 5mm Ilfmd

IS %Aft ReAd nosw Tis &desu "~d

75 L VMW IAu

UO L.. F" M30~1 OWN6 Smmý DL Lamm4

$"#= I PA. 3mm im mo VA ZZSM aaSmm hydkaFfib Chm VA WMu =W afaiftaf

Uom 41Cm.ID o* Saul U46-m.-wepuwr We~f MA am Ib~m L Gmam44

O4Mk 0"OSWAY DL A. KRJmmilb DI his~m LbS.MRC4U I Sum wam"i

DiQ l 17m 0NuW Ii M on~ LmamgMil a"p L&P Lafts) 1Dwwm of caponm and SOAND WON CAM Fab AN%~ CA HMM~mmumoun sm.§"ftmmim 7sSmuum DL WUmn S~AM Oi 1M C. Lum.urn. lI am Mu T. L Woumi Oman" Sbomd

ftsma CIL d somm. RuffDLi. HM Nobw PA on no Tdkm He@Duwmu 41 hmmisw yedbm Nsi0 MY 2 Usk, 41 Cbwwm

Dumwp.. 5i-6uy CA HiPUamuqj to Dukme DL Dv"~ ShmmMdeVO 06 10R6 Tuulm camomi mi.. mgumD p~ S ui

flDAL Mde, noa summd M Deuwo 41 NIlubDL Mos man3. ushof do Mmiaps maui 66.6mi SihamAmip 3~mb bom", 4 d An Ao% MIi 434636 i Loompum AwMM bww" Am. i 6.5w KýsutM*I

Alwnml VA OMu Dupmome 41 hp*Aw DL 3.4mmt UqIUdomiip41Cimuml Dup~ 410mqpb

Clumb"ig Umip Pm c.imj.. c 3m49 Wm 3m Sou" D6 JisM L KWbiim4.m Tm), NY ms6 o,1w.b kIm 4d Tb*mi. Loýi %La (Am IArny)

Caling 41 Cuuemujn Ti mim i Tim. 5,iD& 6mm memo Adef GA XM66 0tmmCbmi Sw"is PAL go OtisAFPHULCA Di hum la b so ha$@% CA 92415.14bmb AM~ Ix 700530 AFNIIMOU.

Di Sam N.0 NF3DC. C4l. 142)4HO V.. N. OL JO L Lim=M $a Dies% CA 324MsUwmawir A Abaft Num" De.. Cam.w

imam. Nbmmt U41 41m DL W~b L MfmpCANADA 74 05U 7X@ Fhal Caid Du Caim Ht

In Asmumb% 1264.M NgUTFmADL Obovi Imp@WW vmb - %L 32MUM15CmuEmm Sim Dqpmmm 016. DI mA LftwmTou~ mb um., IM No* Amului DL Me" Mu)..

VANN VA 1O kamdm TedmijmgDL Moo. L. 3q41 N UL.AFWW

Caift U. SP Smi 41po Di~ OL sJPAlim. No* 146P.iiu 06 an m owmmr 41 Ni.) HaCmom um

244 Re Low=. Owd 3UK VC 21MW 0.6ins147145 NMHUIDIA WWipam% DC 6.1)2W

Ruibmmummwmmbpm uscam co.. bq& 41 PupidomCmf Smw i 31. 3Ummi S.sw So Dip Sme U I wiui

Iu)pm Dn" 1 C U486m Sm Dhpm CA -0

DL 0mm L. MamlmNum4 De0 CamlbAMNK

D6in Lamb rkm awi l Pomm

Di il li.Daupmmew *IF Im Nm-.Us A.. mm No= aaMOS. X103 th~l ?A =303 Waipi DC 6.35U(AMP. O. anmm Sow. A COUMANOW UALM DL NSam SlmmSmom"ii.Qw-ba WC ISt P66.3 (ATM Lama) Dupummu of NmwobCANADA SM bmen&~Aim" C.,I.,C. Nmmmmm

D wAmAmdA VA 032IM4 ?mwb Uulvm*u S*" ed 4 u

Dipmemg 4e opq~ibg Dqmmum .Caumpin IPAmm53ob Pm" CA. CD Di Mimum &L UNA.

DL-OKdwmoo CA 6we Unm*m 41 Soma Caiibm

U IsS p of Mmin= DiMPA. labmmkb,6s 1Dupoftmel.4 P I Iib Mn S&4(campm simm Dm. Mem i

Fag.. LAeaMM C Nrwuk~ d C.011.46

AM A..., bd Me Sm in" Cfew"i

LAMM Aim l5-EUL lPii a Mmayo mad co ow Aehw now Aimp uublws

I y I 1 COM... MWad UMeaib L46OK Ahmdn VA -UN4Isma 8.60m. CA ste Gods no

W DC WINS D $06 Saem M

430 CsA"- I--. bodbe MN. WMmNow V..e NY 3m? A~mV VA =?.NO De.L WA~ie Room

(4 CA001W AnuLADDL -nq Ha5mbe bem ADS ummay P.eMi fmemo Spead Aiedm IWBOOS

Md OD-kpoow Cmou Unemgm DL bo RAWCede 14 md So Neow twun (F.OUI)MT ~Sm ORM CA 3snUN Dqwmm 40 0 Noy~iS~. 1~

DLMAdmo"Wuis, DC am~ 189 `"a'de !WetmlP,

W.I.%PeMWMw ADS. ON 414) MAUA Ama beANN Ceeue Loommi R A D C~..Mwka Flwd CA 9s) Ushommy Im L olw

DL~ Je A. Mudedmmve SoDqesa of ftlysamhH lis" hI " PFe~ PA uns2flab Foopawftm Ulm N-V Thoad $14"If

Medal Ce..m Cairn wCed nz)) 0. N~a bammRmeA usimm Ceiap In") Ransw two"u Mad Slsp KU-34Cmw IL 403i Oubdw$ FL no&=U n -.. AmmeSpam...

0D. Am..m 0 Now. DLwn II ICew Y111

LAWC loho Gmr Do Aedmay Mr. W. A. RuM

3wom 05~. has mo VA Z131 NeOW Tede" I,... Cm...PsmAaW, FA 513 Code 390L #Aim. Mem Amm.,ww Immium). 0.h0de. FL )2mimihShde. Towmie, P555-Leadd..US -us ubanmo Anom Dr. US 0. Rob"a

no N. Now. at.. saw 13 Almmml VA 25)ANN b adetm Rimadned Mmo* CA ow" Tfpead.. P"ge.

DL. Se Pm ON" st TabeeiOgDooftwm Sm,..e Dbdm leMVmN for ON 0mg.. .1 *. Used sme.mai. of Nand Pemad Seo"= ee. W6eadw.. oC 250

*akw.s. OC sm 3No Mpi. Amom 0.. IM. L R.mDommos% IL 40rM Mam

Taheeag lea.

AIAmml ft"6 a RON" b*MA 4w) Mmg.M Ca.. Casenow Teefmed TO."s Comamd DL. Any L Fm. swimCods N42 AM (P551U) NORMS OA3A" "Ral" (m5 3m obsemn Mm.Mamp... IN um Almoolh VA M53 0.. Ebegi. sal

Hmem Pwmi ODhim.a (Cede 262)DqWm Dwome Wmapomm 0. Nemey N. Pany 335D aKseS Poimm

Pewamed ad fusing Dm. Mad Eawma m"d Tuudie Mawd Tomens S3.um. Ca..Need Sm Spam. Gamed ha. Surppar Aft" 1 CuaeL jnM3)24AITh. Code 05F "Il CdmauWabega. DC WO Fam~ 2

PeI.FL 3mK1

use FaCPSLaegbm

Mr. 1. Nsmo lied as1wms uamme Cede OR"S mam. IL 4IPa. UL Tw~paesu Caaomelmdef moo CA am0P. o. an n7 DO. Ma Sb~?m An ameba"e DIPs.O Amimilmam. 5dm. 251 b"MNNmmThe MinUISNDS CAd& 36 iS) WMOIS~ed AS.

Dines bma DeAalmd MUNIN CA ammI meb rook CA NoNod "La T4 pm* DaSOWupw Li.S mio

HV. Spa. AvOW (NgliSSA) Depsmm So5sol Lowele PAD Cm...PeMab.6 FL 3)26 me comged" om ee..Unp4 ue

Dismers Mww Mmabb PA Ism** ?smes Spowm Dpar-e CmAdek MA mmS

MMRDC (Cede 34)L ws~tsomfDug CA Ff3-u DO. trPlums ago Pabed.

Mmledad Mmadm. CeeadiW CA SUNFUl 45 CeI~as

*T-inag Todedoe Dwps... Suaweg CA omi Nab MmanowMMRDC (Cede 1S) D~Mh aso book CA Im)4iSD Mashe FamemD0 Um ilw Umim.

oun imm mADLO&~ u~

SPAM Mwn Aummew "iD Guamwam ans, CA Mus

OL him W. 8"Mram OL WINDSm

W1 Miw Saw" Abm.. OW SAM of obidwwwp DC -so TWO"n " IM.

DL Pub.. & Sidd sooq. CA WiMLOS AN" Romemi %MI nomebolamA L~ M.gdwAIM~f VA iNS NW MH-3

0nmb. fd ONUbaim- - d Caiai OL as" wow

Sums Sub.. CA AN load O dmillm

DL On Swaidame Ndmisa UdiWW

Usama ad MOMMCoabp he% up 316 OL OWNi c UINI

DL Mm" Mi umba im asdCampin b.Moad Ran" abussv 40 MX" Maion lAmm

MS) Ofta& A"=%m LW.Woftnews DC MOMMIM D& TWdA Wb.

ca.s~ Smemai -DIL Ibida Si. Mi Waim ow Sim UdiiLRDC LUOS esm X ANM

3630Mm $0,4 D. onm iMONA 10i PA WI eivala al Gomies

I Mass do~ MaDfL Sam U. U~bmWIn Gumm 4. NSWIrANlkM DiWenwas d bftW sad

4- -modo~OL Suble sbmw SM SRO Duo.Owing Mmi~wem Subh VA MWN

MS Ito 0mb. Solid do smudgW41110 Nib" It. FIOAND Us"AN CaU La A"".

Las Mm.k CA miDL. fluid L Sum,Wall Of snidm OL IbMunia"bSead" Usivma Dipi.d Cwa SsiameIf"IPI CA ~s U mi SmlWA

POIdNISOfImMI PAWOL aum Sim* ammo3w?$AAAUM ISKAND&EMU A1Sea AMime TX 1i1411

Faf L. Yubraacaps, 41oftmss

OL Phow Sapur C~aia usimesab umdaw Smiiy ~bp%.DCw[memo la Miimi

Slefil 6h ofagod hu Dimib eanSIMIM CA 90"11 Medoi Siamo flodin.

PAWmADL Dm~m Tow UiS@ Vi~NA.LDilidid TOMM LAbA Mampa00.Dc 31udweafteesmo immb.S=4ine1d6 at OL SWNM learnSi. Umi. CA MAI VS.I 11

-su WL Mmsafi 0611hilion% ma.C 41abd~miee10IOfa" Slo Somal WOW V01% or MUN

UW-i al 0001Shim. caump

81IM MID AM

DR. n- VmDqrnuds 4 F I~S. Niaomi CiumPDo Pi.t we Mum

CIL Ilem, A. WevFAA -MPAL an UUM

Omw-e 05h OKI