mhhhommmmmu eeeeeeoheohmhi - dtic.mil tokenizer-processor (tknize) ... a finite state machine to...

185
A O-AI16 502 ALFRED P $LOAN SCHOOL OF MANA EMENT CAMIUAO MA P/I' 9/2 VIRTUAL INORNATION FACILITY OF TIE ZWFOPLEX SOFTWARE TEST VENI-ETcluI MAY 82 J5 LEE N0003961-C-OWA mhhhommmmmu EEEEEEohEohmhI

Upload: vuongkhuong

Post on 05-May-2018

213 views

Category:

Documents


1 download

TRANSCRIPT

Page 1: mhhhommmmmu EEEEEEohEohmhI - dtic.mil TOKENIZER-PROCESSOR (TKNIZE) ... A finite state machine to parse data base statements written in this language c) A user-interface tailored to

A O-AI16 502 ALFRED P $LOAN SCHOOL OF MANA EMENT CAMIUAO MA P/I' 9/2VIRTUAL INORNATION FACILITY OF TIE ZWFOPLEX SOFTWARE TEST VENI-ETcluIMAY 82 J5 LEE N0003961-C-OWA

mhhhommmmmuEEEEEEohEohmhI

Page 2: mhhhommmmmu EEEEEEohEohmhI - dtic.mil TOKENIZER-PROCESSOR (TKNIZE) ... A finite state machine to parse data base statements written in this language c) A user-interface tailored to

1,Center for Information Systems Research

Massachusetts Institute of TechnologySloan School of Management

77 Massachusetts AvenueCambridge, Massachusetts, 02139

Page 3: mhhhommmmmu EEEEEEohEohmhI - dtic.mil TOKENIZER-PROCESSOR (TKNIZE) ... A finite state machine to parse data base statements written in this language c) A user-interface tailored to

Contract Number N00039-81-0663 (MIT # 91445)Internal Report Number M010-8205-10Deliverable Number

VIRTUAL INFORMATION FACILITYOF THE INFOPLEX SOFTWARE TEST VEHICLE

(PART I)

Technical Report #10 r

By

Jameson Lee

May, 1982

Principal Investigator:Professor Stuart E. Madnick

Prepared for: -4Naval Electronics Systems CommandWashington, D.C. "O

Page 4: mhhhommmmmu EEEEEEohEohmhI - dtic.mil TOKENIZER-PROCESSOR (TKNIZE) ... A finite state machine to parse data base statements written in this language c) A user-interface tailored to

SECURITY CLASSIFICATION OF TWIS PAGE (W"n Date Entered)

REPORT DOCUMENTATION PAGE BEFORE COMPLETING FORMI. REPORT NUMBER 12, GOVT ACCESSION NO. 3. RECIPIENT'S CATALOG NUMBER

Technical Report #10 T ,

4. TITLE (and Subtitle) S. TYPE OF REPORT & PERIOD COVERED

Virtual Information Facility of theINFOPLEX Software Test Vehicle 4. PERFORMING ORG. REPORT NUMBER

M010-8205-107. AUTNOR(e) S. CONTRACT OR GRANT NUMBER(@)

Jameson Lee N00 139-81-C-0663

9. PERFORMING ORGANIZATION NAME AND ADDRESS 10. PROGRAM ELEMENT. PROJECT, TASK

AREA & WORK UNIT NUMBERS

Sloan School of Management, MIT50 Memorial Drive, Cambridge, MA 02139

I. CONTROLLING OFFICE NAME AND ADDRESS 12. REPORT DATE

May 198213. NUMBER OF PAGES

18014. MONITORING AGENCY NAME & ADDRESS(Il dilferent from Controlling Office) IS. SECURITY CLASS, (of thlo report)

unclassified

IS&. DECLASSI FICATIONi DOWNGRADINGSCHEDULE

I. DISTRIBUTION STATEMENT (of this Report)

Approved for public release; distribution unlimited

17. DISTRIBUTION STATEMENT (of the abetrect entered in Block 20, If different from Report)

1III. SUPPLEMENTARY NOTES

19. KEY WORDS (Continue on reveree Wide If neceeary and identify by block number)

database computer, database management system, SoftwareTest Vehicle, hierarchical system, virtual information

20. ABSTRACT (Continue an revered side If neceeeery end identify by block number)"_-J .... This report describes the software designand implementation of the front-

end for the Virtual information facility of the INFOPLEX database computer.It is part of a major effort to develop a software simulation, calledSoftware Test Vehicle, for the underlying architecture of INFOPLEX.The virtual information facility is a single level of operations situatedwithin the Functional Hierarchy. It supports the use of virtual information,a virtual entity based on procedural relationships and derivations from

DD I FaM"73 1473 EDITION OF I NOV 65 IS OBSOLETESECURITY CLASSIFICATION OF THIS PAGE (When Dole IntereW

Page 5: mhhhommmmmu EEEEEEohEohmhI - dtic.mil TOKENIZER-PROCESSOR (TKNIZE) ... A finite state machine to parse data base statements written in this language c) A user-interface tailored to

... ITY CLASSIFICATION OF THIS PAGltI(hon Data Entered)

Nphysically recorded data. Upon completion, this facility will be

integrated within the current implementation of the STV for the INFOPLEXFunctional Hierarchy which lacks the support for virtual informationprocessing.

NTTS

A va I

SECURITY CLASSIFICATION OF THIS PAOErWhon Data Enteted)

I .. . .. . .. . . . .

Page 6: mhhhommmmmu EEEEEEohEohmhI - dtic.mil TOKENIZER-PROCESSOR (TKNIZE) ... A finite state machine to parse data base statements written in this language c) A user-interface tailored to

Virtual Information Facilityof the INFOPLEX Software Test Vehicle

by

JAMESON LEE

Submitted to the Department of ElectricalEngineering and Computer Science in May,

1982, in partial fulfillment of therequirements for the degree of

Bachelor of Science

Abstract

This thesis is a software design and implementation of the

front-end for the Virtual Information Facility of the INFOPLEX

data base computer. It is part of a major effort to develop a

software simulation, so called a Software Test Vehicle, STV

for the underlying architecture of INFOPLEX.

INFOPLEX is a hierarchical architecture for data base com-

puters, based on functional decomposition of data base oper-

ations. It is a current research project of the Information

Systems Group at M.I.T.'s Sloan School of Management. Within

the INFOPLEX architecture, a functional hierarchy of informa-

tion management functions is built on top of a storage

hierarchy of information storage functions. These two inde-

pendent hierarchies are further divided into many sub-levels,

each of which is devoted to a more specific function of data

base activities.

~2

Page 7: mhhhommmmmu EEEEEEohEohmhI - dtic.mil TOKENIZER-PROCESSOR (TKNIZE) ... A finite state machine to parse data base statements written in this language c) A user-interface tailored to

The virtual information facility is a single level of oper-

ations situated within the functional hierarchy. It supports

the use of virtual information, a virtual entity based on pro-

cedural relationships and derivations from physically recorded

A !data. Upon completion, this facility will be integrated within

the current implementation of the the STV for the INEOPLEX

functional hierarchy which lacks the support for virtual

information processing.

Thesis Supervisor: Professor Stuart E. Madnick

Sloan School of Management, M.I.T.

3

-a.!---

Page 8: mhhhommmmmu EEEEEEohEohmhI - dtic.mil TOKENIZER-PROCESSOR (TKNIZE) ... A finite state machine to parse data base statements written in this language c) A user-interface tailored to

Contents

Page

Title ........................................................... 1

Abstract ........................................................ 2

Acknowledgement ................................................. 4

Contents ........................................................ 5

List of Figures ................................................. 8

Chapter 1 Introduction ........................................ 9

1.1.0 INFOPLEX Overview ................................... 91.1.1 Concept........................................ 101.1.2 Infoplex Architecture............................ 101.1.3 Functional Hierarchy ............................ 121.1.4 Research Issues .................................. 12

1.2.0 Thesis Objectives .................................. 12

1.2.1 Background ....................................... 14

Chapter 2 Virtual Information ................................ 16

2.1.0 Concept ............................................ 16

2.2.0 Classification ..................................... 162.2.1 Factored Facts ................................... 162.2.2 Computed Facts ................................... 172.2.3 Inferred Facts ................................... 18

2.3.0 Specification ...................................... 19

2.4.0 Merits.............................................19

2.5.0 Approach ............................................. 21

Chapter 3 Functionalities .................................... 23

3.1.0 Underlying Data Model .............................. 233.2.0 Active Workspace ................................... 24

3.3.0 Permanently Defined Virtual Information .......... 25

3.4.0 Adhoc Virtual Information ......................... 25

3.5.0 Notion of a Transaction ........................... 26S

Page 9: mhhhommmmmu EEEEEEohEohmhI - dtic.mil TOKENIZER-PROCESSOR (TKNIZE) ... A finite state machine to parse data base statements written in this language c) A user-interface tailored to

3.6.0 Virtual Attributes .................................. 26

3.7.0 Conditions on Real or Virtual Attributes ......... 28

3.8.0 Virtual Entity Sets ................................. 28

3.9.0 Generalized Macro Facility ......................... 29

3.10.0 Extended Functionalities ......................... 303.10.1 User Dependent Virtual Definitions ............ 303.10.2 Inferred Facts of Undesignated Indirection.... 31

Chapter 4 Program Structure ................................... 32

4.1.0 Module Description .................................. 324.1.1 User-interface .................................... 344.1.2 Buffer ....... .................................. 364.1.3 Activity Coordinator ........................... 414.1.4 Tokenizer-Processor .............................. 424.1.5 Language Design and Specification .............. 454.1.6 Finite-State-Automaton (Machine) .............. .45

4.2.0 Internal Global Variables .......................... 46

Chapter 5 Language Illustration and Specification .......... 48

5.1.0 Data Base Statements ............................... 485.1.1 Define Statements ................................. 485.1.2 Adhoc Statements .................................. 495.1.3 Listdef Statements ............................... 505.1.4 Retrieve Statements .............................. 50

5.2.0 Buffer Commands ..................................... 585.2.1 Command Syntax .................................... 59

5.3.0 Formal Description of Data Base Language Grammar. 615.3.1 BNF Supplement .................................... 64

Chapter 6 Finite-State-Machine ............................... 66

6.1.0 Configuration ....................................... 68

6.2.0 Match-Action-Next State Rules ...................... 69

6.3.0 Action Routines ..................................... 76

6.4.0 Listing .......................................... 79

Chapter 7 Major Design Issues ................................ 90

7.1.0 Form of Storage for Virtual Definitions .......... 90

6

Page 10: mhhhommmmmu EEEEEEohEohmhI - dtic.mil TOKENIZER-PROCESSOR (TKNIZE) ... A finite state machine to parse data base statements written in this language c) A user-interface tailored to

7.2.0 Parser Structure..................................... 92

-~7.3.0 Program Control Structure........................... 92

7.4.0 Interactive Editor.................................. 93

7.5.0 Language Design...................................... 94

Chapter 8 Conclusion........................................... 96

Bibliography..................................................... 98

K! Appendix:........................................................ 99

Programs ListingUSER-INTERFACE (USINT).......................... 100BUFFER (NEWBUFF)........................ 106ACTIVITY COORDINATOR (ACTCRD)......................... 128TOKENIZER-PROCESSOR (TKNIZE)......................... 136DATA STRUCTURES.................................. 148

FINITE-STATE-MACHINE Rules.................................. 151

A Very Simple Sample Session................................ 161

7

Page 11: mhhhommmmmu EEEEEEohEohmhI - dtic.mil TOKENIZER-PROCESSOR (TKNIZE) ... A finite state machine to parse data base statements written in this language c) A user-interface tailored to

List of Figures

?age -

A1.1 INFOPLEX Architecture !

3.1 Entity Data Model 24

3.2 Sample Data Graph 27

4.1 Module Flow Chart 33

Page 12: mhhhommmmmu EEEEEEohEohmhI - dtic.mil TOKENIZER-PROCESSOR (TKNIZE) ... A finite state machine to parse data base statements written in this language c) A user-interface tailored to

1.0.0 INTRODUCTION

INFOPLEX DATA BASE COMPUTER is a current research project of

the Information Systems Group at M. I.T. 's Sloan School of Man-

agement. It proposes a new architecture whose objectives are

to provide substantial improvements in information management

performance over conventional computer architectures, and to

provide highly reliable support for very large and complex data

bases.

1.1.0 INFOPLEX OVERVIEW

Progress of modern society has put increasingly more new and

challenging demands upon the capability and performance of

information storage, retrieval, and management. Conventional

computers, whose architecture is designed primarily for compu-

tational objectives, are not suited to meet the requirements of

these new demands. Efforts have been made in four different

areas to build computer systems which will suit our information

needs today, and in the future: (1) new instructions through

microprogramming, (2) intelligent controllers, (3) dedicated

computers for data base operations, and (4) data base

computers. INFOPLEX is a research project belonging to the

fourth category.

9

Page 13: mhhhommmmmu EEEEEEohEohmhI - dtic.mil TOKENIZER-PROCESSOR (TKNIZE) ... A finite state machine to parse data base statements written in this language c) A user-interface tailored to

1.1.1 CONCEPT

INFOPLEX employs the concept of hierarchical decomposition

which organizes information management functions into a func-

tional hierarchy, and the physical memory management functions

into a storage hierarchy (Madnick 78); both hierarchies con-

sist of many independent levels of operation, each of which

supports a different set o' information or storage management

functions through the use of multiple microprocessors.

1.1.2 INFOPLEX ARCHITECTURE

As stated previously, INFOPLEX is an architecture for data

base computers based on hierarchical decomposition. A func-

tional hierarchy of information management functions is built

on top of a hierarchy of information storage functions. Both

hierarchies are further divided into many functionally inde-

pendent levels of operation, each of which is to be supported

by a set of micro-processors operating in parallel with one

another. A global Communication Bus coordinates inter-level

transmission of data. This hierarchical architecture exploits

the advantages of functional modularity of operations, and of

parallel processing of micro-processors to systemize data base

activities and to achieve a prescribed level of efficiency. A

graphical illustration of this architecture is presented in

figure 1.1

10

Page 14: mhhhommmmmu EEEEEEohEohmhI - dtic.mil TOKENIZER-PROCESSOR (TKNIZE) ... A finite state machine to parse data base statements written in this language c) A user-interface tailored to

1 I/\NFOPLEKX Avfchiecture-

I~1 -- vn c.or,

Glcbci _ _ 0 .

Ficvure 1. 1

Page 15: mhhhommmmmu EEEEEEohEohmhI - dtic.mil TOKENIZER-PROCESSOR (TKNIZE) ... A finite state machine to parse data base statements written in this language c) A user-interface tailored to

1.1.3 FUNCTIONAL HIERARCHY

!! Current architecture of the functional hierarchy (Hsu 1982)

with respect to data abstraction consists of four separate lev-

els: (1) external level, (2) conceptual level, (3) entity

level, and (4) internal level. A part of the conceptual level

is a virtual information facility (Hsu 1982). Thess four levels

of information management are highly independent of one anoth-

, er, and each is responsible for a different but necessary phase

of information processing in a data base computer.

1.1.4 RESEARCH ISSUES

Major efforts of INFOPLEX research are devoted to the design,

modeling, and evaluation of an optimal decomposition strategy

for both the functional and memory hierarchy of information

management and storage operation, and also to the study of an

associated distributed control mechanism. This control mech-

anism would be used to coordinate the activities of and

inter-level communications within the hierarchies.

.41.2.0 THESIS OBJECTIVE

This thesis shares a joint mission with a concurrent thesis

by Peter Lu. The two theses are entirely separate in

42

Page 16: mhhhommmmmu EEEEEEohEohmhI - dtic.mil TOKENIZER-PROCESSOR (TKNIZE) ... A finite state machine to parse data base statements written in this language c) A user-interface tailored to

functionalities, but closely related and dependent upon one

another for a complete software simulation of the virtual

information facility on the INFOPLEX data base computer archi-

* 'tecture. This facility would incorporate the design and imple-

mentation of two sub-levels of the INFOPLEX functional hierar-

chy, the virtual information level, and an user interface level

which is tailored for the use of virtual information

-* processing.

This thesis is responsible for fullfillment of the front-end

objectives of the joint mission; the front-end objectives

include the design and implementation of the following:

a) A data base language to support virtual information

b) A finite state machine to parse data base statements

written in this language

c) A user-interface tailored to the use of virtual

information.

d) A processor to process the creation, listing, and

modifications of virtual definitions, as well

as the substitution of these definitions into

data base statements in actual use.

13

Page 17: mhhhommmmmu EEEEEEohEohmhI - dtic.mil TOKENIZER-PROCESSOR (TKNIZE) ... A finite state machine to parse data base statements written in this language c) A user-interface tailored to

This processor would also be responsible for

transforming data base statements into a chain of

tokens, each of which would include an indicator

describing the classification of the token

according to a prescribed classification scheme.

The.combined objectives of this "front-end" and Peter Lu's

"back-end" would fullfill our joint mission as mentioned ear-

lier, namely, to construct in software a virtual information

facility with its own user interface, from here or. referred to

as VIFI, Virtual Information Interpreter.

1.2.1 BACKGROUND

In the three short months in which VIFI was develped, we

labored and wished to exhibit a certain degree of

professionalism in its design and implementation. The merits

of modular programming, of innovative algorithms, of perform-

ance efficiency, of functional capabilities, of

user-friendliness of the proposed data based language, of pro-

gram organization and flexibility, and even of consistencies

in programing style were evaluated against time and labor lim-

itations. A serious attempt was made to incorporate all of

these characteristics into our Virtual Information

Interpreter.

14

. • . ..A

Page 18: mhhhommmmmu EEEEEEohEohmhI - dtic.mil TOKENIZER-PROCESSOR (TKNIZE) ... A finite state machine to parse data base statements written in this language c) A user-interface tailored to

While making these considerations, many sleepless nights of

unceasing arguments plagued the two developers; it was the

intrinsic dissention between the idealist and the pragmatist.

At a certain point, such disagreements grew to be so severe

that it appeared to have left an unpleasant mark on a very close

and strongly bonded friendship. However, a lesson of humanity

was learned from this experience, and our cherished friendship

would continue to grow, and become stronger than never before,

>4 because we have acknowledged a feeling of faith and destiny

which was manifested through this experience. I am expressing

this sentiment here because I consider it the most personally

meaningful and lasting reward of this thesis.

Jm5

Page 19: mhhhommmmmu EEEEEEohEohmhI - dtic.mil TOKENIZER-PROCESSOR (TKNIZE) ... A finite state machine to parse data base statements written in this language c) A user-interface tailored to

I

2.0.0 VIRTUAL INFORMATION

2.1.0 Concept

The concept of virtual information in data base systems has

been developed and examined in earlier research of the Informa-

tion Systems Group. Basically, there is a spectrum of the kinds

of information which may be retrieved from a data base. Along

this spectrum, pure data occupy an extreme on one end, and pure

algorithms occupy the extreme on the other. In between these

two extremes are the information which may be derived from a

combination of data and algorithms; such information are

dynamic and procedural in nature, and are referred to as Virtu-

al Information.

2.2.0 CLASSIFICATION

Virtual information may be categorized into three major

classes: factored facts, inferred facts, and computed facts.

Together, these three classes of virtual information and com-

binations there of, constitute the portion of the information

spectrum between the two extremes of pure data and pure algo-

A rithms.

2.2.1 FACTORED FACTS

1 16

I

Page 20: mhhhommmmmu EEEEEEohEohmhI - dtic.mil TOKENIZER-PROCESSOR (TKNIZE) ... A finite state machine to parse data base statements written in this language c) A user-interface tailored to

Factored facts, subsets of data elements, based on certain

prescribed conditions, or so called predicates, of attribute

values, are often very valuable in structuring information in a

useful manner. For instance, if a certain data base maintains

records of weight, hair color, and salary for a group of

employees, it may be useful to select from this group those

individuals who share a certain condition on their attribute

values, such as having black hair, making a salary greater than

8 dollars per hour, or weighing over 300 pounds. It is impor-

tant that users of information should be able to access

information independent of the particular factoring involved;

this would imply the ability to support multi-level factoring,

or repeated factoring of data.

2.2.2 COMPUTED FACTS

Computed facts are those information which are obtainable

through the application of particular computational algorithms

and operators on data or groups of data. These operators

include arithmatic, comparative, boolean, and other kinds of

functions. In the very least, computed facts include those pure

data manifested in a different form, with a different unit of

measure, or an alias name. For instance: a user may define a

virtual age attribute to be the difference between the current

year and a person's birth-year, a virtual rectangular area

17

_

Page 21: mhhhommmmmu EEEEEEohEohmhI - dtic.mil TOKENIZER-PROCESSOR (TKNIZE) ... A finite state machine to parse data base statements written in this language c) A user-interface tailored to

S, '

*1;

attribute to be the length multiplied by the width, or an

-- f attribute value in the unit of inches to be 12 times the attri-

bute value in the unit of feet. In this sense, transformations

between different units of measure are intrinsic to the oper-

ations of computed facts.

2.2.3 INFERRED FACTS

Inferred facts pertain to implicit relationships which thedata base system may arrive at through certain levels of indi-

rection. In other words, a path, although indirect, does exist

which leads to the desired data in storage. There are two ways

by which the system on its own can support this kind of virtual

information. The first method is by an exhaustive search of all

possible paths, and the second is the application of a certain

degree of artificial intelligence to deduce a viable path to

the target data. Well, the first method is unbounded in comput-

ing time, and even when a path is found, it may not be the

correct path; the second method is far fetched at this time.

Therefore, we will give our attention to a different but compa-

rable set of inferred facts which is implementable, and we give

it the name Pseudo Inferred Facts. Pseudo Inferred Facts are

exactly the same as inferred facts except that all the indi-

rections will be explicitly designated by the user. With this

strategy, exhaustive searche is not necessary, artificial

intelligence is not necessary, and the specified path would

18

4!

Page 22: mhhhommmmmu EEEEEEohEohmhI - dtic.mil TOKENIZER-PROCESSOR (TKNIZE) ... A finite state machine to parse data base statements written in this language c) A user-interface tailored to

always be the designated and correct path. For instance, the

Uncle relationship may be defined as the application of the

Brother relationship after the application of the Mother

relationship.

2.3.0 SPECIFICATION

Users of information, through the virtual information facil-

ity, define their own working environment and the manner in

which they would like to use the physical and underlying data.

Such definitions of virtual information may be accomplished

through a virtual information definition language. The virtu-

al information facility would accept virtual information

definitions and their modifications in the definition

language, and respond to virtual information retrieval

requests through a separate virtual information retrieval lan-

guage.

2.4.0 MERITS

There are several major merits in the support of virtual

information in a data base system. It is dynamic in nature

because its definition may be created, deleted, and modified

readily; its definition applies to all instances of data where

it may apply, and yet there is but only one copy of this defi-

nition stored in the system. By facilitaing the ease of

19

x.

Page 23: mhhhommmmmu EEEEEEohEohmhI - dtic.mil TOKENIZER-PROCESSOR (TKNIZE) ... A finite state machine to parse data base statements written in this language c) A user-interface tailored to

modification, it enhances data base flexibility, by eliminat-

ing redundant physical records, it contributes to more

consistent data, and by being procedural in nature, it enhances

information accuracy through the delay in the evaluation of

data which vary over time or other changing factors until their

time of use. These kinds of merits are based on virtual infor-

mation's association with procedural relationships. For

inbtance: the stored algorithm for computing age would elimi-

nate the need to update the age attribute day by day if it were

physically stored, and would be applied to calculate anyone's

age, thus el.m:nating redundancy of stored information.

Virtual information also conserves the use of vast amounts of

physical storage. It makes unnecessary the storage and

maintainence of those information which may be derived upon

request. This raises the issue of Time/Space trade-off, which

should be seriously considered when deciding which kinds of

fundamental data are or are not to be physically stored. Deri-

vation upon requests will have the added cost of derivation;

therefore, those information which will be used many times and

are also difficult to derive may be the best kind of data to be

physically stored; those information which is seldomly used

and easy to derive may be the best kind of data not to be phys-

ically stored. Furthermore, the situation is made even more

complex as we realize that the definitions themselves will

require the use of physical storage. Thus, it wouldn't be an

20

Page 24: mhhhommmmmu EEEEEEohEohmhI - dtic.mil TOKENIZER-PROCESSOR (TKNIZE) ... A finite state machine to parse data base statements written in this language c) A user-interface tailored to

easy task to decide which kinds of data are to be derived, or to

be actually stored.

The definition of virtual information on a per user basis

would simulate an entire virtual data base for each individual

user. Each one would be free to tailored the data base to his

own preferred view or use through the virtual information defi-

nitions. A particular set of virtual definitions may be very

useful for one group of users, and another set for another

group of users. In this sense, each one has gotten a data base

suited for his own use while not affecting anybody else's usage

of the data base. A logical extension of this scenario is to

implement access control mechanisms such that users may estab-

lish a controlled sharing of sets of virtual information

definitions with one another; the data base administrator may

monitor all such sharing to prevent unauthorized access to a

certain set of virtual information functions. However, in a

scenario as such, a separate catalogue would have to be main-

tained for each and every user, and considerable catalogue

management would be required. Such is the cost for this indi-

vidually user-tailored data base functionality, a secondary

merit of the use of Virtual Information.

2.5.0 APPROACH

21

Page 25: mhhhommmmmu EEEEEEohEohmhI - dtic.mil TOKENIZER-PROCESSOR (TKNIZE) ... A finite state machine to parse data base statements written in this language c) A user-interface tailored to

I The concept of virtual information leads directly to a func-

tional approach to data bases. A virtual information facility

would be treated as a collection of functions, and retrieved

data would be regarded as functional values. Virtual informa-

tion requests correspond to function invocations; this func-

tional approach to information readily supports procedural

relationships on which based the concept of virtual informa-

tion. As a result, a virtual information facility is likely to

resemble very much a language interpreter which accepts func-

tional definitions and respond to functional invocations with

specified arguments.

22

Page 26: mhhhommmmmu EEEEEEohEohmhI - dtic.mil TOKENIZER-PROCESSOR (TKNIZE) ... A finite state machine to parse data base statements written in this language c) A user-interface tailored to

V

3.0.0 FUNCTIONALITIES

There are numerous functionalities to a virtual information

facility, each of which may be implemented to a varying degree

of completeness. Although it may be desirable to implement all

the functionalities there are wherever possible, it may be too

impractical and less than meaningful for the initial version of

the implementation. Thus, we have not implemented the One Data

Base per user feature of virtual information capabilities

which we have described in the previous chapter. Later

portions of this chapter would describe the functionalities of

virtual information which we did implement; surely, not all of

these implementations would be without room for further

refinement, even though they already include an extensive set

of virtual information capabilities.

3.1.0 UNDERLYING DATA MODEL

The virtual information facility lies on top of the entity

set level of the functional hierarchy. in this level, the data

base is seen as a network of entity sets and their attributes.

Each entity set may have a varying number of attributes, some

of them being value attributes and others being entity attri-

butes. (Hsu 1980) The value attributes include a set of

attribute values, and the entity attributes represent

2?

Page 27: mhhhommmmmu EEEEEEohEohmhI - dtic.mil TOKENIZER-PROCESSOR (TKNIZE) ... A finite state machine to parse data base statements written in this language c) A user-interface tailored to

V!

*1

relationships leading to other er:_-y sets F'gure 3.1 brefy

illustrates this model rOe~n ,-

Fig 3.1

3.2.0 ACTIVE WORKSPACE

We have developed an active workspace which :ncorporates a

line editor with full screen display, through which user com-

mands may be issued. The workspace consists of two buffers, an

execution buffer, and a transaction buffer. The transaction

buffer witholds many data base statements which will be exe-

cuted sequentially when the transaction buffer is executed.

The execution buffer holds a single data base statement and

will be automatically executed when a data base statement is

completed. A number of buffer commands is created to manipulate

buffer contents. The details of these commands as well as the

data base statements will be illustrated in chapter 5.

24

----------

Page 28: mhhhommmmmu EEEEEEohEohmhI - dtic.mil TOKENIZER-PROCESSOR (TKNIZE) ... A finite state machine to parse data base statements written in this language c) A user-interface tailored to

3.3.0 PERMANENTLY DEFINED VIRTUAL INFORMATION

Permanent virtual information may be defined through the

Define statement. Such definitions will be stored in a global

dictionary, or so called catalogue, in the form of character

string, and will remain there until explicitly removed or

over-written by a different definition. Examples may be found

within chapter 5.

3.4.0 ADHOC VIRTUAL INFORMATION

Virtual information definitions may be derived for only the

duration of a single transaction. When all statements within

the transaction are executed, the adhoc dictionary would be

erased. Within the transaction, adhoc definition may be cre-

ated, deleted, as well as modified at any time. With this fea-

ture, each transaction would be associated with a catalogue of

its own, and would not interfere with the concurrent activities

of other transactions executing in parallel. At this stage, we

do not support concurrent transactions, but adhoc definiticn

capability is still useful in the principle of transactions.

Surely, the permanent dictionary would also be accessable from

within each transaction.

25

Page 29: mhhhommmmmu EEEEEEohEohmhI - dtic.mil TOKENIZER-PROCESSOR (TKNIZE) ... A finite state machine to parse data base statements written in this language c) A user-interface tailored to

3.5.0 NOTION OF A TRANSACTION

A transaction is a body of executable statements joined

together within a single context. This context is provided by

the adhoc dictionary associated to the particular transaction.

A transaction is created within the transaction buffer, and

will remain there until it is explicitly over-written, erased,

or executed. Merits of this transaction concept are threefold:

a) a group of statements which collectively does a certain task

may be consolidated to exhibit logical unity. b) a shared con-

text may be created and maintained for each transaction, a sign

of transactional modularity and independence from one another.

c) the execution of the consolidated operations in a trans-

action may be put off until a more opportune moment, by which

time new permanent or adhoc virtual information definitions

may be defined either to supplement or to replace existing

definitions.

3.6.0 VIRTUAL ATTRIBUTES

Virtual attributes equated to the results of computational

algorithms acting on available data or of designated indirect

references may be explicitly defined through the Define data

base statement. This feature incorporates the support for Com-

puted Facts as well as for Pseudo Inferred Facts. For instance,

the following is the definition and usage of two virtual attri-

26

-- - -- - -

Page 30: mhhhommmmmu EEEEEEohEohmhI - dtic.mil TOKENIZER-PROCESSOR (TKNIZE) ... A finite state machine to parse data base statements written in this language c) A user-interface tailored to

* butes, income and ship-country, a computed fact, and a pseudo

inferred fact.

Define income as salary - expenses

Retrieve ((teachers)) by ((Vo name, income)

The foregoing retrieve statement returns two vertical col-

umns of data. The first column being teacher's name, and the

second column being their corresponding incomes.

Define ship-country as _ 9 company (country (name )) ;j

Retrieve ((ship)) by ((vO name, ship-country)

This foregoing retrieve statement returns two columns of

data, the first being individual ship names, and the second

being the name of the country to which the ship belongs to. The

entity diagram for this scenario is as follows:

Fig 3.2

-2

__ J ,_ _ - ( . .. . . . .. ...--. ) ,

Page 31: mhhhommmmmu EEEEEEohEohmhI - dtic.mil TOKENIZER-PROCESSOR (TKNIZE) ... A finite state machine to parse data base statements written in this language c) A user-interface tailored to

3.7.0 CONDITIONS ON REAL OR VIRTUAL ATTRIBUTES

Arbitrary conditions on real or virtually defined attributes

may be defined by INFOPLEX users as the shared 'condition' on

their data values from which factored facts may be later con-

structed. For example:

Define old as age > 70

Define rich as assets > 1000000

Retrieve ((people}where(rich and old)) by ((VO} name);

The foregoing retrieve statement would return a list of names

of those people whose age > 70 and assets > 1000000.

3.8.0 VIRTUAL ENTITY SETS

Aside from virtual attributes, we also support a basic notion

of virtual entity sets. We recognize two kinds of virtual enti-

ty sets:

a) Union or intersection of real or previously defined virtual

entity sets based on their real and virtual attribute values.

b) Subsetting of real or virtual entity sets based on certain

conditions on their real and virtual attribute values.

1 29

Page 32: mhhhommmmmu EEEEEEohEohmhI - dtic.mil TOKENIZER-PROCESSOR (TKNIZE) ... A finite state machine to parse data base statements written in this language c) A user-interface tailored to

IFor instance:

Define ClassAB as (ClassA) MU (Name) {ClassB}

ClassAB is defined as the result of a multiple-union opera-

tion on entity sets ClassA and ClassB, based on a common attri-

bute called Name.

Define RichMen as {Men) where (assets > 1000000)

RichMen is defined as a virtual subset of the set Men, based

on the values of its asset attributes.

The complete set of union and intersection operators as well

as the cartesian product operator between entity sets is illus-

trated within chapter 5. Also, refer to chapter 5 for details

of the capability to specify various conditional predicates on

attribute values.

3.9.0 GENERALIZED MACRO FACILITY

Users will be able to define arbitrary definitions and to

give them specific names by which the definitions may be

referred to and later substituted into data base statements.

In this sense, the define statement may be used not only to

29

Ma

Page 33: mhhhommmmmu EEEEEEohEohmhI - dtic.mil TOKENIZER-PROCESSOR (TKNIZE) ... A finite state machine to parse data base statements written in this language c) A user-interface tailored to

define virtual attributes, virtual entity sets, but also ran-

dom definitions as well even if the definitions are seemingly

incoherent without the proper context. When a retrieval state-

i* ment is to be executed, all words within the statement are

first checked against a list of stored definition names; any

matching definition would be recalled from the dictionary and

put in the place of the matching definition name in the

retrieval statement. Chapter 5 includes a detailed description

of such usage.

3.10.0 EXTENDABLE FUNCTIONALITIES

3.10.1 USER DEPENDENT VIRTUAL DEFINITIONS

This particular functionality is not difficult to implement,

but it may be unnecessary at this stage of the project. It sim-

ply would require a separate catalogue for each user which

includes an access control list, proper search rules including

default situations, and adequate coordination and control

* 'mechanisms to manage the various catalogues. It would increase

the cost in terms of time and space efficiency. Thus, we have

not included this functionality in this version of virtual

A information implementation. Nevertheless, if circumstances in

later time are such that the support for user dependent cata-

logues is so desirable as to more than compensate for its cost

of implementation, this functionality may be added readily.

30

Page 34: mhhhommmmmu EEEEEEohEohmhI - dtic.mil TOKENIZER-PROCESSOR (TKNIZE) ... A finite state machine to parse data base statements written in this language c) A user-interface tailored to

3.10.2 INFERRED FACTS OF UNDESIGNATED INDIRECTION

Inferred facts with undesignated indirection, rather than

pseudo inferred facts with designated indirection, is likely

*- to have tremendous costs in system performance whenever it is

to be implemented. As previously stated, this would require

either an exhaustive search or a certain level of artificial

intelligence, both of which require large amounts of resources

in computing power, storage and time. Furthermore, in order to

verify that the indirection the system chooses at each step

along the way is correct, the user has to monitor the computer

decisions interactively; this defeats the original purpose of

not having the user to designate his intended path of indi-

rection. Thus, it seems very doubtful that this functionality

will ever be implemented unless the requirements for user moni-

toring.of the decision process is somehow eliminated.

31

Page 35: mhhhommmmmu EEEEEEohEohmhI - dtic.mil TOKENIZER-PROCESSOR (TKNIZE) ... A finite state machine to parse data base statements written in this language c) A user-interface tailored to

4.0.0 PROGRAM STRUCTURE

Our implementation is done with special attention to modu-

larity. Each primary module incorporates numerous internal

sub-modules whose very existence are not known nor relevant to

the implementation of other primary modules. Aside from the

PL/1 modules, we have designed a data base language, and a

finite state push-down automaton, each of which will be cate-

gorized as a single module as well. Figure 4.1 illustrates the

control structure and data flow of all the modules in our

implementation. Single arrow heads in the diagram represent

control structure transitions and double arrow heads represent

data flow.

4.1.0 MODULE DESCRIPTION

The front-end as designed and implemented in this thesis

includes the following modules:

(1) USER-INTERFACE

(2) BUFFER

(3) ACTIVITY-COORDINATOR

(4) TOKENIZER-PROCESSOR

(5) LANGUAGE DESIGN and SPECIFICATION

(6) FINITE-STATE-PUSH-DOWN AUTOMATON (MACHINE)

32

Page 36: mhhhommmmmu EEEEEEohEohmhI - dtic.mil TOKENIZER-PROCESSOR (TKNIZE) ... A finite state machine to parse data base statements written in this language c) A user-interface tailored to

I ILI

- !Tlbr4,I

I.,~ I

* , "I -

I - . . "• : Z

Page 37: mhhhommmmmu EEEEEEohEohmhI - dtic.mil TOKENIZER-PROCESSOR (TKNIZE) ... A finite state machine to parse data base statements written in this language c) A user-interface tailored to

All programs are written in PL/i under CMS operating system

on IBM VM/370.

4.1.1 USER-INTERFACE

This module is named USINT as a PL/I program. It is currently

the options-main program of the entire virtual information

facility. It diverts control to one of two INFOPLEX implemen-

tations, one of which includes our virtual information facili-

ty, and the other does not. Once the implementation with

virtual information facility is selected by the user, this mod-

ule serves as the communication link between the BUFFER module

which interacts with the user and the ACTIVITY COORDINATOR mod-

ule on the lower level which supervises the execution of data

base statements.

This module has five internal routines:

(1) XBUFF

(2) GETS

(3) REPLACE (internal to RETDSPLY)

(4) RETDSPLY

The XBUFF routine strips individual executable data base

statements one by one off the buffer, and pass them down to the

next level for further processing.

34

Page 38: mhhhommmmmu EEEEEEohEohmhI - dtic.mil TOKENIZER-PROCESSOR (TKNIZE) ... A finite state machine to parse data base statements written in this language c) A user-interface tailored to

The GETS routine is a generalized tool which actually does a

substring command from the first character of a given string to

the first occurrence of a given character. After the execution

of this routine, the portion of the original string up to and

including the given character would be eliminated.

For instance:

strl = 'abcSdef'

Gets (strl, '$') would return 'abc'and the value of strl

becomes 'def'

The REPLACE routine is also a generalized tool to replace all

occurrences of a given varying character string of length two

* or less, by another varying character string of length two or

less.

For instance:

stri = 'abc,def'

Replace (strl,',','55') would change the value of strl to

'abc55def'

The RETDSPLY routine simply displays the current retrieval

statement which is being processed to indicate the correspond-

ence of subsequent outputs to this particular statement. It

35

Page 39: mhhhommmmmu EEEEEEohEohmhI - dtic.mil TOKENIZER-PROCESSOR (TKNIZE) ... A finite state machine to parse data base statements written in this language c) A user-interface tailored to

*makes numerous calls to REPLACE because many characters have

been previously translated to enable the application of the

finite state machine.

4.1.2 BUFFER

. The BUFFER module is named NEWBUF as a PL/l program. It con-

tinuously interacts with the user during a virtual information

session. It has a transaction buffer which corresponds to the

"transaction" concept of virtual information, and which would

accumulate successive data base statements until the entire

transaction is to be executed. It also has an execution buffer

which would be automatically executed upon the completion of a

single executable data base statement. The word "execution" in

the context of this module simply means the return of control

to the module which called it, USER-INTERFACE. When returning

control to the caller, if the transaction buffer is to be exe-

cuted, then the transaction buffer content will be moved to the

execution buffer, and if user requested the termination of the

virtual information session, then a control bit passed to it

from USER-INTERFACE would be set.

In order to facilitate the using of virtual information, this

module incorporates a full screen but simple line editor which

is coupled with the existing buffer commands. Buffer commands

enable the moving of data from buffer to buffer, execution of

36

Page 40: mhhhommmmmu EEEEEEohEohmhI - dtic.mil TOKENIZER-PROCESSOR (TKNIZE) ... A finite state machine to parse data base statements written in this language c) A user-interface tailored to

either buffer content, input of buffer content from a CMS file,

saving of buffer content to a CMS file, and termination of the

* active session. The editor commands are INSERT, DELETE, and

TOPLINE; they enable a simple editing of the transaction buffer

content. The usage of these editor commands and buffer commands

is described in Chapter 5.

In essence, this module establishes the Active Workspace

environment described earlier. It is the primary module of the

external level of the functional hierarchy, developed specif-

ically for the use of virtual information facility on the next

lower level.

This module has the following internal routines:

(1) LDSPCH

(2) BUILDBUFF

(3) TRNSLATE

(4) FINPUT

(5) KBLKS

(6) GETS

(7) NEXTWORD

(8) RMVFBLKS (internal to NEXTWORD)

(9) FSAVE

(10) WAIT

(11) REPLACE

(12) HELP

.37

Page 41: mhhhommmmmu EEEEEEohEohmhI - dtic.mil TOKENIZER-PROCESSOR (TKNIZE) ... A finite state machine to parse data base statements written in this language c) A user-interface tailored to

(13) SETMKS

(14) BDISPLAY

(15) DELTE

(16) WTHNLM

The LDSPCH routine contributes to the format integrity of

each user inputted line by constructing a header which is con-

catenated to the front of each line. The header begins with a

"@" character, which is suceeded by a numeric character string

representing the number of leading blank characters in this

line, and ends with a ":" character. In this manner, all lead-

ing blank characters of each line may be removed. The

advantages of using such a header are twofold; not only can

storage be conserved, but also a fixed structure be imposed on

all user inputs to reduce complexity.

The BUILDBUF routine constructs either the execution or the

transaction buffer one line at a time from each user input

line. Markers on either buffer is repositioned to enable prop-

er editor display. it returns a boolean value of "true" if

there is at least one completed data base statement in the cur-

rent buffer.

The TRNSLATE routine checks for missing quote terminators

for character string constants and back-slash terminators for

comment lines. It also translates ";" characters within com-

38L.__ _ _ _

Page 42: mhhhommmmmu EEEEEEohEohmhI - dtic.mil TOKENIZER-PROCESSOR (TKNIZE) ... A finite state machine to parse data base statements written in this language c) A user-interface tailored to

ments to "%2" and consecutive single quote characters within

character string constants, representing an actual quote char-

acter, to "%l". Such translations are necessary to avoid

ambiguity and complications in the input recognition stages of

the process.

The FINPUT routine serves to input transaction buffer con-

tent from a CMS file whose file name is "file" and file type is

given by the user through the "finput" buffer command. Ori-

ginal transaction buffer content is erased. Characters

"" "@" are replaced by "%4", 1%011 ,1% 3 ", "%5 so that

they would not interfere with finite state machine command '.an-

guage.

The KBLKS routine serves to remove all leading blank charac-

ters from the current input line, and also to keep the number of

them removed in variable "ldspaces".

The GETS routine is a general tool as described earlier with-

in the USER-INTERFACE module.

The NEXTWORD routine returns the sequence of characters in

the input line up to but not including the first blank charac-

ter. If a blank character is not found, the entire input line is

returned. Comments are automatically removed and are not

recognized as part of an input line.

3Q

Page 43: mhhhommmmmu EEEEEEohEohmhI - dtic.mil TOKENIZER-PROCESSOR (TKNIZE) ... A finite state machine to parse data base statements written in this language c) A user-interface tailored to

The RMVFBLKS routine is internal to NEXTWORD; it serves to

remove the blank characters preceding each word in the input

line. Its name stands for remove-front-blanks.

The FSAVE routine is the counter part to the FINPUT routine.

It writes the current transaction buffer content into a CMS

file whose file name is "file" and file type is given by the

user through the "fsave" buffer command. The characters trans-

lated by FINPUT and TRNSLATE routines are restored before they

are written into the CMS file.

The WAIT routine serves as a time delay to hold messages to

user on display terminals long enough to be readable by the

human eye.

The REPLACE routine is a general tool as described earlier

within the USER-INTERFACE module.

The HELP routine displays a brief explanation of each buffer

command to the display terminal.

The SETMKS routine sets or resets markers in either the exe-

cution or the transaction buffer for buffer-display purposes

of the full-screen line-editor. The name "setmks" stands for

"set-markers".

40

Page 44: mhhhommmmmu EEEEEEohEohmhI - dtic.mil TOKENIZER-PROCESSOR (TKNIZE) ... A finite state machine to parse data base statements written in this language c) A user-interface tailored to

The BDISPLAY routine serves to display the contents of both

the execution and the transaction buffer. It implements the

full-screen characteristic of our line-editor.

The DELTE routine implements the delete-line function of the

editor. The transaction buffer markers are properly reset

after each invocation of this routine.

The WTHNLM routine serves to verify the logical correctness

of editor command correctness. If the parameters are out of

current buffer boundaries, then the routine will return 'O'B

4.1.3 ACTIVITY COORDINATOR

This module coordinates all activities on the level of virtu-

al information processing. It directs the moving of program

control through various modules on this level. A number of

debugging tools which prints out various trees, token chains,

and tables are included within this module, and can be used in

times of need by inserting a "call" statement any where within

the module.

This module contains the following internal routines:

(1) GETS

(2) PRINTT

* 414

Page 45: mhhhommmmmu EEEEEEohEohmhI - dtic.mil TOKENIZER-PROCESSOR (TKNIZE) ... A finite state machine to parse data base statements written in this language c) A user-interface tailored to

V

(3) PRINTM

(4) PRINTX

(5) PRINTE

(6) PRINTR

The GETS routine is a general tool as described earlier with-

in the USER-INTERFACE module.

The PRINTT routine is a debugging tool which can be used to

print the chain of input tokens.

The PRINTY routine is a debugging tool which can be used to

print a snap shot of the finite state machine.

The PRINTX routine is a debugging tool which can be used to

print the execution tree.

The PRINTE routine is a debugging tool which can be used to

print the entity set table.

The PRINTR routine is a debugging tool which can be used to

print the revised entity set table.

4.1.4 TOKENIZER-PROCESSOR

42

____.. .. ..________

Page 46: mhhhommmmmu EEEEEEohEohmhI - dtic.mil TOKENIZER-PROCESSOR (TKNIZE) ... A finite state machine to parse data base statements written in this language c) A user-interface tailored to

This module serves to tokenize retrieval statements, and to

execute "define" , "adhoc", and "listdef" statements. It is the

only module of the virtual information facility which communi-

cates with the dictionary of virtual information definitions,

besides USER-INTERFACE which makes one call to dictionary for

initialization. When tokenizing each retrieval statement, vir-

tual definitions are recalled from the dictionary whenever

appropriate and substituted directly into the retrieval state-

ment.

This module contains the following internal routines:

(1) GETS

(2) NXTKSTR

(3) RMVFBLKS (internal to NXTKSTR)

(4) TOKI (internal to NXTKSTR)

(5) DEF

(6) BDTKCHN

(7) MSG

(8) LISTDEF

(9) DEFDSPLY (internal to LISTDEF)

(10) REPLACE

The GETS routine is a general tool as described earlier in

the USER-INTERFACE module.

43

- - I - . . . .. . . . . ._ "

Page 47: mhhhommmmmu EEEEEEohEohmhI - dtic.mil TOKENIZER-PROCESSOR (TKNIZE) ... A finite state machine to parse data base statements written in this language c) A user-interface tailored to

i The NXTKSTR routine is the core of the tokenizing process; it

recognizes from the input stream the next token in the form of a

character string. Each token is a separately recognizable

entity. This routine is called repeatedly by the BDTKCHN rou-

tine which builds an entire chain of tokens.

The RMVFBLKS is the same routine as described in the BUFFER

module.

The TOKl routine is the main body of the NXTKSTR routine. It

recognizes the next portion of the input string, which is to be

transformed into a separate token.

The DEF routine serves to execute the "define" and "adhoc"

data base statements. It creates and modifies virtual informa-

tion definitions in the dictionary of virtual definitions.

The BDTKCHN routine builds an entire chain of linked tokens.

Each retrieval statement is transformed to such a chain of sep-

arate tokens before further processing.

I[

The MSG routine outputs a message line to the terminal and

prompts the user to press the "enter" key to continue.

44

.4

Page 48: mhhhommmmmu EEEEEEohEohmhI - dtic.mil TOKENIZER-PROCESSOR (TKNIZE) ... A finite state machine to parse data base statements written in this language c) A user-interface tailored to

I

i

The LISTDEF routine executes "listdef" data base statements.

It would recall the definition in the dictionary which is to be

listed, and output the definition to the terminal.

The DEFDSPLY routine is internal to the LISTDEF routine. It

serves to process a stored definition for terminal display.

Retranslation is needed to reconstruct those original charac-

ters which have been previously translated.

The REPLACE routine is a general tool as described in the

BUFFER module.

4.1.5 LANGUAGE DESIGN and SPECIFICATION

This module incorporates the design and formal specification

of a data base language which defines and retrieves virtual

information. Chapter 5 is devoted exclusively to explaining

and describing this module.

4.1.6 FINITE STATE AUTOMATON (MACHINE)

This module serves to parse the data base statements written

in the language illustrated in Chapter 5. It consists of a set

of Match-Action-Nextstate rules which is one of the inputs to a

generalized parse program written by Peter Lu. A change in the

grammar of our data base language readily corresponds to

45

t!

Page 49: mhhhommmmmu EEEEEEohEohmhI - dtic.mil TOKENIZER-PROCESSOR (TKNIZE) ... A finite state machine to parse data base statements written in this language c) A user-interface tailored to

changes in the rules of this finite state machine; thus, free-

ing us from changing the parse program itself. Chapter6 is

devoted exclusively to explain the workings of these rules.

4.2.0 INTERNAL GLOBAL VARIABLES

USER-INTERFACE MODULE:

execbuff -- execution buffer

trnsbuff -- transaction buffer

firstlast -- passed to BUFFER and used to indicate when

to terminate end of session.

line -- used to hold user-input line

BUFFER MODULE

execbuff, trnsbuff, firstlast (same as in USER-INTERFACE)

prstline -- current input line, char(80)

strnsp -- prstline, stripped of leading and ending spaces

cplnvar -- strnsp, char(80) varying

ldspaces -- number of leading spaces on input line

key -- current input word to be investigated

ACTIVITY-COORDINATOR MODULE

46

Page 50: mhhhommmmmu EEEEEEohEohmhI - dtic.mil TOKENIZER-PROCESSOR (TKNIZE) ... A finite state machine to parse data base statements written in this language c) A user-interface tailored to

DICTIONARY -- virtual information dictionary

ENTITY -- entity set representation

TOKEN -- token representation

MACH -- finite state machine representation

XTREE -- execution tree representation

XCFINGE -- entity set table representation

UNIT -- current data base statement to be processed

TKLSPTR -- pointer to the list of input tokens

GO -- indicator to proceed with beyond the

tokenizer-PROCESSOR stage

TOKENIZER-PROCESSOR MODULE:

unit -- current data base statement

tklsptr -- pointer to list of input tokens

diction -- dictionary

go -- indicator to continue processing

(set only for retrieval statements)

kind -- numeric indicator for arithmatic and

string constants.

word -- first word of data base statement

47

Page 51: mhhhommmmmu EEEEEEohEohmhI - dtic.mil TOKENIZER-PROCESSOR (TKNIZE) ... A finite state machine to parse data base statements written in this language c) A user-interface tailored to

*1

* "5.0.0 LANGUAGE ILLUSTRATION AND SPECIFICATION

This section contains an illustration and a formal

specification of the data base language implemented on

the virtual information facility, as well as the buffer

commands which are implemented to provide an interactive

environment in which virtual information processing

may be continued.

5.1.0 DATA BASE STATEMENTS

5.1.1 DEFINE STATEMENTS

A user may define the character string x to be a

macro definition of the character string y by the

following statement:

define x as y

def x as y

For instance:

define currentyear as birthyear + age

define old as age > 60

define employee as (worker)

def a as 2+3+4/5*8

def inches as 2.54 * centimeters

Using define statements to simulate functions:

define sum#a#b as #a + #b

49

Page 52: mhhhommmmmu EEEEEEohEohmhI - dtic.mil TOKENIZER-PROCESSOR (TKNIZE) ... A finite state machine to parse data base statements written in this language c) A user-interface tailored to

This specifies a function with two arguments, #a and #b.

The value of sum#2#4 when evaluated would be 6.

To remove previously defined definitions:

define age remove

define age remov

define a remove

5.1.2 ADHOC STATEMENTS

Adhoc statements are similar to defines statements;

the only difference lie in the target catalogue

identity. Adhoc statements operate on the adhoc

dictionary, and define statements operate on the

permanent dictionary.

adhoc x as y

adhoc curentyear as 1982

adhoc sgfhg as kruilko

adhoc age rem ;

adhoc avg#x"y#z as (#x + #y + #z) / 3

In both define and adhoc statements, virtual definition may

be defined on top of other virtual definitions; in our

implementation, we allow a maximum of 10 nested levels of

virtual information definition. Thus, if one defines a

recursive definition, our system would terminate the

entire process of replacing definition names by their

associated definitions by the eleventh attempt in replacing

49

Page 53: mhhhommmmmu EEEEEEohEohmhI - dtic.mil TOKENIZER-PROCESSOR (TKNIZE) ... A finite state machine to parse data base statements written in this language c) A user-interface tailored to

VT

the same definition name.

5.1.3 LISTDEF STATEMENTS

These statements list the stored definitions in the

dictionary by name; the search order is:

adhoc --> permanent.

listdef age

listdef employee

listdef sgfhg

5.1.4 RETRIEVE STATEMENTS

Our retrieve statements are powerful enough to

retrieve the following kinds of virtual information

from either real or virtual entity sets:

a) computed facts

b) implied facts

c) factored facts

Computed facts are those information derived from an

algorithmic computation on existing data; implied facts

are those information derived from indirect associations;

factored facts are those instances of a particular

group of facts which share a certain condition on their

attribute values.

We support the following kinds of computational

50

4

Page 54: mhhhommmmmu EEEEEEohEohmhI - dtic.mil TOKENIZER-PROCESSOR (TKNIZE) ... A finite state machine to parse data base statements written in this language c) A user-interface tailored to

Cw

operators with four levels of precedence, left to right

within each level of precedence, and together with

parenthesized precedence capability:

- I , lowest order of precedence

(plus, minus, and concatenate)

(binary infix operators)

-/ , next order of precedence

(multiplication and division)

(binary infix operators)

next order of precedence

(exponentiation operator)

, - , highest order of precedence

(arithmatic pre-operators)

Aside from these built-in operators, we also support a

number of built-in functions as enumerated below:

functions with no arguments:

date usage -- > nextdate = STR (DATE , 2:'$') + 1

first, one would use the STRING function to

obtain the relevant portion of the value

returned by the DATE function, and then this

value is incremented by 1.

A date in the system may be stored in the form

month$date$year, or any other pre-determined

.4

Page 55: mhhhommmmmu EEEEEEohEohmhI - dtic.mil TOKENIZER-PROCESSOR (TKNIZE) ... A finite state machine to parse data base statements written in this language c) A user-interface tailored to

manner. The STRING function is very much suited

for the getting of relevant portions of data

stored in this form.

Functions with one argument:

These functions operate on entire entity sets; in this

sense, they are vertical operators, not of the unilateral

kind which we are usually familiar with.

Any valid expression may serve as an argument to

built-in functions.

MAX(y) usage --> max (length + width + hight)

refers to that particular instance of the

entity set whose dimensions have a greater

sum than all other members of the set.

retrieve ( { employee) where ( salary = max (salary)

by ( {v0' name);

gets the name of the employee who earns

the highest salary.

SUM(y) usage -- > where ( sum ( y ) 100 )

yields true if the sum of all instances

of y in the current entity set equals 100.

MIN(a+b-5)

for each member of the set, the value of

argument expression is first calculated,

52

Page 56: mhhhommmmmu EEEEEEohEohmhI - dtic.mil TOKENIZER-PROCESSOR (TKNIZE) ... A finite state machine to parse data base statements written in this language c) A user-interface tailored to

then the minimum of them all is taken.

ABS(x+y~z)

returns the absolute value of the argument

expression for each member of the entity set.

SGN(index) returns -1 if argument is negative

returns 0 if argument is zero

returns 1 if argument is positive

SUM(x!2)

sums up the squares of the variable x, yielding

one single value.

POS(v) -- returns boolean value for each

instance of attribute v in the

entity set.

ZER(x) -- returns boolean value for each

instance of attribute x in the

entity set.

Functions of more than one argument:

STR ( b ,nth occurrence of 'x', mth occurrence of 'y'

returns a substring of b from the nth occurrence of 'x'

to the mth occurence of 'y' exclusively.

usage > STR ( b , 4:'x' , 5:'y' )

53

Page 57: mhhhommmmmu EEEEEEohEohmhI - dtic.mil TOKENIZER-PROCESSOR (TKNIZE) ... A finite state machine to parse data base statements written in this language c) A user-interface tailored to

a retrieve statement is of the following basic form:

retrieve ( list of real and/or virtual entity sets

separated by commas and each with an optional

predicate clause

by ( entity set designation

list of items to be retrieved

The first set in the list would be known as {vO)

The second set in the list would be known as (vI)

The tenth set in the list would be known as {v9)

A maximum of ten such sets on this level is permitted.

We hope to demonstrate the functionality of the

retrieve statement through the following examples:

retrieve ( (esl) ) by ((VOl x)

gets all those "x" attributes of entity set "esi"

retrieve ( (esi} ) by ( (vO) x+3

computes and returns all x+3 instances of entity

set ESi which has the attribute X.

retrieve ( {esl},{es2) ) by ( (vO) max (x) )

by ( (vl) y*4 , min (y*z) )

First, it gets the instance of "esi" 's attribute "x"

which has the highest value of all instances of "esi" 's

attribute "x",

54

Page 58: mhhhommmmmu EEEEEEohEohmhI - dtic.mil TOKENIZER-PROCESSOR (TKNIZE) ... A finite state machine to parse data base statements written in this language c) A user-interface tailored to

Second, it gets the "y*4" elements of entity set

"es2", y being an attribute of "es2" , then it gets

the instance of "es2" 's "y*z" which has the minimal

value of all other instances of "es2" 's "y*z",

"y" and "z", both being attributes of "es2"

retrieve ( (esli where ( ( ( xl < x2 ) and ( x3 = x4 )

{es2) where ( yl = ( 1,2,3,4,5 ) or y2 = y3{(vO)) where ( xl I x3 = str ( b,4:'$',5:'$' )))

by ( (vO xl, x3)

by ( (vl} yl, y2)

A complete set of predicate conditions on real and

virtual entity sets is supported with "and" , or'

"xor" and " connectors, with < " "="

" =" "-<" and "->" relators. the default order of

precedence is from left to right unless otherwise

indicated by the use of parentheses.

For instance: the following are equivalent conditions:

xl = x2 and x2 > x3 or x3 < x4

(xl = x2) and (x2 > x3) or (x3 < x4)

((xl = x2) and (x2 > x3)) or x3 < x4

(((((xl = x2))))) and x2 > x3 or (x3 < x4)

Each where clause is attached to the entity set

specified immediately prior to the clause itself; the

55

Page 59: mhhhommmmmu EEEEEEohEohmhI - dtic.mil TOKENIZER-PROCESSOR (TKNIZE) ... A finite state machine to parse data base statements written in this language c) A user-interface tailored to

only restriction on the kinds of entity sets allowed

to have where clauses attached to them is that they

are not one of the following:

{vO, (vI,(v2},{v3),{v4)

{v5), (v6), fv71, (v8), {v9)

This is so because these entity sets only refer

to some other entity set which was already specified.

According to this principle, the following entity

sets may have associated predicates because they are

themselves the specification of new virtual

entity sets:

((vO} {((v9))

....................(vg)}

The second entity set in the foregoing retrieve

statement has an associated predicate which specified

yl = (1,2,3,4,5); this predicate requires yl

to be of either one of the constants within the

enclosing set of parenthesis. however, when using

this kind of comparison, we make the restriction

that all values which appear in the enclosing set of

parentheses must be either an arithmatic constant

or a string constant.

The third condition clause in the foregoing

retrieve statement illustrates the use of the string

functions; the function call is attempting to return

56

4 ,-. - , "

Page 60: mhhhommmmmu EEEEEEohEohmhI - dtic.mil TOKENIZER-PROCESSOR (TKNIZE) ... A finite state machine to parse data base statements written in this language c) A user-interface tailored to

.. .. ..II

the substring of b, from the 4th occurence of

the '$' character to the 5th occurence of the '$'

character. The predicate would yield true if the

*results of concatenating xl and x3 is equal to the

retrun value of that function call.

retrieve ( { {esl) mi (x,y,z) {es2} )

by ( {vO weight )

This statement retrieves all instances of the weight

attribute of the virtual entity set composed of the

"multiple union" of real entity sets esl and es2, based

on the common attributes "x", "y", and "z".

Each virtual entity set, enclosed by a set of left and

right braces, may itself be composed of two other virtual

entity sets as the result of a set operation, and each of

these two component entity sets may also be composed

of two other virtual entity sets as the result of a

set operation, and each of these component entity sets

so on. In this manner, virtual entity sets may be

built very quickly one on top of another, each with its own

set of predicate conditions to be met.

Five set operators are supported between two entity

sets: they are, multiple union, multiple intersection,

single union, single intersection, and cartesian product;

namely, MU, MI, SU, SI, and CS. The semantics of these

57

4

Page 61: mhhhommmmmu EEEEEEohEohmhI - dtic.mil TOKENIZER-PROCESSOR (TKNIZE) ... A finite state machine to parse data base statements written in this language c) A user-interface tailored to

?1°

operators are described in the co-thesis by Peter Lu.

usage--> ((esli SI (x) (es2))

((esl} su (y,z) {es2))

((esl) MU (yz,zl) (es2))

The operands of MU, MI, SU, and SI can be a list of

attribute names separated by commas, but the operands

to CS must be two in number and the first one must be

preceded by a " " sign to indicate its cartesianess.

((esl) cs (id,class) (es3})

An arbitrary WHERE clause representing a predicate

condition may follow each and every kind of prescribed

virtual entity set.

{{esl) where (color = 'red') cs (id,class)

{es3) where (num> 7) ) where (size < 5)

5.2.0 BUFFER COMMANDS

These interactive commands may be issued by the user via

a terminal session with the virtual information facility.

They are the means by which an interactive environment is

constructed in which the data base commands may be executed.

The buffer is divided into an execution and a transaction

buffer. an adhoc dictionary is built for the duration

of each transaction in which many data base statements

may be strung together and executed sequentially. Thus,58

Page 62: mhhhommmmmu EEEEEEohEohmhI - dtic.mil TOKENIZER-PROCESSOR (TKNIZE) ... A finite state machine to parse data base statements written in this language c) A user-interface tailored to

within a transaction a user may operate on either the

permanent dictionary shared by itself and any other

transaction executed before or after it, or the adhoc

dictionary which is for its own exclusive use.

A completed data base statement in the execution buffer

will automatically trigger the execution of that

statement; therefore, the execution buffer is not suited

for the stringing together of multiple statements.

Each buffer command may be entered from within either the

transaction buffer or the execution buffer, and may be

recognized by two or more initial characters of its full

name. furthermore, the contens of the execution buffer and

at least 10 lines of the transaction buffer will

always be displayed on the terminal.

5.2.1 COMMAND SYNTAX

(1) FINPUT Istarg

This command will read the contents of the

cms file whose file name is "file", and file type

I is whatever is entered as "lstarg", into the

4 transaction buffer. The original content of the

transaction buffer before the execution of

this command will be erased.

(2) FSAVE Istarg

59

Page 63: mhhhommmmmu EEEEEEohEohmhI - dtic.mil TOKENIZER-PROCESSOR (TKNIZE) ... A finite state machine to parse data base statements written in this language c) A user-interface tailored to

This command will write into the cms file whose file

name is "file", and file type is whatever is entered as

"istarg", from the contents of the transaction buffer.

Upon completion of the command, transaction buffer

* content will be empty.

(3) TRANSACT

This command lets the user enter the transaction buffer.

(4) ENDTRANS

This command lets the user terminate the transaction

buffer and enter the execution buffer.

(5) TERMINATE

This command terminates the virtual information

facility and returns control to cms.

(6) RUNTRANS

This command executes the contents of the transaction

buffer statement by statement.

(7) DODELETE

This command does the same as "runtrans" except

that upon its completion, the transaction buffer

contents will be erased.

(8) ERASETRANS

This command erases the contents of the transaction

buffer.

604

- '

Page 64: mhhhommmmmu EEEEEEohEohmhI - dtic.mil TOKENIZER-PROCESSOR (TKNIZE) ... A finite state machine to parse data base statements written in this language c) A user-interface tailored to

(9) KILLEXEC

This command erases the contents of the execution buffer.

(10) HELP

This command gives a brief description of all buffer

commands.

(11) INSERT lstarg 2ndarg

This command would insert a line of text into the

transaction buffer. the first argument is a

destination of the line number within the buffer

after which the inserted line is to be inserted,

and the second argument is the text to be inserted.

(12) DELETE Istarg

This command deletes a line from the current

transaction buffer, and the exact line number is

specified by the first argument.

(13) TOPLINE Istarg

This command specifies the starting line number of

the ten transaction buffer lines which are always

displayed, and that number is designated by the

first argument.

5.3.0 FORMAL BNF DESCRIPTION OF DATA BASE STATEMENTS

<defstmt>::= "DEFINE" j "DEF" name defopt

61

Page 65: mhhhommmmmu EEEEEEohEohmhI - dtic.mil TOKENIZER-PROCESSOR (TKNIZE) ... A finite state machine to parse data base statements written in this language c) A user-interface tailored to

2 <defopt> < "AS"l <***>

< "REMOVE" I"REM"1 >

<adhocstmt> "ADHOC" name defopt

<liststmt> "LISTDEF" name;

<retrstmt> "RETRIEVE" (vsets )byspec

<vsets> < "T" vsetsl ""{"where" (cond) >

< "T" vindrs "T" >

{/vsets

<vsets.l> name

combs et S

<combsets> < "(" vsetsl "} where" (cond) >

< "(C' vindrs 'T" >

{setop vset.s2}

<vsets2> < "{" vsetsl ""("where" (cond )}>

< "C" vindrs "T" >

<vindrs> < "Vo"lt "V111 I V2" I "V311 I V4" I "V5"l

I< "T" vindrs >I

<setop> <CS) I <ncs>

<ncs> < "1MI"l I "lsi" i"MU"t It 1 suIf >

(reflist)

<reflist>::= refi l refl}

(CS> "lCS" varef ,varef

62

Page 66: mhhhommmmmu EEEEEEohEohmhI - dtic.mil TOKENIZER-PROCESSOR (TKNIZE) ... A finite state machine to parse data base statements written in this language c) A user-interface tailored to

<exp> <(exp)> Iexp-infl Iexp-inf2 Iexp-pre

2 Iexp-pwr Iexp-prim

<exp- "nf1> -exp infi-op <(exp)> I exp-inf2 I exp-prim

Iexp-pwr

<exp-inf2> exp2 inf2-op <(exp)> I exp-prim Iexppwr

<exp2> <(exp2)> Iexp-±Jnf2 1exp-pre

I xp-pwr Iexp-prim

<infl-op>::= +I-I "<inf2-op>::= * /

<exp-pre>::= pre-op < exp-prim Iexp-pre Iexp-pwr >

<pre-op> +I-

<exp-prim> ref const

<exp-pwr>:: exp-prim !< exp-prim Iexp-pre >

<ref> refi funref

<refi> { Ivaref

<varef> name {(varef))

<const> fixed Iinteger

<fixed> integer (integer1

<digit> 0 1 1 I2 I3 1 4 1 5 I6 17 I8 I9

<integer>::= digit Idigit integer

<funref> :=singfun I strfun

Idi <singfun>::= funame (x

K 'funame> "MAX" I"MIN" I"ABS" I"POS"I "SGN" I"ZER" I"SUN"1

<strfun> "str" (strarg)63

Page 67: mhhhommmmmu EEEEEEohEohmhI - dtic.mil TOKENIZER-PROCESSOR (TKNIZE) ... A finite state machine to parse data base statements written in this language c) A user-interface tailored to

<strarg> ::= exp , exp ( : exp ) strargl

<strargl>::= { < @ exp { exp I > < , exp ( exp I

********** ***************** ********* ********************* ******* *

<cond> < ( cond ) > I condl

<condl> : smpcnd I < cond condop cond2 >

<cond2> :: smpcnd < ( cond ) >

<smpcnd> :: < ( smpcnd ) > I < ( not ) ( smpcnd ) >

I < exp relop exp >

<not> : <> not

<condop> : "AND" j "OR" I "XOR"

<relop> = > I<

I { not 1=

I { not ) >

{ not <

*******************************************************k**********

name any character string of length less than 17

and composed only of the following characters:

abcdefghijklmnopqrstuvwxyz$*&_¢?"

5.3.1 BNF SUPPLEMENT

* comments are enclosed within "\" characters

* quote characters within string constants are represented

by two consecutive single quote characters

string constants are enclosed by "'", single quote

F 4

Page 68: mhhhommmmmu EEEEEEohEohmhI - dtic.mil TOKENIZER-PROCESSOR (TKNIZE) ... A finite state machine to parse data base statements written in this language c) A user-interface tailored to

characters

I ***> designates any arbitrary character string

-,' * single line comments enclosed by "\" characters are permitted

before, after, and within each data base statement,

. as well as before and after each buffer command line.

They are eventually removed, and are not recognized

as part of any input line.

• the system makes no distinction between lower and upper

case characters.

.1

t 65

Page 69: mhhhommmmmu EEEEEEohEohmhI - dtic.mil TOKENIZER-PROCESSOR (TKNIZE) ... A finite state machine to parse data base statements written in this language c) A user-interface tailored to

6.0.0 FINITE-STATE-MACHINE (PUSH-DOWN-AUTOMATON)

In this chapter, the Finite-State-Machine used to parse data

base retrieval statements would be briefly described. A

Finite-State-Machine consists of a number of states, one of

which is a start state, and one or many of which may be an end-

ing state. There also is a pointer which would point to the cur-

rent word being examined on the user given statement being

processed. In our case, the user input is always a retrieval

statement written in the data base language presented in

Chapter5, and each word would always be a single token along

the token chain to which the original retrieval statement has

already been transformed to.

Each state within the machine has a set of match-next state

rules, and collectively the union of these sets of rules regu-

late the behavior of the finite-state machine on any given

input. Each state attempts to find a match between the current

word and the match section of any one of its rules, and if a

match is found, then control is passed to the state identified

by the next state section of the matching rule, and the input

pointer points to the next word of the statement being proc-

essed. If the end of input is ever reached, and control happens

to be within an ending state, then the machine halts and is said

to have accepted the statement which it had Just processed.

66

Page 70: mhhhommmmmu EEEEEEohEohmhI - dtic.mil TOKENIZER-PROCESSOR (TKNIZE) ... A finite state machine to parse data base statements written in this language c) A user-interface tailored to

I

Our construction of such a finite-state-machine went

straight on to meet a number of problems. First, such a machine

has no provisions for any processing except for moving from

state to state. Thus, we augmented our design to a

Push-Down-Automaton which is a finite-state-machine with aux-

iliary memory and data movement capabilities. in fact, in order

to generate an execution tree and various tables from a given

retrieval statement, we had to augment the processing ability

of the automaton by a set of action routines, and transform the

format of state-rules to a tri-tuple consisting of a match sec-

tion, an action section, and a next state section.

The auxiliary memory we have chosen for the automaton is in

the form of two stacks, an operator stack and an operand stack.

The ratch section of each rule has provisions to match either

the current word on three different sources, the input token

chain, the top of stack #1, and the top of stack #2. Actually,

when the source is the input token chain, an added ability to

match for a given class of tokens as well as a specific token is

available. The action section of each rule has provisions for

pushing and poping the current input token, onto or off either

stack #1 or stack #2, and for invoking other action routines

which generates an execution tree and various tables from the

current elements on both stacks, and modifies the current con-

tents of the stacks. The next-state section of each rule

contains the state ID number which identifies the next state to

67

Page 71: mhhhommmmmu EEEEEEohEohmhI - dtic.mil TOKENIZER-PROCESSOR (TKNIZE) ... A finite state machine to parse data base statements written in this language c) A user-interface tailored to

1

which control would be passed to after the proper action rou-

tines in the matching rule have been executed. Our machine is

deterministic in nature; by this we mean that for a given com-

bination of input token, top of stack #1, and stack #2, there is

at most one nextstate from which control may go to after the

current state. A non-deterministic machine would have been

condensed, but also more complex -and harder to implement

because of the need to backtrack over decision points.

The construction of this push-down-automaton is logically

divided into two parts, the writing of the

match-action-next state rules, and the writing of a program

which takes these rules as an input and sets up the proper envi-

ronment in which data would be matched, actions would be per-

formed, and next-states would be go to, exactly according to

the specifications of the prescribed match-action-nextstate

rules. The front-end of the virtual information facility, as

presented in this thesis, is responsible for the writing of

these match-action-next-state rules, and the implementation of

these rules is a responsibility of the back-end.

6.1.0 CONFIGURATION

Action routine implementations are part of the back-end

written by Peter Lu in his concurrent thesis. These programs

are within the PARSE module as illustrated in figure 4.1 . As

68

Page 72: mhhhommmmmu EEEEEEohEohmhI - dtic.mil TOKENIZER-PROCESSOR (TKNIZE) ... A finite state machine to parse data base statements written in this language c) A user-interface tailored to

part of the front-end, the match-action-next state rules are

within the FINITE-STATE-MACHINE module and currently reside in

CMS file "file machin" A DEFMCH module is written to estab-

lish the machine environment, taking the contents of "file

machin" as input, and the PARSE module, when called upon, acti-

vates the finite-state-machine.

6.2.0 MATCH-ACTION-NEXTSTATE RULES

Match-Action-NextState rules is a 3-tuple of information.

The first component presrcibes what to match for a certain ele-

ment on either one or more than one of the following sourses,

the input stream, top element of stack #1, and top element of

stack #2. The second component is a sequence of action routine

invocations; these routines are to be executed whenever the

matching component of the same rule matches. The third compo-

nent is a state-number representing the next state to which

control is to go when the action routines in the same rule have

been executed.

The three components of each rule is separated from each oth-

er by the "\" character as illustrated in the following:

\ match component \ action component \ nextstate number \

69

Page 73: mhhhommmmmu EEEEEEohEohmhI - dtic.mil TOKENIZER-PROCESSOR (TKNIZE) ... A finite state machine to parse data base statements written in this language c) A user-interface tailored to

Furthermore, since these rules prescribe the transitions

from state to state, they are referred to as the "transiion

rules", and the full specification of a transition rule is as

follows:

t \ match component \ action component \ nextstate number \

The specification of a state is accomplished first by writing

the following to indicate the identity of the state:

s \ state number \

and then by a listing of the match-action-next state rules

which belong to this particular state. The sequential order of

rules in this list can not be inter-changed, because when con-

trol comes to each state, the rules will be trid sequentially

in the order of their position on the list. Thus, a sample state

specification is as the following:

s \25\

t \retrieve \del \26\

t \ ( \ pop,l \ 27 \

t \ + \ pop,2 \ 36 \

The foregoing rules would first try to match the word from

the input, if it is matched, then the action routine "del",

7 1

Page 74: mhhhommmmmu EEEEEEohEohmhI - dtic.mil TOKENIZER-PROCESSOR (TKNIZE) ... A finite state machine to parse data base statements written in this language c) A user-interface tailored to

delete input token, would be executed, and then control would

go to state number 26. If the first rule did not match, then the

second rule which attempts to match a "(" character on the

input would be tried. If this rule matches, then the "pop" rou-

. tine would be executed, and stack #1 would be popped, and con-

trol would go to state number 27. If the second rule did not

match, then the machine would try the third rule, which matches

for the "+" operator on the input stream, if it is matched, then

stack #2 would be popped and control would go to state number

36. If none of the rules for a the current state matches, then

the machine would signal premature termination, which means

that the input is invalid, and diagnostic messages would be

sent to CMS file "file error". State number 0 is the final state

of the machine, and if control is ever passed to this state,

then the input is valid, accepted, and successfully processed;

the associated execution tree and entity set tables would have

already been generated and available for use by the following

stages of the virtual information facility.

The match component of each rule is composed of zero to three

separate parts, each of which is separated from the other by a

"," character. The first part represents the input source, the

second part represents the source from stack #1, and the third

part represents the source from stack #2. If non of the three

parts exist, then that particular rule would match everything

and anything, and the corresponding action routines would

71'Ii

Page 75: mhhhommmmmu EEEEEEohEohmhI - dtic.mil TOKENIZER-PROCESSOR (TKNIZE) ... A finite state machine to parse data base statements written in this language c) A user-interface tailored to

always be executed if the machine ever tries to match that

rule.

For instance:

s\l\

t\a,b, c\del\2\

The foregoing rule would match simultaneously a character

"a" on the input stream, a character "b" from the top of stack

#1, and a "c" character from the top of stack #2. All three

sources must be matched before the corresponding action rou-

tines may be executed. If any of the sources does not match,

then this entire rule is not matched, and either the next rule

in the sequence would be matched or the machine would signal

premature termination if there are no more rules to be matched

for this state. Thus, at most three sources may be matched in

the match component of the rule, and at most one single token

may be matched on any one source.

The action component of each rule may contain calls to more

than one action routine. These action routine invocations are

written in sequential order and are separated by the "I" char-

acter; these routines would be executed in the order of there

appearence in the action component. For instance:

s\5\

72

L " _ * . . ... . . ,, ,, - . ... - ~ . b--. -,, -'S 1, , -

Page 76: mhhhommmmmu EEEEEEohEohmhI - dtic.mil TOKENIZER-PROCESSOR (TKNIZE) ... A finite state machine to parse data base statements written in this language c) A user-interface tailored to

t\ + \ push,2,i@ I del I pop,l \ 6 \

The foregoing rule would match the "+" character on the input

stream, and then execute the three action routines in sequen-

tial order. First, it would push the " " character on to stack

#2, as specified by the first routine call, then it would

delete the current character on the input stream, thereby

advancing the input pointer to point to the next input token,

and then it would pop stack #1, popping off stack #1's top ele-

ment.

In order to facilitate the matching of a group of symbols,

not necessarily all of the same classification, we have devel-

oped the concept of a "cluster"; a cluster simply is a union of

one or more prescribed symbols which may be matched under one

cluster name. All cluster names begin with a "@" character, and

they provide an added convenience for the making of transition

rules. For instance, an arithmatic cluster may include the +

and - characters, and be named "@sumop". By using the name

@sumop in the match component, we may match either the + or the

- characters. We currently have the following groups of clus-

ters:

@sumop --- +

@sumop> + , - , * , / ,

@multop --- * ( /

73

Page 77: mhhhommmmmu EEEEEEohEohmhI - dtic.mil TOKENIZER-PROCESSOR (TKNIZE) ... A finite state machine to parse data base statements written in this language c) A user-interface tailored to

@multop> *,/,!

@conc I

@conc> --- , + - , /

@virt --- vO, vl, v2 ,v3 ,v4,

vS , v6 , v7 , v8 , v9

@rel > <

@cmp and , or , xor

@setop --- cs , mu , mi , su , si

It was mentioned earlier that a rule may match from the input

source a token of a specific classification; to do this, the

rule indicates the class of characters it would be matching for

by writing a ":" character and followed by the class represen-

tation character. There are altogether eight different classes

of tokens, namely the classes A , N , D , 0 , B , Q , M , and S

Characters belonging to class A are the following:

abcdefghijklmnopqrstuvwxyz

Characters belonging to class N are the following:

0123456789

Characters belonging to class D are the following:

Characters belonging to class 0 are the following:

74

Page 78: mhhhommmmmu EEEEEEohEohmhI - dtic.mil TOKENIZER-PROCESSOR (TKNIZE) ... A finite state machine to parse data base statements written in this language c) A user-interface tailored to

Characters belonging to class B are the following:

Characters belonging to class Qare the following:

Characters belonging to class M are the following:

I &>#$

Characters belonging to class S are the following:

stream, does nothing, and then passes control to state 4.

s\3\

t \:M\\ 4\

The parsing of any language may sometimes be facilitated by

the creation of sub-pasers which parse a subset of the

language. The usefulness of this idea is demonstrated by our

using of the sub-routine concept in the finite-state-machine.

Two sub-machines were written, one to parse expressions, and

one to parse entity set conditions which calls on the

75

-- - 9

Page 79: mhhhommmmmu EEEEEEohEohmhI - dtic.mil TOKENIZER-PROCESSOR (TKNIZE) ... A finite state machine to parse data base statements written in this language c) A user-interface tailored to

expression parser. The idea is to pass control to the

-+ sub-parser, and leave subroutine return command and address on

2 top of stack #2 before entering the sub-parser. When the

sub-parser finds a negative state number as the next-state, it

should have that return command and address available on top of

stack #2, and should pop that element off stack #2, and then

pass control to the state identified by that command. For exam-

ple, the following rule passes control to a sub-parser which

starts on state 30, and also specifies the return state number

as 80 when the sub-parser finds a negative-state number in the

nextstate component. With this strategy, the last rule which

indicate the successful parse of the sub-parser must have a

negative state number in the nextstate component.

s\20\

t \ d \ del I push, 2,subr:80 \ 30 \

6.3.0 ACTION ROUTINES

The following action routines are written within the

back-end of the facility and are available for use in the

action component of the match-action-next-state rules:

Routine Usage Semantics

76

Page 80: mhhhommmmmu EEEEEEohEohmhI - dtic.mil TOKENIZER-PROCESSOR (TKNIZE) ... A finite state machine to parse data base statements written in this language c) A user-interface tailored to

POP pop, l pops stack #1

pop,2 pops stack #2

PUSH or P push,l,i@ pushes the input

p,l,i@ token onto stack #1

push,l,c@ pushes the input token

concatenated by a

*-1 and its classification

onto stack #1

push,2,i@:2 pushes the input token

concatenated by ":2"

onto stack #2.

Frequently, this is used

to associate the expected

number of operands to the

operator which is being

pushed onto the operator

stack.

DEL del advances the input pointer

to point to the next input

token.

77

Page 81: mhhhommmmmu EEEEEEohEohmhI - dtic.mil TOKENIZER-PROCESSOR (TKNIZE) ... A finite state machine to parse data base statements written in this language c) A user-interface tailored to

Non-stack related routines:

GENNODE or GD

generates an operator node with its specified

* number of children which are found on stack #1

as a partial execution tree. The address to

the partial tree just generated is placed

back on top of stack #1.

INDX

adds the first level of indirection to a

data element

ADDON

adds an additional level of indirection

VIRTX

exchanges the addresss of the indicated

virtual entity set

MULX

generates a multiple "OR" node

78

Page 82: mhhhommmmmu EEEEEEohEohmhI - dtic.mil TOKENIZER-PROCESSOR (TKNIZE) ... A finite state machine to parse data base statements written in this language c) A user-interface tailored to

ATTWHR

attaches the address of a condition to

its associated virtual entity set

GENENT

generates a new virtual entity set

VIRTA

adds the cuurent virtual entity set to

an entity set table

The non-stack relate routines listed above have much to do

with the internal workings of the back-end and are left to be

more precisely explained by the back-end documentation.

6.4.0 LISTING

The rules written in CMS file "file machin" are not readily

readable because of its syntax; a FORMMCH program is available

to format it to a readable form as shown in a listing of the

rules on the following pages:

79

.4,~~4'.'**-.*,1-

Page 83: mhhhommmmmu EEEEEEohEohmhI - dtic.mil TOKENIZER-PROCESSOR (TKNIZE) ... A finite state machine to parse data base statements written in this language c) A user-interface tailored to

V- 00 co 4 C- c C ~ C C4~cr ifc 0 D~C4 wl V m- --4- t- m- -T

I- .

-4 ccP-

'ulU

in z z u iun - - .u- -

'U~~ -)N i4NC 04 ~ 0 11 (-1 a~ - --

-. 4I. C-4 C4 NWW W N a

Aw

-V -4 C 4C 0 C" N " 0C ;

:7 00AO1to - 0.0.0 0 *Aj A A 0 -- i 'a zn no ~ o in===n m =w=w=wD w = u: z

*U *x 0. CL a ~ 0 CLa .Q .a .aI .a

j 80

Page 84: mhhhommmmmu EEEEEEohEohmhI - dtic.mil TOKENIZER-PROCESSOR (TKNIZE) ... A finite state machine to parse data base statements written in this language c) A user-interface tailored to

N11

-u -1 C'-N - ,N~ N ~ N

00

*N 0

LOI CA OLn

I @0 00 O

*~3 -x 0..

z -. 0 N N14 mW a IN NU ON U W UW w

12 6- 0 0 0 coc coo cc 0

40 -- -- 4 C4 04 Coo N

14 LM (z 0-- 0 U 0) m (

in M -j N . M W 0L L

2- U) U U)0 J ..J .J; ~ .. J. LJ L J

i.E 0. N 9N - C4 4 C 4 04 i 04 0 04 N00 000 00 0

~ 81

Page 85: mhhhommmmmu EEEEEEohEohmhI - dtic.mil TOKENIZER-PROCESSOR (TKNIZE) ... A finite state machine to parse data base statements written in this language c) A user-interface tailored to

~~71

LUJ

LU4

Z uj-zj

Zz w WN LU LUW LU LU LUAW

LU LU -LU* LU" -- LU" R- LUR LU w LU L0 0 R 0 -R De Ro 0ie N -0 N4 0 0

-- "-oI -- O--

! OLU U W.LU C4 N N CN - N C4 N -- OON CN N f

V!Roo- 0.1C CZin 0) LA)U tf) U).) LO 0 ) 0.ZZ u v)ui u Ui U) ) i

U. 0.0C0. CL.I. 0..M.0. IL.0. C 0C (0 . .0 .0 0. CL Q I L L 0

i a. N: N a f I x 0) 0CC

LU LL ID WU) W; W w U, W L- -n ...) .9. m )V r )m r m4 n(

*LUL Ln402 f

US S L LU ~ L L **. U L LU82

Page 86: mhhhommmmmu EEEEEEohEohmhI - dtic.mil TOKENIZER-PROCESSOR (TKNIZE) ... A finite state machine to parse data base statements written in this language c) A user-interface tailored to

0 CIO-C7

Cl I tn Lag CLn O tn 9) ILn

C4 L

0z

LIn

4A -ji j --

-z 0- In c

4 20

0 -- 0 0 0 0

LiIL 0 -- -

0; 1 0- *i Cli3f x .- .

- 4 z- w w UU UL C 0 .

6 C .0 CL *-.

w~ A 0~

4K w La I-L .. co 9 1n In In

I.. I -j In iZ iJi

1-041 1 1- 1-W 1i 1-

LiiU 0. 00 0 . .0 w 00 0.. 0. 0.0

I- C4 04 Id - r. 0 0£L- 0i 1 C4, eo 9o( - In 0-4 0

.~ 4 ~ ~ qi- n In In I In ndig

83S

Page 87: mhhhommmmmu EEEEEEohEohmhI - dtic.mil TOKENIZER-PROCESSOR (TKNIZE) ... A finite state machine to parse data base statements written in this language c) A user-interface tailored to

00 00 0 0-£ C4 m in, V W £0 -40 In M2 w 0 7.i cu 0 CD CI 401 0 '£0

inau

-Z 0i

ZZ £0 -i -j*0 w ..a J w A

41.)~a *. C .J -- C;

In -- 39 .. I.- w -3

-U i- -7 3 aM 0 mZ I- ---4 .-. M =- =M ~w aM -

0. -' 04 m V in to0 >0r

toI- I n w0 ww1-0 4CLatCVa- a I-I ac n0

i- iaMCl i iUN i- i N -- iMl iaM - , - W . . M .U W . a M.. Lu . i

84

Page 88: mhhhommmmmu EEEEEEohEohmhI - dtic.mil TOKENIZER-PROCESSOR (TKNIZE) ... A finite state machine to parse data base statements written in this language c) A user-interface tailored to

-o C4 Or w u- r-O 0 M Nl V' In Uco f- r,. t- p-C' r- r-t I- r- 0I cc U cc CID 0

*j 0

uj 0Z)I a.

I -j. u a - 0 a JRLU ~ ~ ~ ~ ~ 20u L -U u A ' a

1- 0 - 0 >I-

I2-w v j V Vw ;-~

'0 -; C; C4 0 C4 .- I -J 0 -- O -

viI W A 0 -7 0n V2 0 0 0

1-4 *i 2 = Z0 I'd :3 5 SLL. - IL L L IL -9 - U -

M 0

~ W ~ 0 11

w - - C- Is C- C- C- w w- us C- 0u w 0A

1-0 0- P- I.- I-1 - 1- . . C - 1- . C; I" I" 1.- (4 1- V- o-

85

Page 89: mhhhommmmmu EEEEEEohEohmhI - dtic.mil TOKENIZER-PROCESSOR (TKNIZE) ... A finite state machine to parse data base statements written in this language c) A user-interface tailored to

40( or- NNc

dl)w Ow0 co O0~ 0D 0- 0- 00 00

Wj a 0

W4 --

20ZW 0-- 0 0-- a oj z -

.4 -j- -i k 4W-- .J RJ a to

4 .. J. "I R W ..

C4 0- C4 0 w4 C4 Ni Ni Cx0 0 xJ -- 0o -- -i .:E ID .3EIV) -I0 0j 0 W W. (A0 a 07nML L L( .F

U IL. CL - - *- IL CL- L0 L CLC. C L MC

0. 80.-

11 C4 4C C4 Ni OCI i Ci

9- Z 0 )w ow- LO. I- C. r. Ci.

4 cc 0 44 go do -o 4 1 Mo 9m - 0 W4 m ~ 0 00 0 -00vi0. 0 0.0. n 0 0. 0. 04 0in &M0 0.. 00 00

.86

Page 90: mhhhommmmmu EEEEEEohEohmhI - dtic.mil TOKENIZER-PROCESSOR (TKNIZE) ... A finite state machine to parse data base statements written in this language c) A user-interface tailored to

cz .-

00 0 04 04 0to f N

go 10 co

A C4 jC. 0.. C4 C4 2- Z 4

I.0 m m~ 0

U,~C CL CL U, U , , U6U ,

C4 i

Page 91: mhhhommmmmu EEEEEEohEohmhI - dtic.mil TOKENIZER-PROCESSOR (TKNIZE) ... A finite state machine to parse data base statements written in this language c) A user-interface tailored to

z L

--I -4 N (N ( N ( It ( N C4( C4 C4( (C 0

11

m. v to 0 1-

LO V;ll I t4i I

tw-- m IW ;m- 9 1-

1-44

, 88

i • ED ii : ) -ii I ( " t ED. EI I

Page 92: mhhhommmmmu EEEEEEohEohmhI - dtic.mil TOKENIZER-PROCESSOR (TKNIZE) ... A finite state machine to parse data base statements written in this language c) A user-interface tailored to

.. ... . ... 0 W .. . - 0 . . .. .. . . .. ..

- -!

MAE

22

z4

'0 C; vi 04 C!-"

MA VA US

z -n - " - " - - , n 0 A m

-z Cd -.de C C d d C CL

.4 (

It ow C4 '0" M* t- in 6 0

l W MA# A M A * A M A M A MA in MA * M

bO .*0 in - -~ - -o~ . s- -l - 4 -~ i6 .

4Cd4040 0 0 4 4 4 ~ 4444~ 4 449

Page 93: mhhhommmmmu EEEEEEohEohmhI - dtic.mil TOKENIZER-PROCESSOR (TKNIZE) ... A finite state machine to parse data base statements written in this language c) A user-interface tailored to

7.0.0 MAJOR DESIGN ISSUES

The following are some of the decisions which we had to make

in the design of the virtual information facility.

7.1.0 FORM OF STORAGE FOR VIRTUAL DEFINITIONS

How can virtual definitions be stored? We had two viable

alternatives. One way is to store the definitions just as they

are, in the form of character strings, and when in use, the

definition would be substituted within the actual data base

retrieval statement in place of the virtual definition name.

The alternative to this strategy is to parse the definitions

ahead of time, generate the associated execution tree and enti-

ty set tables, and when in actual use, the partial execution

tree would be simply attached to the main execution tree as an

extended subtree, and the main entity set tables would simply

be augmented to include the partial table built from the defi-

nitions.

The method of parsing the definitions first is similar to the

process of compilation. When in actual use, previously defined

definitions need not be processed again and again. The method

of storing the definitions as they are is similar to the proc-

ess of interpretation. Each time a definition is used, the

90

Page 94: mhhhommmmmu EEEEEEohEohmhI - dtic.mil TOKENIZER-PROCESSOR (TKNIZE) ... A finite state machine to parse data base statements written in this language c) A user-interface tailored to

entire process of parsing, tree building and table generation

would have to be repeated.

Storing definitions as they are enhances the flexibility of

virtual information. Definitions may be created, modified, and

even deleted with great ease and efficiency. Furthermore, it

eliminates the need to rebuild itself when users request a

listing of the stored definitions. It also would enable a gen-

eralized macro facility in which not only legitimate and

coherent definitions may be stored, but also the seemingly

illogical and incoherent definitions as well.

Parsing -he definitions as soon as they are defined is not an

easy task. Many times, without the proper context in which the

definitions are to be used, the associated semantics are not

always clear. Even if we can get around this problem by

restricting the potential contexts in which each definition

may be used, we still would be encounter complicated problems

in frequent tree manipulation. Re-shaping an execution tree is

a very "messy" task, and would be prone to erroneous branch

connections; traversing a huge tree is also not a reasonably

efficient operation.

Thus, mainly for the foregoing reasons, we have decided on

the first strategy, storing them as they are until invocation,

to store virtual definitions.

91

I.

Page 95: mhhhommmmmu EEEEEEohEohmhI - dtic.mil TOKENIZER-PROCESSOR (TKNIZE) ... A finite state machine to parse data base statements written in this language c) A user-interface tailored to

7.2.0 PARSER STRUCTURE

Two strategies were given serious consideration for the

parsing of data base statements written in the language speci-

fied in Chapter5. One method is to construct a

FINITE-STATE-MACHINE which includes a set of

match-action-next-state rules that correspond to the grammar

rules of the data base language. In this manner, these

match-action-next state rules are inputs to the actual parser

just like the data base statements which are to be parsed; with

this approach, changes in grammar rules readily correspond to

changes in the machine rules. The other method is to construct

a conventional parser in which grammar rules are part of the

parser program itself.

The finite-state-machine strategy has a highly modular char-

acteristic and gives added flexibility to the data base lan-

guage in terms of modifiability; however, it would be the first

of such machines ever written by the author. The decision was

made in favor of the finite-state-machine because the definite

gains of this approach seem to surpass the potential for fail-

ure of its implementation.

7.3.0 PROGRAM CONTROL STRUCTURE

92

Page 96: mhhhommmmmu EEEEEEohEohmhI - dtic.mil TOKENIZER-PROCESSOR (TKNIZE) ... A finite state machine to parse data base statements written in this language c) A user-interface tailored to

A decision was made to implement a centralized and horizontal

control structure for the passing of program control from one

to another. The alternative is to build a vertical control

structure in which modules are nested one within another, and

control may propagate many levels deep before suddenly jumping

out to the top. The centralized control structure features an

activity coordinator to which control must return to from each

module before it is passed to another. Although the vertical

approach may seem more natural, the horizontal approach is more

adapted to the idea of a single virtual information level with-

in the hierarchical design of INFOPLEX. Furthermore, the

horizontal approach contributes more to program modularity

with its regard for each module as a separate and un-nested

entity. For these reasons, the decision was made to build a

centralized and horizontal control structure.

7.4.0 INTERACTIVE EDITOR

Consideration was given to the question of whether or not to

build an interactive, full-screen editor in real-time, similar

to a miniature EMACS or XEDIT editor as part of the

user-interface developed for the virtual information facility.

The seriousness of the consideration remained questionable to

this day. The argument against it is that the buffer program

already supports the capability of inputing the transaction

buffer content from an arbitrary CMS file; a user of virtual

93

Page 97: mhhhommmmmu EEEEEEohEohmhI - dtic.mil TOKENIZER-PROCESSOR (TKNIZE) ... A finite state machine to parse data base statements written in this language c) A user-interface tailored to

V

information may readily use the XEDIT editor available on CMS

to edit their transaction stored in a CMS file, and later input

that transaction to the transaction buffer through the FINPUT

buffer command. Our interactive, full-screen line editor would

take some effort to develop, and still would not be nearly as

powerful as XEDIT. In other words, resources may be better uti-

lized if spent on other areas of the virtual information

facility. The argument for such an editor is simply that it

would provide the added flexibility to change modify buffer

contents from within the virtual information facility.

Finally, a decision was made to build a primitive line editor

with display capabilities. This is a compromise between the two

extremes; not much resources in terms of man-hours would be

spent building such an editor, and it would give users of vir-

tual information an added flexibility and convenience in being

able to edit their transaction buffer content from within the

facility.

7.5.0 LANGUAGE DESIGN

A decision was made to support infix arithmatic and string

operators instead of operators in prefix or lisp notation.

Although infix operators give rise to a language more difficult

to parse, they are more user-friendly. Also, a decision was

made to support the capability of explicitly over-riding the

94

Page 98: mhhhommmmmu EEEEEEohEohmhI - dtic.mil TOKENIZER-PROCESSOR (TKNIZE) ... A finite state machine to parse data base statements written in this language c) A user-interface tailored to

AD-A11S HE ALFRED P SLOAN SCO OF MANAEMENT CAMBRIDGE MA FP/ /2VIRTUAL INFORMATZON FACILITY OF THE INFOPLEX SOFTWARE TEST VEMI-ETCjU)NAY U J LEE NOOO9-61-C-0663

UNCLASSIFIED NilO!2Ol@-2 O NL

EhmhnmmmhhlImhllmlllllllllllllllllb

Page 99: mhhhommmmmu EEEEEEohEohmhI - dtic.mil TOKENIZER-PROCESSOR (TKNIZE) ... A finite state machine to parse data base statements written in this language c) A user-interface tailored to

natural operator precedences by the use of parentheses in

arithmatic, string, as well as boolean expressions. This capa-

bility makes more difficult the parsing process, but gives much

added power and flexibility to the language. In essence, the

added advantages of infix operators and use of parentheses for

specification of precedence are considered well worth their

cost of implementation.

J

95

Page 100: mhhhommmmmu EEEEEEohEohmhI - dtic.mil TOKENIZER-PROCESSOR (TKNIZE) ... A finite state machine to parse data base statements written in this language c) A user-interface tailored to

8.0.0 CONCLUSION

Progress was made steadily and swiftly all through the first

two months of design and implementation. Then, as precious time

passed by each day, increasing hours of work were required for

the prompt completion of all thesis objectives. Eventually,

all available time was devoted to thesis work and efforts on

academic courses became nearly non-existent.

Finally, the complete design and an initial version of the

implementation were completed and integrated with Peter Lu's

back-end to set up the first virtual information facility in

operation on the INFOPLEX software test vehicle, a software

simulation of the INFOPLEX data base computer. An extensive set

of improvised test cases were written and tested on the facili-

ty. The internal interface to the back-end, namely, the

instructions issued within the finite-state rules does to con-

form to expectations. Although the back-end is not yet able to

integrate with the lower level of INFOPLEX to access real data,

it is able to generate correct information requests based on

the execution tree and entity set table established through

finite-state-machine instructions which are issued within the

front-end.

The results of the implementation give support to the designdecisions which were made, especially the decision to con-

96

Page 101: mhhhommmmmu EEEEEEohEohmhI - dtic.mil TOKENIZER-PROCESSOR (TKNIZE) ... A finite state machine to parse data base statements written in this language c) A user-interface tailored to

struct a finite-state machine and to keep virtual definitions

as they are. Most thesis objectives were achieved except for

the need of more rigorous test cases to establish the integrity

of the facility. Instantly, this facility, when eventually

integrated to the next level of INFOPLEX hierarchy, would

greatly extend the power and capability of the data base.

97

Page 102: mhhhommmmmu EEEEEEohEohmhI - dtic.mil TOKENIZER-PROCESSOR (TKNIZE) ... A finite state machine to parse data base statements written in this language c) A user-interface tailored to

Bibliography

I. Harry R. Lewis, Christos H. Papadimitriou,'Elements of the Theory of Computation'Prentice-Hall Software Series

2. David Gries,'Compiler Construction for Digital Computers'John Wily & Sons

3. Jeffrey Folinus, Stuart E. Madnick, Howard Schutzman,'Virtual Information in Data Base Computers'Center for Information Systems Research, M.I.T.

4. A Klug, D Tsichritzis,Multiple View Support Within the Ansi/Sparc FrameworkCenter for Information Systems Research, M.I.T.

5. Meichun Hsu'FSTV - The Software Test Vehicle for the FunctionalHierarchy of the INFOPLEX Data Base Computer'Center for Information Systems Research, M.I.T.

6. Thomas A. Standish'Data Structure Techniques'Addison-Wesley Computer Science Series

7. Aho Ullman'Principles of Compiler Design'Addison-Wesley Computer Science Series

8. Tak To'SHELL: A Simulation for the Software Test Vehicleof the INFOPLEX Data Base Computer'Center for Information Systems Research, M.I.T.

9. Chat-Yu Lam, Stuart E. Madnick'INFOPLEX Data Base Computer Architecture'Center for Information Systems Research, M.I.T.

10. Bruce Blumberg'INFOSAM - A Sample Database Management System'Center for Information Systems Research, M.I.T.

98

*1

Page 103: mhhhommmmmu EEEEEEohEohmhI - dtic.mil TOKENIZER-PROCESSOR (TKNIZE) ... A finite state machine to parse data base statements written in this language c) A user-interface tailored to

Iv

Ii4!

99

Page 104: mhhhommmmmu EEEEEEohEohmhI - dtic.mil TOKENIZER-PROCESSOR (TKNIZE) ... A finite state machine to parse data base statements written in this language c) A user-interface tailored to

* 0000000000000000000800000000000000000008000000

w o- woam o rw 4r o4

tos Z

SM In

o w. N. S

-, 0.Z o- ;N 0

us i

-2 -- *4 ; , SzU.j IV 9- 0 0 i1 0.9

Z - .4 Z .ze .- VA 9a' .USlw 20 aw 1 4 - ' c t

A C) S.. o .- M

o . In vi -m 50I

99-~~- j -6 4 9- 4 W 4

z A- 1- -1 -1 1- 1- 0w 1-- 054~ ~ - 4 z Z .

z 4-X 088 9- S0 0 a. .R 0U9: - adU8 a

- w 8.5 9--00

Page 105: mhhhommmmmu EEEEEEohEohmhI - dtic.mil TOKENIZER-PROCESSOR (TKNIZE) ... A finite state machine to parse data base statements written in this language c) A user-interface tailored to

008000080000f4G ~v w w M ~ rw

*1wVL ) Dt )(

. .. .. .. ,§

zW

C4 W W~~w cc

a..-

u~ ~ z I. S

IA I- MA U O. -- &- 2 0. - ' O

U U..w- 2 Ui

P - Z- a (

~~ -" - 4 x~

101

Page 106: mhhhommmmmu EEEEEEohEohmhI - dtic.mil TOKENIZER-PROCESSOR (TKNIZE) ... A finite state machine to parse data base statements written in this language c) A user-interface tailored to

888888888885588585888

z

w 0--

X P- - P.- 0

wZ PI, lz w _9- -w l isZ i

I- -9- Pa ,

m4-e102

Page 107: mhhhommmmmu EEEEEEohEohmhI - dtic.mil TOKENIZER-PROCESSOR (TKNIZE) ... A finite state machine to parse data base statements written in this language c) A user-interface tailored to

C00000 00000000000200,w Ln r-ccm C M W 0 r- m Nmmomm-~Q 000000-0-:

0

w us

zzww

0 z Inx -4

Z 004

w UN z

.- 0x w mo.I-ut2

z 0. 0 020

(w10-

Page 108: mhhhommmmmu EEEEEEohEohmhI - dtic.mil TOKENIZER-PROCESSOR (TKNIZE) ... A finite state machine to parse data base statements written in this language c) A user-interface tailored to

000000800000~00008000000000~0000000

000000000000000;0000000000000000000

. . * . ..

In v ! - -*. - z

7 z ~ -- 2 U.

Six Cm *0WWWWW4--- w__ U L

-. >.,a . .. n9-cL. 9X-- a

.- e 9-- - - - - * L

On (N < 00 00 -w~r z -. Z4 4 .(B I

Q4 x x P- 0 m CLw 2 .04 - ZI *i ww w 0.w 9-

CL 0 L UIL I&W U.m 0 .. U )

0.00 - w jz1-1- - a 0 0 QJI U.W IL..- w-0ww N 0444IL K 2 in

.jZZIL44w-jw-juuuuQ" 0 wwonunw..f-

c0&: CG 3J w w 1

104

Page 109: mhhhommmmmu EEEEEEohEohmhI - dtic.mil TOKENIZER-PROCESSOR (TKNIZE) ... A finite state machine to parse data base statements written in this language c) A user-interface tailored to

00000000008~oo000000000000

a >>.n

LUo-. 0 A0

zz -E

> > ~~-JZ 01 Z

-Z L I. w . : 1

E z Cc - -3 - ~ .

CL :,v 49U A tc - -I -j i- W w u

OMMOj 0.E

dc-U.I.105

Page 110: mhhhommmmmu EEEEEEohEohmhI - dtic.mil TOKENIZER-PROCESSOR (TKNIZE) ... A finite state machine to parse data base statements written in this language c) A user-interface tailored to

0000000003000000000300000000000000000000000000000000000000

00l~ 0 0 00 0 0D0-800000 00000-8808

w w w* w .ww w w w w w w

* I4 0 -1 0 In* LL 4u 0n -9 M -~n= w En z I I an 2- 0*l.Ln4 2wD Un 0 I- -

* n4 4 IU =- z o

* ~ c 0W.L ZZ.:. 4-04 x~ u

*-0 L * wL z . WI- Wj M U 0 . a L C. U1-z Z~..4 ~ 0 (0~Z 0 1)

Z m 144 COX4 . 0 In Z a 0- M- Zn w- - ZW* "U 1Z -wg- X Z a.. &- I . U , La n >p-X IIW CZ aZ 'n I- 0 . M ~4 4I.4-1- x 0 3 w U W4U0 u- *" coC r WW

3" A3w. ( U. L0. *I uw- U n 1-X .. j

in CO~ M nw- zww4wo- M4 w..j Z4K 4 w CC. a U0Ia. w- w LU .n 2)c w~ a - w ~ coJi L 4w0n . InIL un~ 0 IL LLat U 0-I- L ; V-1)4

V) uInz 04 Za..MWOLa~wz- W...ju 2 I> 0In * -J~l M U1. O 4~ a ~ ~ J ;. w _j :. w 8

j d wmI j MZ-Z wux1-1-Z33 21 -1 >4a..; =ZI~- 0-0~l z4 4 wI.w>~~ M 41Xn Z~ O 1 0Zn n I 2 > 1J

11. 4 In> 4 0-0 .V); >-- z ow In W > > O0 *0 2 Na.. xZIn. a-U toI Z -wZ- w Vn 00 0 -- ;: o -E >

(n 4 o LLW 1 .9 Lnnow 0 LL 4W w I a . -)I- -- 0 IS4 -4 Z I1Unww<U<XO...l _2E Mw .- C0e- cc

: (n M In d ~4z= ii w .1 -i Z. -1 1- U ZMO-04 I -i .IC'2Oc. * pn1- -- X<On..IO 4u 11 11-4w cc-

M * n)*O UZM =won2uU4-U- w4 .-.- 1 >U1 -- 0 'I 11U.. w 4 M ~ww 3 .U. Z->M oU 4 w ;a

II. 3* 4 -- w wLIa.3Z n 1 9u w 4z~a~' '0 4 cc9zL6 0 M>2.j Mnl ZWO i aw o~ M~ - w ~ 0. 'A) W. 02coW (A II4.4 M =2 .. 12 OWOY I.-. LL -0 1--I -*U44w-Xu *Z CO M w4 aW..-J-W wIn V U20-C MM0U -904 - L.LAWN-.-Z --CL - WMw 4-0(1-4 P- 2LI-x1-0-)02n *04 440221-WM20.WM-M) 4 24~)( ZOM L wZ 4 ... 39 -w .J WI~.. IUw..Ic4w I-CC0Cw:IV).Fw.MIXw * 1o mj~z D o- 101 -u-0 C3 U uUmwt-, *4 - .. awu-0a- Z-...a ow.

z M2.> -4 - WM 34 Cf. 0V- -')1 *1- &U...~iw 1Z40.40Z.j4040 M 010 2 z 0.0 1 *1 w -0LL L6.IU a -.In0, *I- .InUv0Inw02M Z- - 0: w

o ~ a 41U1 0U 0 20. U1. 31- .. w .-cZ.'. . . .M. 0 .441< w. -Ow w--9 wi >x wI uC 001 ..

M L 0... x- ww 0.w 3-w -..I2 z Wo 0 L' Inn~~ 0. 1-4 ;1.:Z ZM I0a~u 1Z AWCU W 1-1 w -- Z01 Z. .I --- nL On 0( UwC .C -jL I- I.-n

!24 1..- 0 ww wouw40 j 0.1Zu 0-I-0. .a.oww.3.U - Z -ZZ * ox22 OIXI-M-2wM O-OZ M&0-0.U--- M..0--- I-W-J.a

I -.- .w- .- w 000 00- w 00 L .1 c t

106

Page 111: mhhhommmmmu EEEEEEohEohmhI - dtic.mil TOKENIZER-PROCESSOR (TKNIZE) ... A finite state machine to parse data base statements written in this language c) A user-interface tailored to

0 w w w Dw 0wr - If, - r- r- r- , cow c oCI oco0 w 0~O00 0 4 M ci04000 00 00 00 00 0------

w ~ j w W W WW W w LLu LU Lu uL uL UL uL uLuL uL uL uL uL L uL uL uL uL LLu Lu Lu ww w ww w w Lu LuWW W WW W WW L LLA LLu W Luzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzz

3)

zLu -Uwj V) m L 0

w * IL

0u 0 z -IU-I

z * 4 -.. c

A ui U a aw UA CO f U. L"W2 Lu z 4c . j j

.5 . C L in ( I z

0. - 4 0- .'r m in

Ln 0 V) z 04W"- .

L uJ 0Z -. u 0 w U- 0 *L U n,- ;*2A

.- 9- LU w ..- 0 2U. f m U:IV 0 Lu U.X -0 *U I-- lO .- U. n D 0 c j.:4

x 0 u - OM'0 cc U 20

LuI: : C t Lu Z-- cc cc lU 15 on.. 0~$I-- W1 ". atU z .9. 16 4

Uj a. u 0 n m N - 0 =~W : w > >>

E0 0 Kz 0-00 4 ii.z J o 4 0Jw~.

0 M00-0 0 Lu Lu,-0Lu00In - -1 00 00- 00- .. 0U a

Z W P- I- o--lCLui C CL 0- Ixz 107 i

Page 112: mhhhommmmmu EEEEEEohEohmhI - dtic.mil TOKENIZER-PROCESSOR (TKNIZE) ... A finite state machine to parse data base statements written in this language c) A user-interface tailored to

0008Q 0000000000?00000000 000000000 00000000 000000000 80

0000000000000000000000000000000000000000000000000000000g3333333 333333 333333X3333 333 3)33333333

zzzzzz z z zzzzz zz zzzzzzz zzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzz

z0 LU

Z Z -

ZjZ I Zxz x

I - J - I

w LnLU z - I

gn LL. L

u , z Z a .J 1 . 1

LU 49 in 2 i

. w . t. 0. ..

cc U. 99 to. IX -B

'0 -j z - - z - .

vi n A -00 'nM ( 0 L. n 0- 0. Uzin 0ZWI. V)Z I- cc 0 Z

coU9 Z U- UOU Z wU oI -

4 1 In 11, 1 N , I 11 I U. o- ILL. U. j 1j D * I.J- 0 1w . 0U. 4Z .. a'. n L U I L a'- 'U. LU L~'. LU U. W U. U. WU. J --.J-- U. I * 0 M~

0 L60Z 00 0 0. aia0 0 0:) nz OZO j' . 0 j .WEL"9-0 low cc n Ic 0 1 1 11060 1 ca I-4'-K 11- in > x LL

.q!4 - i xi. 2X W 0. (06 .Jo* ; .JCK U.

108

Page 113: mhhhommmmmu EEEEEEohEohmhI - dtic.mil TOKENIZER-PROCESSOR (TKNIZE) ... A finite state machine to parse data base statements written in this language c) A user-interface tailored to

000001?o

000000000000

zzzzzzzzzzzz

1,

z

I" _

22

I-77

Page 114: mhhhommmmmu EEEEEEohEohmhI - dtic.mil TOKENIZER-PROCESSOR (TKNIZE) ... A finite state machine to parse data base statements written in this language c) A user-interface tailored to

000000000000000§000000~t-~C, 0 -C0W4O C4C0q f0

000000000000000000000333333333333333WLU a LJ L 3j 3l 3 3wwwwwwwwwwwwwwwwwuzwwww

ZZZZ ZZZ ZZZZ ZZZ ZZZ

z

U

I-.

01 000

0. ,- I

CZ4 W. U

CLOi : -CL

0 ~ ~ 49c p0N -- w!- O I -

- 4W0 M u4 1U" Mi .4 0- Q z

wo * 4 1-. *m a&~m a-oa& ua

Wa hi - O.U,

.O .04 0

u~JU atj uin.. ta. 0- -J C CL

*i01 in go 8W 4i W

110

Page 115: mhhhommmmmu EEEEEEohEohmhI - dtic.mil TOKENIZER-PROCESSOR (TKNIZE) ... A finite state machine to parse data base statements written in this language c) A user-interface tailored to

C4 C

000 -. .

* 00000000000000j00000000000000323 339 3333X3X3 2 XX X3

*A A WW WW W Lfj w w WWWIU W Jw ww w ww w w ww U W .zzzzzzzzzzzzzzzzzzzzzzzzzzzzz

-~0

'0I-.-

mujxw --

V) fl lalizw~ I.-L 0

cc -9 1). + m

EZ m N - 1 WW Z U

a.Ig 2aJ x.U W WM. I- X-~ -. -. 9J.J 99 >A

m U.o uu. -Z ~ .. :;I- U w4co 0 _j j = jCc1C

Page 116: mhhhommmmmu EEEEEEohEohmhI - dtic.mil TOKENIZER-PROCESSOR (TKNIZE) ... A finite state machine to parse data base statements written in this language c) A user-interface tailored to

00008 0ooo8000000008 000000000 000000000 0000o~eoo

33)3333 33)3 33 3 3 3 33333 33333X33 X3X333333 33333z~zzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzz

(N4

z +.

a 4K

I-SU V) I.-VI x z .0z -.3z P-- - .E

(A: .- z 0.e .

c -0 '- C, I- zP-LQ

0 - mi z xz0Z .5 LA 0t o O- 2xw I- W u

La . SM 1.M C - 0Z 0 zMZ 0 I-S . SM 05-LL

SMockz 4 $.- 0- - 0 .-- nzI 0.- _2 Ou Z a 2 o .

I -m u Z - 0 - 9 * - u~ 0 .Mg - SM:iw c O 9 +- uSu w -jZ >t0- V) I. U I0 - ak m af: > wsf0 tLai .9 Z * o Z (n .-. I-- C W .4 0. Vs I. SM O)ls-

PO 1* 0 2V 2- -f - b- 0 UiUOV .tS 1- 0.v 1 *.*1W 0 UI. .. 0-*S

uZ4.Z *. !! . -1- - ~ - - 4 * tP.-U.-O 0 mi '-f (atl-f z 0i- w > V -ft,- M tS

U1 * .'I- 43- "C- )-. Z x N (zft9LI ~ ~ ~ l - l..t - 00 :;a. :4-- -: >.'-- I- waD -J V t-. I )-.* M 0-t~i 0.-C 5ZZ -UV-U 11 a '00 CL. .- -- 0 nsca - 4~. z If.JM I -- - z .u Q U V) tV i..(L0

00W3 i 0 8 CL I.-. SMZ c SM u Wa. 0 CL z8tS mu I - OOCDI I-SI a =- ~USM a.

:34 CL.- i- '-Z ' IWW- .. - I" &nI - P.- -W1.- 0 0 4 ow . Ox- o 0 -xICW a.L P- m U SM u 0 Z

4VI C, V I i~ > II -- ot-. UNJ IC *tI Z - 0 1-a .co c w -- 0t X wi 00i MO 0Ix 0 Z..0 -- SMO fLaV 1 -- t uN x0 0 SM uL-I - K v) wWSM0

z~~~~ ~ ~ ~ 0 V- LUL xI i- w A L -.

0-303.0, x-w 0 ..J.I-0J0 CLVi

20C..5 I - 0 &020J 0

CL-If 3V 0 I-- "I-- - tw O O A0C jUz u -z I IA V

Im 112P4 :2.

Page 117: mhhhommmmmu EEEEEEohEohmhI - dtic.mil TOKENIZER-PROCESSOR (TKNIZE) ... A finite state machine to parse data base statements written in this language c) A user-interface tailored to

00000§Q 0000000080000000000000000000

* 0000000000000000000000000

LuZw

C4~ a

00 in 9i

14 5-- 4C

.,. C, z

0 e at .goo;I.

P~f.-AO in Z .

3- - -- Z

0.. iIw uCe

Lu z-.C at-

11 3

Page 118: mhhhommmmmu EEEEEEohEohmhI - dtic.mil TOKENIZER-PROCESSOR (TKNIZE) ... A finite state machine to parse data base statements written in this language c) A user-interface tailored to

800000000080000000008000000000000000000080000000008900

f. 0 0M~ rw M O CW n o rM ~ ~ rw l

* 0000000000000000000000000000000000000000000000000000003333 XX33 3 3 3 33333 33X3 33 39 933 1 XX 3 33 3 3333X33 3 3 X3933 33XX3zzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzz

- ~ 33

LU D0

U. LA. w z

z ~ L 0.0.

a!0!A cj- U UU. -u

2. 0- M..JZ I.-

W.L w D

w~~~ I-4 . .LU MLU z. LUL Z u n e

I- 0 . 000V

i 2L W - 4 V.0u

.. s-LU 49 - -.0 w U.

M2-U Z A2 .x U..0 ... -i W C) -- 4 1-1

-U; 0 4 4-0 1 -1 In Q CLC C V cId)N LUG' ON MA wS ifl.- 4 I'D. .wL

a a ~ ;*:-2 w a Z U.& I.I Z 0I *-- z (A z-4 -LUU *- - 0- 4 0 U. IfK 0t* M 0

V U. W -j a .. ; a. £- w wo w. 0 . Z0uiM ,a L0 .14, 31* LU 0Z A -IJ -. V ... I-) --

" IZ.9 1- -0 LU.4c .J )0 W2 LU 2- .m 09 --, n

w cc6; 3. W U4

H oo. 0inmI&-&0& w w3 &L am-Onc U.'*0..I- 000 M...a -j

a. w I--..Z0I- -1 0. ---- .L I 0 I-L jz. Mu inmUJx

zr ~ Z I.- I-.1~ I-- _I ~~u. Z 4. .. j 0 U

114

Page 119: mhhhommmmmu EEEEEEohEohmhI - dtic.mil TOKENIZER-PROCESSOR (TKNIZE) ... A finite state machine to parse data base statements written in this language c) A user-interface tailored to

0000000800000000082 0000000000000mm MOMM800000000

000000000 00000000000* 33333333133R33 333333333300330300g3LU~~~~~~ RU 39 XU 39 LU LU LU LU3 XU 3U 3U 3U 3UL UL UL ULUL UL UL ULU UL UL

zzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzz

LU+

ILA

zJ

Iz ZC

Page 120: mhhhommmmmu EEEEEEohEohmhI - dtic.mil TOKENIZER-PROCESSOR (TKNIZE) ... A finite state machine to parse data base statements written in this language c) A user-interface tailored to

082000000OO000O0000008

z

zz

35-0 ceI

9 UI- W

.a4 I-..I X a

at ~ ~ ~ n I. II i

us5 ul wJ

5 , - U .zI -- i f

w w

116

Page 121: mhhhommmmmu EEEEEEohEohmhI - dtic.mil TOKENIZER-PROCESSOR (TKNIZE) ... A finite state machine to parse data base statements written in this language c) A user-interface tailored to

00100800008200009000000000000 0O0.100000O003 1X3 9X3 X 3M3 t 391w Iwz Izzzzz zzzzzzzzzzzzz

H;jF-0

0-0-

OOO OO OOOC O OO OO-OO OO OO COVOO OO33 3 3R ) 33-33 333 u3 3 03 33 3t ~ ~ I i I Ue U UUUU* ~ ~ r zzz DzzzzzLzzzAzzzzzzzzzzzz

Ix~ ~ ~ ~ ~ a L 0 ,co c

0 Z u0

a4 a UOm0 L-

0 _j- U- > co Z0 am -1U 0 -j &0 M

LLI. "4

a~ &,I, at00,0,0 uZ, - Zj 1- I-i~ 1.a

z tlz0 1- -9azLU0 ).c 0 OwO *.

a ~ ~ ~ ~ 2' z . ) >i )wa 4 K w=I. - o . wfU- > 0

I. - 4 0 zVIw ot - c x-0 C.DU L .4 j

In'~( -4 = -W M amw o &WV

u Z .- *j CO C 4- o- 0 - .

&8 2 ZZZ- x &W - 0. O

..z -l4 004wa -19 -2M*00 0 UI a

Z U -i * . -

.- 0

117

Page 122: mhhhommmmmu EEEEEEohEohmhI - dtic.mil TOKENIZER-PROCESSOR (TKNIZE) ... A finite state machine to parse data base statements written in this language c) A user-interface tailored to

F 000000000 00000000

zzzzzzzzzzzzzzzzzz

0-'

w-I-

is jx C

wIn1-0C 10~4 U .a

U. Xw

118

Page 123: mhhhommmmmu EEEEEEohEohmhI - dtic.mil TOKENIZER-PROCESSOR (TKNIZE) ... A finite state machine to parse data base statements written in this language c) A user-interface tailored to

-A

.47

0000000 0000000 00 000 000000000000 000000000 0000000000000 000000

W IMW t W W W W W W LAW W .W a W W W W w W W w W W W W Wa Wa - I-1. W w. Wa W W-u WJ Wl W- Wl u W W ul Wl Wa WM WM WM WM WM WM W W W W Wzzzzzzzzzzz2zzzzzzzzzzzzzz22zzzzzzzzzzzzzzzzzzzzzzzzzzz

z

WW

Iaa.

Z i L.0CLL 0 (.U. z aDO4I

o . A -wZ im LJC ..co zo M :z - 4K0

-j L 0 .-.C C1 L L l A-s J> z xc0- U.0 X W $- -- 2 2 1 0 10 1.- 0 Z c >LjL, i f->

; >4 Ou W - W *. :: ( i.

o . Z m. ccc L Z U 0. z- 2

ma- 0* * M -- -i ( 6 l0 0. - 0 CZ 0. m-- a-.M N a c0t. Z4 - .. O ~ U- Z - 01- l. 0C'

LL 0 I- 0z -A at = 0 0 0I 2

OW w~ 2 6-W - W. - U . 0 0 2 Ia 2C *. j~ * 6 Z -C 0. - WCc Iw 0 eM 4A-w 6- Z L - 2 n W U CI Im. z a W SM > I-DZ -> Z u

LL, A -CM -0 1.- < - m- 4 - D*.UtC 9 z I 6 Z -- Z 02 LC4 Z .- 0 -- " 9 .

a40 .- 0 *'wO . I. c - 11 0 IL- 'Z ~. - z 1- w.. u WI- c - v~ W cs . W0. L -- 1--Z Z 49 W- - " I (z M0- 0i- *- ; a O.. P- 0- --.. I(z

0 w LLI vM WS -.. J I- - 0WW - ooJ M XW O m03WU Z 40 2--a W' 05 - UM -J Z- OiUIo UJ Ia- m-. I IO4AiOz

OWN" CL .WV X* SMtA - W4 CL1 CL i Lm CDt U I.. U. 5i J " (L CZ 0 .4 L6 >s l- ts S U.0 f a.a -W~ CK - U m WWU uL

> VUZ W -j Z Z ---- CL C IL li "1..~ 1.J. ZU Z. P-- j -c i-a ZI-3 f~ -wj 4c I-P- i W Zww

'-Za- um -2 '6 z z U. z- -. U.Itl.J.0 Ifl- I

1190 >

Page 124: mhhhommmmmu EEEEEEohEohmhI - dtic.mil TOKENIZER-PROCESSOR (TKNIZE) ... A finite state machine to parse data base statements written in this language c) A user-interface tailored to

00000000000000 000000

z

~$*$ z

uJj w ,

-i i d i I -0 6a

CL CL ILC ~ w a W4

mi Uj La .u Z C

129

Page 125: mhhhommmmmu EEEEEEohEohmhI - dtic.mil TOKENIZER-PROCESSOR (TKNIZE) ... A finite state machine to parse data base statements written in this language c) A user-interface tailored to

00000009000000

39 3X X393 3 X X X3 39Xww ww w w~ w LUL U UL U UL L ULzzzzzzz zz zzzzzz

0U0LU LuEA N!

31zLU'.a

U. 0

z c00

.j0

zU

0*

Z 0 w 0

s:::0 -0w~

121

Page 126: mhhhommmmmu EEEEEEohEohmhI - dtic.mil TOKENIZER-PROCESSOR (TKNIZE) ... A finite state machine to parse data base statements written in this language c) A user-interface tailored to

000000%000000000§9000000008 0

'nLuuj0 0 0 00 ww w w o00000000000000000000000000000

3 t3 393 X33 3333 X393 393933 33 X333 333Azzzzzzzzzzzzzzzzzzzzzzzzzzzzz

0

UA

U; z

z 2e

- ~> 4 01- nin1 M ~ f

S0 WUI -I

~z I z~ I

IX ~ ~ UA-c-C -a. :0 Oa.N -MA4

> >0 0n z3'. I x c .c

a. )-- a.' t-- .

wu SM X x J

Page 127: mhhhommmmmu EEEEEEohEohmhI - dtic.mil TOKENIZER-PROCESSOR (TKNIZE) ... A finite state machine to parse data base statements written in this language c) A user-interface tailored to

* 0000000000?000000000 00000000%0

* wwwwwwwwwww~wwzDIOWWW"

* zzzzzzzzzzz zzzzzzz zzzzzzzzz

at 2

'Al U. l . lz .4

P-. UL L )

SA m 0

m: w 01- t- 0 L

W.oJWUC. Ll I.- I-C Lu

0 0U. =' nA U LU -

* z - cc ccU. 9z (ce z .Z z L1 I.-Wi.'W -IXZ ZMZ I- z£'U. ~0 0291z 0 Z 0 LU

w 4 -- - I. I--

1. 0 14 CZ Z * -0 Uzz l

0 U ~ -0-( I. 0 0

Z4 0 Z 4wZ-Z 4 4 mi a >I1-4U 4 . 4 U 4 1 .Z M W

w w3 LU I. S.

S1- - ,- Z I- u I- w - U0 DI- - LU-wjw= - . .

I u W CL -- U4U o-z 0 co

0~~w- )~LLU

ZUW

W40 0:uo. . V . .q I

.- w m o -4

Page 128: mhhhommmmmu EEEEEEohEohmhI - dtic.mil TOKENIZER-PROCESSOR (TKNIZE) ... A finite state machine to parse data base statements written in this language c) A user-interface tailored to

00000000000000,00080000m(W A~!O 0 C- - ' r . C.) Ln

00000000 0 oo 00 oooo 0000Law tw w u LU Lu w LUw LLu Lu Lu W Lu Lu LU L W u u u Lu L LU L

* zzzzzzzzzzzzzzzzzzzzzzzz

Z

x +

o 49 co c

-L ot w

** ~ m < Z* ~ a

X4.-I

- x I '

A *aLn..jj- N 40 m *'

0' .~i4 %A u,-on~- -j z 1 040.W

124

Page 129: mhhhommmmmu EEEEEEohEohmhI - dtic.mil TOKENIZER-PROCESSOR (TKNIZE) ... A finite state machine to parse data base statements written in this language c) A user-interface tailored to

00070000800008j00009

ii'w

00000000080 00000000OR000Ox~~~ ~ ~ X3x3 xX 93 9333 9393 X 3 tX3

ujwuj~j 4zzzzzz zzzzzzzzzzzzzzzzzzz

0* (

Z.

+ a

II'.

a Ce )- 0 0 (N *: z~Z w - a W

LL. Z.ZZ.z mau

z. * Z w w_j 9 Z Ju C . I-.

zm~~ ~ t. j1- I~

4~I - U4 I- W Sl -

!: 2U 0 wg z 0 0 lz0 0

-inq In 0 o- IL- - t- w

8 zc2 -- u.. W. In Xn Ij wn +

inU4j Z - c I- v Z Pi I- - _j.I- o- - xCL 0: _j X L, z u m n cl zn u, = .ICenw.... V j

~fJ j 7 J U .~ Z4 w .- I( NI-.- .j..C1f* & ww- mu "I A.I

9L CO- mu 0. j i*1 Q wu -

1 125

Page 130: mhhhommmmmu EEEEEEohEohmhI - dtic.mil TOKENIZER-PROCESSOR (TKNIZE) ... A finite state machine to parse data base statements written in this language c) A user-interface tailored to

00000000--*o0000000000000000*A W w wWW LWWWWW AJw 4. j W iw -2zzzzzzzzzzzzzz

U.

4n U

cc U.. I- =1 .1

I 0- - - (I-

ov

ww a

SC126

Page 131: mhhhommmmmu EEEEEEohEohmhI - dtic.mil TOKENIZER-PROCESSOR (TKNIZE) ... A finite state machine to parse data base statements written in this language c) A user-interface tailored to

* ~ 0 oooooooooooX33 33333333M344.4j W w4. W 44w W W wW 44. w44 w w4. LU LUL U Uzz z z zzzzzzzzzzzzzz

-0

C,4-

z4-.

. - .. .

10. 8-4-.9 O

c -9 - z I.

0 - VU c U

012

Page 132: mhhhommmmmu EEEEEEohEohmhI - dtic.mil TOKENIZER-PROCESSOR (TKNIZE) ... A finite state machine to parse data base statements written in this language c) A user-interface tailored to

000000000000000000000?000000000000000000000000-cd~wiflU-C4 CN c il-~~0~l W- 0mom ~ ~

~zU ZU . U U U U U ~ U U U J U U U

.0 *w~ 9

*4 9j4u-

*0 *UC-

.~Z 0

*2 u

X~ w .

9z 0 4K 4

.x 0-9 *.M* .ca z zx

M -4 31t -w K j o 0w0 1

* w -- W V) * . K -z M'S - w ta t Z* . .~- 40 . 4

9-000. §Zt4 4 .

*C Cc 9- - x C*w w zQ Z 5JZ VI U DM2 -C IW I -T

V Q4 9c 0- .4 w L -

If ** *A N*;W14V

-. 4 M -j -

d .uuu .t .LW Kt otc .o ft9 w ( 'Z- u

- . w& 0 XI-128

Page 133: mhhhommmmmu EEEEEEohEohmhI - dtic.mil TOKENIZER-PROCESSOR (TKNIZE) ... A finite state machine to parse data base statements written in this language c) A user-interface tailored to

-ji

*i I*I.-

A A

~Ln

.CA

zz

000 0 0

AJ I.-

IL _ _

. zll u

ac w

12

Page 134: mhhhommmmmu EEEEEEohEohmhI - dtic.mil TOKENIZER-PROCESSOR (TKNIZE) ... A finite state machine to parse data base statements written in this language c) A user-interface tailored to

000800000800om m mIw )VL Df

h. ~ -a 1- 1-1 1~ - 1- i n -~

In in

1- z

-4 -aK4

0- -9"I. .

0 U. ;:'. ,

W4 .4 0 a

01- -1. 4uXL

0 0 .&ltlC 0 0

Cl. liu 3c 1

A , -

413

Page 135: mhhhommmmmu EEEEEEohEohmhI - dtic.mil TOKENIZER-PROCESSOR (TKNIZE) ... A finite state machine to parse data base statements written in this language c) A user-interface tailored to

000000000000000000~MOMMOMCM 0 0000000

§§888888888800;0009-9--- - - - 9 9 9 9 0

UO U)U U.IU U U U

z -

0 0-. -

UL _j

x < C J-.0

Wu W L

z u- c z ..0- i0 -...

I. I - - -

0 -w w Q C

to W9 - .4

0 a * 0fCL 9-

,C---w.Jo9L

*z -

a. 9

131

Page 136: mhhhommmmmu EEEEEEohEohmhI - dtic.mil TOKENIZER-PROCESSOR (TKNIZE) ... A finite state machine to parse data base statements written in this language c) A user-interface tailored to

I. I.I.i

04

*0 S

0 Z; I U-z -4~ X-

z - -30 *.W

L6 0z m-. * ~a

0.. ~ ~ 4L C4 V .. M

6t. Z Z.1~ -- * . I *-w-- *41InZZi .U I4 .- 4 O - '.. -

-09-.U - I.

z .. -'_ I- 9 .I 0 _

2 I - ZZIL *la Q I- ce- ,-

ccz-! -iw 22 . t1 M 1 . -.

)U W* >Ifl 9- &fZO

0 -

0 i0 ItW In 0.- #A. 0, Z4mu U. uZ ZZZO& .0.0ZZ *.. m

0 9*.~- 9LL.OO - fZ Z IfL UIfL UJa. p I.- I. -

I.- =D 02 M mm a..~~. 0m

132

Page 137: mhhhommmmmu EEEEEEohEohmhI - dtic.mil TOKENIZER-PROCESSOR (TKNIZE) ... A finite state machine to parse data base statements written in this language c) A user-interface tailored to

00000000000000000e00000000000000000800000000q Ln4DOr 01 ON ~ fl C4 m v ~f) I O 0- 0 r-w0(mo r- m C 0r 4mw ) ID t-

000000000000000000000000000000000000000000000

w -4.

44.

UUO -C

2 m im .- 44 W .:1- 92

4- 9 z .- =4li 1

z f- P.L

U :W-CI4%u w I U. A

I cru . x , a.9 ::

-U W. I.- t44 - - -JC4-b . I-I Z. - ~ j I- -I-t LK .9I im0 K . x

MI .1.2.- 4a.4 z %A m0 jaI u

A AlA. .04 I-- ma: w A~

4- Q L 4 U-b- -

I-- lao CX 4/a - a.

0. AI" LA Q CL 0 CL 4 2 CC A % )4-

Z .. 1 of a . W-. - 00Ut6u -a a. $-.,-.i-01- -a

'I 2 L IX 0I- - zm C C

133

Page 138: mhhhommmmmu EEEEEEohEohmhI - dtic.mil TOKENIZER-PROCESSOR (TKNIZE) ... A finite state machine to parse data base statements written in this language c) A user-interface tailored to

082000000 000000008Q 00

0000000000000000000000000000000000000

14 4

in v

z .

Ai LU~W U)-

* 134

Page 139: mhhhommmmmu EEEEEEohEohmhI - dtic.mil TOKENIZER-PROCESSOR (TKNIZE) ... A finite state machine to parse data base statements written in this language c) A user-interface tailored to

Ut

* a

0I -

(.40U

* 4

aUa-U4

I"

135

Page 140: mhhhommmmmu EEEEEEohEohmhI - dtic.mil TOKENIZER-PROCESSOR (TKNIZE) ... A finite state machine to parse data base statements written in this language c) A user-interface tailored to

0000000O0 0000000008000000000000000000000000000000

Z.z

-U

> w -

w9 0 zO9

9-u 0- 0 40

IX w- w 3

z. 00 0 n CL

CL -c w

Ix - 0at-. j aI 4.- 0 .-- I4-

in 9 - Xm (Di z t'Ou a w* z -- W

NU . -n- -9 00 0-n29

Luz=-- Z4 4 .4 1 - * 14) :): - 0 im win 0 Mw~ cz -U X4 0 Ow IiI u0 0, U U. -z9 0- '.. x .n 8 I..

0 in g O9a!

fnw.i0- P-9-f U0 014. 9- .J 9- x .j-i-f j

2QU49- -009- .- -j a: - - - I-in z zuzm CwZ:Zuwuj~m-nw. 0VM W = ZZCuJ:N.J-0 3U9-3a.0 w.n0 I uI

* U UUQVU dU 49 0 4a-4$ 00000 CCU 3 A U

136

Page 141: mhhhommmmmu EEEEEEohEohmhI - dtic.mil TOKENIZER-PROCESSOR (TKNIZE) ... A finite state machine to parse data base statements written in this language c) A user-interface tailored to

5?0000008?00000 0000o

zz~zzzzzz~zzz zzzZzz zz z

z

n

2z

Ac.9U4-z

lz

u t - I - V-L

- Ujx x -N N I

-- zU 0M z4-L0*0-Wm

-4w-. lw z

w U -vU U, i-00 .i ~ N N U

US-N r w0J **~ 137

O.,Z. Q 4-.O I- I

Page 142: mhhhommmmmu EEEEEEohEohmhI - dtic.mil TOKENIZER-PROCESSOR (TKNIZE) ... A finite state machine to parse data base statements written in this language c) A user-interface tailored to

0

I Ia

z U -

ft - 0 -a; - act In ~

Us I.. 1-

I.9- - C~XU9- *usJ Z ) wLtl i

x0 -.90 ~ j1- w

Uf-C W -W U.-.3" -0alCx x - x 6 aX Z QWZ

&t - * > wS-u ZIf

0 138L"~ 4r-- it-

Page 143: mhhhommmmmu EEEEEEohEohmhI - dtic.mil TOKENIZER-PROCESSOR (TKNIZE) ... A finite state machine to parse data base statements written in this language c) A user-interface tailored to

000000000000000000* C Wf r)- n -~ 00 m t U

000

zzzz zzz zzz

z0

LI

A zMA

z z

0 2

I. -- u

L6 a - =W~ L6

139

IANN

Page 144: mhhhommmmmu EEEEEEohEohmhI - dtic.mil TOKENIZER-PROCESSOR (TKNIZE) ... A finite state machine to parse data base statements written in this language c) A user-interface tailored to

oooooooooooooo 0000008Q 0000000000000000008 000000000Q 0000000

~~flC~l-0000 0 U e00 fI?0O CC~qfIt

OZ222iizoi22~~in -- ozozooo2z2i~zz~~zii2i2i22

IC'-

co2CowCO-

M-

9 0 in-

> 0- 44 0.

0 '0 ~0 Z0004 -- o I.-2 inWZ 0

Z2 0 0-2 1--

Xw 0 x N4~ I.-Z z~

In x

10 a tl 6;J 9 C 1fI- o- CZ ou 1-

... -4#2F Z 4 2 -LK 9x--IX x. z.m w cc~ -z u

l l . *4 .93 12 - P.- - .-.- 0 U 1- 1.1. - .. to-.. x m 4 4 -In ti (Alf I

l.-'N~~WZl ; ~ ...D ? 2 X -~ xl~~ x.l~ * ' 1~-u > XZ ZzlU 4 uf t-zI.

-. 'ZO.-~~U -4~ In.I M ''I-- W a~ 0 0u.-.

U.o t' "I In I-. mi- Z1--4 ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ 2 -- c4 I ww o- r x1 jx Z01- k& jLU 4 2 1- j A n U wMC U n OU~ - t

Z - n nnu~ CAn.1441 n n

-9 5 42

uj IZLU b-LU L :) .ZL1-0

Page 145: mhhhommmmmu EEEEEEohEohmhI - dtic.mil TOKENIZER-PROCESSOR (TKNIZE) ... A finite state machine to parse data base statements written in this language c) A user-interface tailored to

008000000000800000000000000000000000000000000000000

0 000 00000000-20jR22-~ioz~zizzozi22zizz NzzlNz C

0

W~ ~ ~ (A Cul lm z0C

I. U, N Z-: = ~ =kn- Ix9 0- = . S - A9 o 9-I

1 0 - = . - z 1. - x n -2 2 - 2 2 2 0

-.. ~ c Inc a, 9 - -1 -UIX *-mn 0 2 :2A

9-1~~ ~ ;".. 9- - In F., -'', -Lu uj -I j UU tP- * 0- In m - I-- - DL -

:1 .. .** -3- w U a Aul~En U9-=

141

Page 146: mhhhommmmmu EEEEEEohEohmhI - dtic.mil TOKENIZER-PROCESSOR (TKNIZE) ... A finite state machine to parse data base statements written in this language c) A user-interface tailored to

4goooo~oooo80000800008?00000

CY) 4 M w 0 r-9 0 m 0 wr- C4 -Tkn L rlw00N r-f~-r~r 8 r-40

N"NNNNNN N NNNNN

IA U

a *. 0

1 '. Z ca-

z .- u .. I

In~ .N 0z ~ 02LL -- 2U 0394 - . U

ZU > -j >0 0J~I m

w ; aa ~ -

0j a -z 9. *q UU 0 U0 -- 4c a

&n 0 -. m z Cie 031U 0 N- z0- UA >9.0 U

1- -a0 z ZA -- Z--U~~L >O.- - -oI

In a x 2 -W 4z 0 -w

Z 20 4 -e zf2 z 2 > z-,- > w -a 0, -U& -c

9-3 0 -NI- U z~ z Z - -- Ia. *j -- > > I& 1- . 0- - -3L

ix 9-j0 .22 -- 2-- "

C4 z - *- , -

UZ* I--- 4 U - Z -jN NC 0D in . . . I.-*9-wUU "4) K4 a4 -. W 0 .2 ZU V) *C4NWu 49 xg u cx 31 *Z '2 E z z -

02~a - 0 U - .'Z I.i P-) C. -2. ct 60IZ~J 4090 2jP- - V a -UW 11 00 0 -A 2- I

UK a 2 0 cc a- iP- 2 ZU j1 j a lz z* -u0 0. -0 aJ z- -* N ~ ~ -4

U vi 1.4 ' - **, z . LI 14 AL. 0 4

; ZC.W4ZW .n 2l( -- WU03-0 U UA 01 l- x w *0 wv

U. ~ 4 0 U 4 ag %R~lU 91 U. &M C0z2.3 *U- j-D j 0 0. U *.O 2

U4~~~~~~L 'A,~ DO9 n ' -3-n.~.3U9J UO Jcoo cc.0IN2 UK- I -L I U U 1 9

U09.-I UOI 9- 0U 2 1422 *.0 v

Page 147: mhhhommmmmu EEEEEEohEohmhI - dtic.mil TOKENIZER-PROCESSOR (TKNIZE) ... A finite state machine to parse data base statements written in this language c) A user-interface tailored to

000000000000000000000000000000000000000?0000000000000000000

S00 00000 00 0-2000002

-iioi2oiiz~zizizzRZz IzzzzzZ~

z

* z I-

o -I-L

x .iUz

x 0. 0 iLnC0; 3 - Zm

ID uj 0- LI--J .

3- - 0 x- Z wi =

-z 4- . z -U

IZ~~ 0 LI- --

if 0. atA 0 IA ina

0 w 4 *rn --3 4 - *o -O 1- 0 0 1 -- -C -- , ai-- . n 32uj- .x LI > - Ln i 3 .i I. m- i a.Z0 -> - - CDZ I-J --

z. - L.. - Q I. 3. V Lu O .J Lm -1 8 w0N 1-a.-.- I- if .U xi 2- - a .- -0. 4A In..

I. i1 -J A ~ 0- W W.Z ZZ V) -0 tn- Z in in A

-(U I-00- '00 00 Zi0 .W i~iU~--00ILO 1.--I l-l Wl- ;* 1. U -1 1

M. uoin-If w IL = 0 aC x Li ifix 4Lua *n* 4. N 1-- 2 A A -- 0 A A *O Lon -J j v 0 .- X 0 U Z -nW W0 A A A0- 1- W0- J -o4in4oxx Z ( 4Z, P- J J~I - 0 ia- U. ~ j ,, 0Cc X P- X n- U Z 4 tx - *" Lnl~~ I- 9L. LL-

-t -rn N -1 i

Z 0 0 -c

0 0 0 01 4 3

Page 148: mhhhommmmmu EEEEEEohEohmhI - dtic.mil TOKENIZER-PROCESSOR (TKNIZE) ... A finite state machine to parse data base statements written in this language c) A user-interface tailored to

u.

P- Z I-.

X

V"Jo4 -

IL I-

C2 CK

&-xu

4 144

AlioL.

Page 149: mhhhommmmmu EEEEEEohEohmhI - dtic.mil TOKENIZER-PROCESSOR (TKNIZE) ... A finite state machine to parse data base statements written in this language c) A user-interface tailored to

000000000000008000000000000000000

1 0 0 0 -,~~0 0a -

z -

AM 093-

4 0P- 3z

U 9- 00~ U,

- 1-.1 91.

0 0>

zz

0U *. .-A 0 .

> .0 0--P- ta 9 -Cc

P-I- N 4 l- -)-

a- - K 0LL z $-

M C4 2 0 ---u zaLL4Z - 0 * 9

u 0 *x W(2 AM

0 Z U.- -0 UA ZAM0.

oawm OZ0n-o" ..0'E 2W Z iJ .

0 s X ; O N 0 99 IA U. JPPRO.~ mot9woo.

%-00000 ~9 00- U0. (.3(1

145

Page 150: mhhhommmmmu EEEEEEohEohmhI - dtic.mil TOKENIZER-PROCESSOR (TKNIZE) ... A finite state machine to parse data base statements written in this language c) A user-interface tailored to

0800000?0000?00008000000

00 0000 000 o0o0000

Zwo

-.LU I

I.-W 4A)"..j.i. I 0 44 Ilat .I W. U 44 414I& .. I

-wu m.I. 0..2

Zmw-In~J~UU ZAIL~I~J~I

OX0 z UHl 146

Page 151: mhhhommmmmu EEEEEEohEohmhI - dtic.mil TOKENIZER-PROCESSOR (TKNIZE) ... A finite state machine to parse data base statements written in this language c) A user-interface tailored to

0000000000000000000000800000000000

00~0 00OO 0 0 00-O~0z z~~oozRoojo

zzzizzz zz z z zz zz z

ofI LAI

LU

ftw -

o-

ft - a -I-

cm 00ZMt -

-U ,*n C - Z -2

I L. .4 L

LU U L" - ~~IK4z- -f j w-Wn f

IZ - Z2 C4C . ta )C44 WJf M COE I!E aWUW- N 0V )x c

vS IfU- 0- Ca.U0t f- cc at - U; U, - wUM UL Lift 2 Z A I In ft

a.4) t * 1 WW w La :;i w

MU. . 0.~- WWW- it Wj tJ jJ

WU M.

147

Page 152: mhhhommmmmu EEEEEEohEohmhI - dtic.mil TOKENIZER-PROCESSOR (TKNIZE) ... A finite state machine to parse data base statements written in this language c) A user-interface tailored to

* 0 00 08000008388888

fK~ .9-X-

.**~~I -(.-0~- a

X3 U8 -0 0 - oom -. iqw*0- 0 * 0 WW- 0- ~w 0 z x U W

8 . 0 - w' * u c - IU I~( ' I J 0 (~at 9k u0 a InW0 C4 xU. : - Ifl I0 mc

IL I I. U 8l L -CX U d ati Ciu. 2 Z.- 0 cx z CL. IflA-

.EWZO 12 1A~ Wjo -- Il-4WW >

.9 U 00 j WI -W -WZNC"N IN

o Z Z " U X- U 4 UZ z

0 0

148

Page 153: mhhhommmmmu EEEEEEohEohmhI - dtic.mil TOKENIZER-PROCESSOR (TKNIZE) ... A finite state machine to parse data base statements written in this language c) A user-interface tailored to

90000820000801 0820000200

444w4 44 44 4 4 4444mv -0MSU d .IT0w- Nmvw -w0vU t -c 7 oc 44 4 4 44 4 4 44444 l

z

I.- >

m0 Az V) 9 0- 1-

4 ALAaI- I-. I.-.

zz u0 4c- --M Zce0 a a - 4 -

LAJ* 4 w Z ... Uu zc

Z A- *- 1-a A*i U. aa I-. i *

cc 0 49- - 0* 1. 0

>~ ~ ~ Z.a * 0 -" = czw vc- > z z- W IL ZU-U

V) In 0 0 O _ I aI 0 -LaDL) Cc 0i ZZI2-ZU- 0

La > -a z ~ - L %

-z z :.4 4 M 0j a Ca4U LJ

I xC 4 IfL - ma we I.-. cw

- u 44 .- Co z lz M IZW.o o- .. -0 w l m - l. LuJ-

0 N '' > .0 0

U. 4 ! z - i. z- C; 0

La~ ~~~~ La' wa U IL or0 (* m .-)( U' cz m La L Z -00LLI-tIfwf z. 0z t ! - 00 40 L 0 . ft)z a 2 - O 1- . 0-)< U . LL CL P- L Iz zm oLa.1

n-L:a : ; w : 0 C u.m1-~ 0 -4n UW*1i 1L-0 10 -U- 04 - Z4 C P-m- Z CL I-l~~U

Z40C~ C 0 U w I Z0.21204 qq 4 49lcoU

U- 1--C..4 0- u CC.) La0 uC.)M()r

0 w 4Z U z xL zC4 C4 C4 N~

C C4 C'4) -

0 0 0

149

Page 154: mhhhommmmmu EEEEEEohEohmhI - dtic.mil TOKENIZER-PROCESSOR (TKNIZE) ... A finite state machine to parse data base statements written in this language c) A user-interface tailored to

* 0008Q000000000

~0006La

z

I-

LL. U.

z

w - t i 04

- MI (Mi--w ~.Zl N. e

I-5

r"7

Page 155: mhhhommmmmu EEEEEEohEohmhI - dtic.mil TOKENIZER-PROCESSOR (TKNIZE) ... A finite state machine to parse data base statements written in this language c) A user-interface tailored to

000CY -N -- - ---

LU)0- . 0

lzI- V! 4 C4

I U) u

-7 J - j J -7 j4Z I -- (0 0 (w(N a u w -w w j -

0 00..- ..- o 0 0 0 .. JJ- NJ C4 0. -49 U) CL- W*-~ - aWW- a a

C4 NI 0 N * 00~ 0 ( N ( N (N c 0 0- 0 0

JOO-O O -- a -- zi 0-C -- "I I- . C.x X M XAka v) L A 0

-L Z a.. *0a CLMC 0C a aC a a m 9e(aC as L C CL CL a 0.

* ~~~ U) C.L WU (N (N(N~ ~ ~ ~ ~~~C yNN( 00N( 0(N( (N (( N N

*~ ~~~ w. A . ~ *O 0

LaiaIL 00 A CZ

cz 0 .. V :-&nsN6 .... I' ..) I

- -*C . - - - - -

1- a~a~151

Page 156: mhhhommmmmu EEEEEEohEohmhI - dtic.mil TOKENIZER-PROCESSOR (TKNIZE) ... A finite state machine to parse data base statements written in this language c) A user-interface tailored to

.77

00

x~ uN

ac'im ma -. 0L0C

C; C -o 7o. 7 1 ..

.x 0

IW a U . ZZ Z *-

m 0 I In In CA(4 N -z 04 oe C4NNat N

Il . 0. .4 0 cc 0 0 000 W 0 00 c 0 M0 I

U 0U aw w N0 w in w gi -C. W w w

- - i- - N - l 14N N M N N

(A i 0 0 I

4 4. 4~ 44 UNN r; 4NN "4 44 4 N N UN W;l 4N

Page 157: mhhhommmmmu EEEEEEohEohmhI - dtic.mil TOKENIZER-PROCESSOR (TKNIZE) ... A finite state machine to parse data base statements written in this language c) A user-interface tailored to

I.-

CZ

-zz WL i LA A

4.E 0 .h RJO3 N -(4

If (VI (' Vi "W Vi W 144W

00 k 0-C J' 0 0 0 (CL 0 04 0n (A CLi A

C-L 0 a. Q.C CLCLCL9L I 0. am. 0 . CL 3 0. 00

C4 In **0- -'-4 ~ ~ ~ ~ ~ . M t. ~ *4* ~ - .)5 5 (In~ ~~ ~~~ ~ ~~ CL - lz a ~ .w-- n w - -- -

.0 uJ W In.-0.('4~~c -jCI.i (4. ''(4 4 (.C - 0 ( .( .( .

* ** .(4** ( 4 ~Q 0.-u w ZZZ ii uz zz w zz 9~ zj &M

*15

Page 158: mhhhommmmmu EEEEEEohEohmhI - dtic.mil TOKENIZER-PROCESSOR (TKNIZE) ... A finite state machine to parse data base statements written in this language c) A user-interface tailored to

C-4 Ifo -w ' r'd 0 N 4 MMMMC - 4~WI')

wn I--LJi -

1- 0

Z-

:Fz 0. Ua-j '

4 ZO4 Z

00 99 )WW W W

I.0 - a. !t 030LAO -- --A *.. Cd --

oo 0 0lI CL CL IL CL a.

V- ID M -0 in-In to ~ In to

., . .- . . . .mz =~ .. ct ~ ccd) Uig I. 00 a, ii cc &A0 a~ a aa

-l up, WP- C4 0 C0 t Ow- W M,

-~i I.- i

-a In b) In In 4 n in I n in -)W

154

Page 159: mhhhommmmmu EEEEEEohEohmhI - dtic.mil TOKENIZER-PROCESSOR (TKNIZE) ... A finite state machine to parse data base statements written in this language c) A user-interface tailored to

0 c 00 0 0~ i l mo m~ O to

0 W,

-z

oo

12 w mS -w *i wWi

* 0 Z

C4(U I. . m z 0 c . w

Lu z II ILu -

wl M, w * ID w I D

if ell in i

Lu.~~0 wu Lu * u * L uLu L u l lu

4--r 466 40 -00 -- 4CI4C ~ 4-4 4-, -4IwI

vnin IA 0nI In AU f ) I

155

Page 160: mhhhommmmmu EEEEEEohEohmhI - dtic.mil TOKENIZER-PROCESSOR (TKNIZE) ... A finite state machine to parse data base statements written in this language c) A user-interface tailored to

ev- r- co 0' t.(

ca~!-- 0 a

w - c

-i-

Kz 0

w A w

a- -

(N ~~~~ ~- (4 - ( ( N ( - (C(

w wU vv ~

.2x - hA n L

Enn in 0 8 .U

0in 0 N el

CY f6

Page 161: mhhhommmmmu EEEEEEohEohmhI - dtic.mil TOKENIZER-PROCESSOR (TKNIZE) ... A finite state machine to parse data base statements written in this language c) A user-interface tailored to

toD W 4 CV N r'q t CNdI -W v- v~~ ooGo cc Goo Go cca m ~ 0- 0- 00 00

CL 0.LWU 0 0

e4 --

z0 000 O. x00IL

Z w 0-00--Z c- -

-- R- I -- in In0

0 0 0J - >- -0 .0 mf f

Li0 0 C

-*Z 0. *! C4 C4 N C4 N-4 N

P z- ta a Ln V) V) 0, ma .. ~l0E 0.E 0.En CLUW-4 =0 SM m :)0 wLO Z 0~ m C 0 : 0 m

U. I .49 0 00. 0. I. CL 0.4 IL. 0. 0.0 0LC.0. 0.0.L

In 40 V)60N ( ~ Ia - - a a o * a * as aso 0 0 0

MA im *j ku uj m- L- mj1 U LA

W WM W w wm SM w) C W W W U, I

0-I i .- # I .- f-~- 5 t 10- *1 1. m- I-s v - 0-80 5- ~ I.-r4e4 $.r466 40 40 40 0 4900 -9- 4 1 C004 0040 400 A 00

I. - I.- '.- 0- t- go- , a --- I.- ~ ,- ~ 5in Wn Wn In n in W) vp (A ta V)En E

157

Page 162: mhhhommmmmu EEEEEEohEohmhI - dtic.mil TOKENIZER-PROCESSOR (TKNIZE) ... A finite state machine to parse data base statements written in this language c) A user-interface tailored to

0- N 0 C4f - ~ l C

ul 40 z 0

CL I L I I LC .C

WJ mix UM w LIL

zwL iW UAr ; U-z . .z d 5 .vpta I Wu

15

Page 163: mhhhommmmmu EEEEEEohEohmhI - dtic.mil TOKENIZER-PROCESSOR (TKNIZE) ... A finite state machine to parse data base statements written in this language c) A user-interface tailored to

~77

w..

Z

zz0 0 0 1 0 1 0 1

W0 vi i 00 W0 In 10 10 0 L

M-- CL C LI L 9 LC L& M0 LI LC

to 10 1 01 0 ON

1159

Page 164: mhhhommmmmu EEEEEEohEohmhI - dtic.mil TOKENIZER-PROCESSOR (TKNIZE) ... A finite state machine to parse data base statements written in this language c) A user-interface tailored to

-4

*1

w

U'U

I. - )

-9 0j A uj

U. Z CL 9 L C)9 LC 0 IL -

04 C4 4 m l CL -0l --V n 0 t

24w) U U) U) U U) -J w U) JA w) I-I uj

*~Iw V4 Iw wI U ~ -. . a . P - a. a . - - t I - - .- 0 . 1a -

vp V) n in 0 IA 0

160

Page 165: mhhhommmmmu EEEEEEohEohmhI - dtic.mil TOKENIZER-PROCESSOR (TKNIZE) ... A finite state machine to parse data base statements written in this language c) A user-interface tailored to

060

MAA

1- 0

0

2i 1-0 A

0. % A 49 -Ijfl A

4i Z w>- 4 0A ZMA U)~ X

LIn.fM = 00 2u

0n t 1.- 0- xZ49 =Z W I-- L6

co -J CL I- " Z

cc C 1-.-0 0&f >

-zL LA. I.- In I-L

Wuino 0 .u

Ii. MAMAMAL" Z 0.0- 0. P---

o L 4O -U 0 ;Qm- :.4 .~ 4 11 'a3

o Mt l z A. 8A 31- 0 -f~ cc ZM I. AW 4m

0in 40* 0 2 in m- M - 2 0U N Z LMA-

in MA .1044CI M

(A 0.. . -jm 2MA L - *3..

en 0P- mU *0 >A

1 39W 24W j 4

0 I0 - ih i MA.Za I.- w U n O co

w ~ ~ ~ ~ -X )0L .MA MA-0 enZ Z. w-

I-MA tW = I-A wo wu 1- 69J0 4c2

w W AiZ w *-cwo -4C4Iox- ox- -

Page 166: mhhhommmmmu EEEEEEohEohmhI - dtic.mil TOKENIZER-PROCESSOR (TKNIZE) ... A finite state machine to parse data base statements written in this language c) A user-interface tailored to

A

I

0.

C

zU

In

zIn

C

C-l

CMAzInMAUII.MAU

In ZzU W

Uz ~u

- U -

t ',*--.~

Page 167: mhhhommmmmu EEEEEEohEohmhI - dtic.mil TOKENIZER-PROCESSOR (TKNIZE) ... A finite state machine to parse data base statements written in this language c) A user-interface tailored to

IL'U

4U.z0

2(20

a- 0 V

0 U. a

I'. U

-ww IUL'x

Page 168: mhhhommmmmu EEEEEEohEohmhI - dtic.mil TOKENIZER-PROCESSOR (TKNIZE) ... A finite state machine to parse data base statements written in this language c) A user-interface tailored to

.U.

m L

0'- ft L

WA zI'

u'1

2* to

Page 169: mhhhommmmmu EEEEEEohEohmhI - dtic.mil TOKENIZER-PROCESSOR (TKNIZE) ... A finite state machine to parse data base statements written in this language c) A user-interface tailored to

.16

.lju

-l 0

z L6SNS I-

0 -

Z z62 D-

mZ

m w

0 Q-j- S

Page 170: mhhhommmmmu EEEEEEohEohmhI - dtic.mil TOKENIZER-PROCESSOR (TKNIZE) ... A finite state machine to parse data base statements written in this language c) A user-interface tailored to

*.1.

4-

LU0

u

- Cc

z 8-lu L~x

39 Z w O

N.* CC w

40 0 4

w 0>

ol

i.

-w

Page 171: mhhhommmmmu EEEEEEohEohmhI - dtic.mil TOKENIZER-PROCESSOR (TKNIZE) ... A finite state machine to parse data base statements written in this language c) A user-interface tailored to

*> z

00 zz

Izw

..... ..

Page 172: mhhhommmmmu EEEEEEohEohmhI - dtic.mil TOKENIZER-PROCESSOR (TKNIZE) ... A finite state machine to parse data base statements written in this language c) A user-interface tailored to

II:1

I

i

wI x

w9-

-i 2w

U

4I~2--9-oI~

* wU-- 40

,1 a

- w9-8 S-u2w w-- SliZ

4 Wll~ Zas w

MD

,. ~&h.9-.-

Page 173: mhhhommmmmu EEEEEEohEohmhI - dtic.mil TOKENIZER-PROCESSOR (TKNIZE) ... A finite state machine to parse data base statements written in this language c) A user-interface tailored to

AA

0

A

0

z

4U

x

31

Laz >, W 00a Zj w I.-n

020200 00 044444* 40004 40440*44444tz ~~ 4,Momwm 44U.~ 444 4_4040

LU uj A .. . . **-.0. -*A .. . .-.

u1 I .I I I-IuIP- I 101 uII-1 01 u . . 01 uiiI I i il I I l -1u10 1w10-1uI I I Iu1 I I11-

Is - 0-0 -0 .- 1 -0 .- P -I . 0 . . .- P-N-P 1 -0 -P I.- I.- I-. I.-VV VI I IV V 1- 0- 1.. 1-./ 1-./ 1-.I/ 0.l 1 P.- 0. 0. P.- 0-IVV 0.IIV V 1.-VV 1.-

Page 174: mhhhommmmmu EEEEEEohEohmhI - dtic.mil TOKENIZER-PROCESSOR (TKNIZE) ... A finite state machine to parse data base statements written in this language c) A user-interface tailored to

I-j 1

Z0 .

400

w Z 1-

Page 175: mhhhommmmmu EEEEEEohEohmhI - dtic.mil TOKENIZER-PROCESSOR (TKNIZE) ... A finite state machine to parse data base statements written in this language c) A user-interface tailored to

to LU 4

wIt > w440U 4 AI40 4 04 0 t

w o! 0 V 0 i N4 o I

a 0 400 0 N 4 4 4040 0 40490 t 40 4040 2440i Ac o a 0 0 9 0 0

I.- j < I.OXI-~ iX- 4'0 I It j 1..W 1

0 WU0 0 0 In 0lug

IIJ NW ... U0 Il I * N 0 0 In U I 0 071I

- w 09-w.9-~0W0 *0 W 0~~ W0 0 WdiW 0

Page 176: mhhhommmmmu EEEEEEohEohmhI - dtic.mil TOKENIZER-PROCESSOR (TKNIZE) ... A finite state machine to parse data base statements written in this language c) A user-interface tailored to

T >

*In 40 V40 to *U f

0 to w 0 40 tow N OI I 0t- 40 m4004 4 . 0 S 0 cc to .

cc 8 i 0 4 0 A - to 40 ftC....................* *ft0 -40 * 0 If w ).. 0 0 Z 4I 40 toD40 40 0 0J

ICLa 0 Id 1ftN0N IfZ2l 4L 1-4040 &flNSU01aUhI U & 1Zia~ zi ou zK 0.c~i~* UKI/ UKIR URv) umw 49 a19 00J - . , .. . It to *0 f o 1- w4 ; P- w4K ZKK XKKK aaKK I-KK Aa -

-U-U 1KU - -V.- I- t-- -P-f- K-

t if I n : C4 I I. A.IA.u.AI o N - .AI-

172-

Page 177: mhhhommmmmu EEEEEEohEohmhI - dtic.mil TOKENIZER-PROCESSOR (TKNIZE) ... A finite state machine to parse data base statements written in this language c) A user-interface tailored to

.11

II

IonI

f-J

IA W zW 40 4 . .0. M -4a N .. .4

-C 0-

-UK -UK

Page 178: mhhhommmmmu EEEEEEohEohmhI - dtic.mil TOKENIZER-PROCESSOR (TKNIZE) ... A finite state machine to parse data base statements written in this language c) A user-interface tailored to

Ir

4° I

.17

*I I

Page 179: mhhhommmmmu EEEEEEohEohmhI - dtic.mil TOKENIZER-PROCESSOR (TKNIZE) ... A finite state machine to parse data base statements written in this language c) A user-interface tailored to

*31

to4 0 t :'

t.) 4 04 U to oo .In > - w40 ift I- .

w > AJ 4 - J0 40444040 > LU &

40 40N 40w > i 2to 9z40 1- 40-j 0

to N LAJ 0 40I m> at40 4 0 4 40 w0 if 40 inm 40In ; 46, 40.. a L 0 , ujFA

cc -0 4 0 0ow vi 9 t N 0" .TOTT KO

inn~X~l 02 2 2

ccM* ** .4

an~~~ 5' ~ .a~-

Page 180: mhhhommmmmu EEEEEEohEohmhI - dtic.mil TOKENIZER-PROCESSOR (TKNIZE) ... A finite state machine to parse data base statements written in this language c) A user-interface tailored to

F in toD 0 - C'4

W C.) -4 -4

ae C4 v 0C0 00 00 C 0 0 r. 00 00 c- 0U In ~ (

A U a A aU * U

I 1 0 I p a N Jl j IN a j10 0 a

w 1W .J IwWW-J5 W.4I f W W. fW~ WM W WW.~ W.J w W..x( aiiMZS MM M -!a aU OD CU o a oetw - - 0 - -iz: :t .. nX .

X.J).~i Xju X.u .J 3(.U.~i X Q .J 1Ju x jW

~A16

Page 181: mhhhommmmmu EEEEEEohEohmhI - dtic.mil TOKENIZER-PROCESSOR (TKNIZE) ... A finite state machine to parse data base statements written in this language c) A user-interface tailored to

'I

II

I INIUI

cex

A 0j.wV

0 0 0 0 0 0

I--- -a X C--Xx~~ (N X-ju Xj j

u~ - . C'i u~ (N u 0..J-. x

Page 182: mhhhommmmmu EEEEEEohEohmhI - dtic.mil TOKENIZER-PROCESSOR (TKNIZE) ... A finite state machine to parse data base statements written in this language c) A user-interface tailored to

w

ww

LU*

0 x x

**0w

0 .O 0w w L

-- 0-'- - 0- *g1 .

Page 183: mhhhommmmmu EEEEEEohEohmhI - dtic.mil TOKENIZER-PROCESSOR (TKNIZE) ... A finite state machine to parse data base statements written in this language c) A user-interface tailored to

mi

0

A.

a Zw

* (,~ LIJ

9 -z * I-

X' 0 -i&

z > L*

U 0- -

0M 40>V

Page 184: mhhhommmmmu EEEEEEohEohmhI - dtic.mil TOKENIZER-PROCESSOR (TKNIZE) ... A finite state machine to parse data base statements written in this language c) A user-interface tailored to

j

1i-l

.4

*1

InI'.3

C

rI-,C

ft -

tz

Page 185: mhhhommmmmu EEEEEEohEohmhI - dtic.mil TOKENIZER-PROCESSOR (TKNIZE) ... A finite state machine to parse data base statements written in this language c) A user-interface tailored to