engrave - an expert system that understands and generates ... · introduction engrave is an expert...
TRANSCRIPT
"ENGRAVE - An Expert System that Understands and Generates Musical Notation"
by
Joseph Edward Tatem
Thesis submitted to the Faculty of ~he
Virginia Polytechnic Institute and State University
in partial fulfillment of the requirements for the degree of
MASTER OF SCIENCE
in
Electrical Engineering
APPROVED:
J. R. Sochinski
July. 1987 Blacksburg. VirginIa
L-D 5/fi ~B55 \q~1
f'1~ C,l,
ENGRAVE -- AN EXPERT SYSTEM THAT UNDERSTANDS AND
GENERATES MUSICAL NOTATION
by
Joseph Edward Tatem
Committee Chairman: John W. Roach
Electrical Engineering
(ABSTRACT)
ENGRAVE is an expert system that performs the work
or a music engraver. The system takes a digital image of
a conductor's score as input and produces
engraved-quality copies or the individual parts.
ENGRAVE consists of a vision system and a graphics
system. The vision system contains a low-level section
that recognizes features and a visual expert system that
understands their meanings. Under the current computer
vision paradigm, low-level processing is general-purpose
while domain knowledge is used only in "high-level"
routines. We hypothesize that domain knowledge is
necessary at the low level as well ror a vision system to
work reliably. The visual expert uses a frame-based
approach to explain the objects round in music. In the
graphics system, we make use of the concept or "spatial
planning" to generate correct musical notation. We show
by example that spatial planning can be applied to other
domains.
Acknowledgments
I would like to take this opportunity to thank
those who have been inv01ved in the project and have
supported me while I was working on 'it.
John
He was both
right mix.
Roach was a most able and competent advisor.
demanding and understanding with just the
Somehow. John knew when I needed a push and
when needed a break and was nearly always available at
odd times and on short notice to discuss problems and
ideas. Much of the success of this project is due to
John's insistence on thoroughness and quality work.
Special thanks to Jim Sochinski for making his
knowledge of musical notation readily available and Tor
helping to provide the detail necessary to produce a good
system. This research had its beginning when Jim inquired
into the possibility of building such a system as
ENGRAVE.
My girlfriend. then fiance. then wife. and now
soon to be the mother of my child. was a constant and
much-needed support and shoulder during both hard times
iv
and good times. Thanks for sticking with me. Dale.
would like to thank Andy Tatem, Hike Stone, and
Chrfs Philipoom for putting me up and for puttfng up with
me f'or "a couple of' weeks."
My parents went above and beyond the call of
parental duty to provide financial support. sage advice
and prayer. I deeply appreciate their sacrifice to help
me finish. Also, many thanks to the members of the home
fellowship that meets in their house for your unflagging
support.
Jesus
Most importantly. I
Christ who gave me
course."
v
would like to thank my Lord
the strength to "finish my
Table of Contents
Chapter 1 Introductfon •••••••••••••••••••••••••••••••••• !
Chapter 2 2. 1 2.2 2.3
Literature revlew ••••••••••••••••••••••••••••• 9 Printing •••••...•..•• Recogn i t Ion .....•..•. General information ••.
Chapter 3 Low Level Vision ••••••••••••••••• 3.1 Implementation .••......•.•. 3.2 Line Angle and Thickness. 3.3 Labelling the Background. 3.4 Horizontal Lines •.•..••... 3.5 Horizontal Line Deletion. 3.6 Figure Segmentation .• 3 • 7 Summary •..•.•••.••..•..
.9 1 1 12
· .20 . .. 24 . .. 24
• .30 · .. 35
· .37 • •• 41
· .45
Chapter 4 V, sua 1 Expert System ••• Implementation.
...••.•••.•.. •• 54 4. 1 4.2 4.3
An Example. Conclusion.
. •.•. 57 · .59
• •• 66
Chapter 5 Spatial Planning Tor Musical Output •••••• .•.• 72 5. 1 Planning Ahead •••••.• . . . . . . 73 5.2 Constraint Gu Idance .. .74 5.3 Description •• ..•• 77 5.4 An Example. • •• 80 5.5 Beat-Units .• · .94 5.6 Rendering ..••. · .97 5.7 Constraint Guidance In Another Doma in •• ..104 5.8 Literature Rev i ew •. 116 5.9 Conclusions •••••••••• . . . . . . .• 1 18
Chapter 6 Discussion •••••••••••••••• 6. 1 V lsi on •••••
• • • • •• • • • • • • • • • • • • 122
6.2 Graphics ••• 6.3 Conclusion.
• •• 122 ..124
• •••• 125
REFERENCES •••••••••••••••••••••••••••••••••••••••••••• 126
V I sua I ExPert Rules about Note
Appendix A A. 1 A.2 A.3
Rules ••••••••••••••••••••••• 131
A.4
Rules Rules A. 3. 1 A.3.2 Rules
Durations. about Measures .•..••••.• about Notes ••••...•.••.•• TO-FILL Rules for Notes. IF-NEEDED Rules for Notes .• about Staves ••..•••...••..
vi
131 132
• 134 • • 134
• •••• • 135 · 138
AppendIx B Spatial Planner Rules ••••••••••••••••••••• 139 B.I Constraint Guidance Rules •••••••••••••••••••• 139 B.2 Renderfng Rules •••••••••••••••••••••••••••••• 140
AppendIx C Dfagrams o~ "usfcal Characters •••••••••••• 154
AppendIx 0 NOTES ••••••••••••••••••••••••••••••••••••• 171
AppendIx E Spatial Planner Graphfcs •••••••••••••••••• 175 E.I Character Symbols •••••••••••••••••••••••••••• 175 E • 2 Beam i ng. • • • • • • • • • • • .. • • • • • .. • • • • • • • • • .. • • • • • • • • • 1 75 E.3 Ties and 51urs ••••••••••••••••••••••••••••••• 176
v, ,
List of Figures
1) Block Diagram oT the ENGRAVE System •••••••••••••••••• 2 2) Copy oT Handwritten Input to the ENGRAVE System •••••• 3 3) An Example oT ENGRAVE's output ••••••••••••••••••••••• 5 4) Output Trom [Hiller and Baker, 1957] •••••••••••••••• 13 5) Output Trom [Boker-Hefl, 1972] •••••••••••••••••••••• 14 6) Output Trom SMUT [Byrd, 1977] ••••••••••••••••••••••• 15 7) Bach's "Art oT the Fugue" as displayed by SMUT •••••• 16 8) Tenor part Trom Figure 7 as displayed by ENGRAVE •••• 17 9) Output Trom [Smith, 1973J •••.••••••••••••••••••••••• 18 10) Edited output Trom Mockingbird ••••••••••••••••••••• 19 11) Example thresholded image... • ••••••••••••••••••• 25 12) Example thresholded image... • ••••••••••••••••••• 26 13) Example thresholded image... • ••••••••••••••••••• 27 14) Example thresholded image... • ••••••••••••••••••• 29 15) Computation oT line angle •••••••••••••••••••••••••• 31 16) Line angle fmages correspondfng to Figure 11 ••••••• 32 17) Line angle images corresponding to Figure 12 ••••••• 33 18) Line angle images corresponding to Figure 13 ••••••• 34 19) Horfzontal lines ••• in Figure 11 •••••••••••••••••• 38 20) Horizontal lines .•• in Figure 12 •••••••••••••••••. 39 21) Horizontal lines •.• in Figure 13 •••••••••••••••••• 40 22) Musical Tigures remaining ••• Figure 11 •••••••••••• 42 23) Musical Tigures remaining ••• Figure 12 •••••••••••• 43 24) Musical Tigures remaining ••• Figure 13 •••••••••••. 44 25) The Tinal, thinned image ••• Figure 11 ••••••••••••• 46 26) The Tinal, thinned image .•• Figure 12 ••••••••••••• 47 27) The Tinal, thfnned image ••• Figure 13 ••••••••••••• 48 28a) A sloppily written input •••••••••••••••••••••••••• 50 28b) Horizontal lines detected in the fmage •••••••••••• 51 2Bc) Horizontal staFF lfnes removed Trom the image ••••• 52 28d) The Tinal thinned result ••••••••••••••••••.••••••• 53 29} Example oT a note Trame .••••••••••••••••••••••••••• 60 30) Diagram oT input data Tor the vfsual expert system.61 31} A more diTficult measure to understand ••••••••••••• 68 32) A plan and the notation that It represents ••••••••• 75 33} Skeleton plan Tor Figure 8 ••••••••••••••••••••••.•• 82 34) Measure 10 has been moved to the second lfne ••••••. 83 35) Notes with longer duration take up more space ••.••• 8S 36) AlloT the measures ••• now Tit on the line .••••••• 87 37) The Tinal conTiguration... • ••••••••••••••••••••. 88 38) A beat-unft ••••••••••••••••••••••••••••••••••••••.• 9S 39) The same beat-unit ••• in the skeleton plan •••••••• 96 40) Beat-units and sub-beat-units created •••••••••••••• 98 41) Beat-units combfned •••••••••••••••••••••••••••••••. 99 42) Combining level-1 sub-beat-unfts •••••••••••••••••• l00
v, t t
43) Final conFiguration .•••••.•••••••••••••••••••••••• l01 44) Conventions For spacing ••••••••••••••••••••••••••• l03 45) An example of ENGRAVE's output ••••.••••••••••••••. l05 46) An example showing ties and slurs ••••••••••••••••• l06 47) An example showing beat-units and beaming .•••••••• I07 48) A plan aircraFt carrier controller's screen ••• I09 49) The blocks Found in the controller's screen ••••••• lll 50) Initial placement of obJects •••••••••••••••••••••• 113 51) Resolution of objects oFF the screen •••••••••....• 114 52) Final screen...... ••••••• • ••••••••• 115 C-1) Whole note....... ••••••• • •••••••••••.• 155 C-2) Closed notehead.. • ••••••••••••••••• 156 C-3) Open notehead.. . •.•••••.••.•••••• 157 C-4) Sharp •••••••••• C-5) Natural .•••••••
• ••••••• 158 • •••••••••••••• 159
C-6) Flat •••• • ••••••••••••••• 160 C-7) StaFF line ••••••••.••• • ••••••• 161 C-8) Repeat ••••••.••••• • ••••••• 162 C-9) Repeat (beginning). • ••••••• 163 C-10) C-I1 ) C-12) C-13) C-14) C-15) C-16) 0-1 ) E-1 ) E-2)
F lag. • • • • • • • • • • • • • • • • • • • • • • • • • • • • ••• 164 Treble cleF....... • ••••••.•••••••..•..•••• 165 Bass cleF........ .••••••••• • •••••••..• 166 Figure "3" For time signatures. • ••.••••••• 167 Figure "4" For time signatures.... • ••••••••.. 168 Quarter rest....... • ••..•••••..•...••. 169
.. 170 •. 174
Intersection or two degenerate conics....... .178 The geometry For calculating slurs and ties .•.••. 180
E f ghth rest .............................. . NOTES Example .••..•••••.••..•••••.••••.•.••••.•
fx
LIst o¥ Tables
1) Summary of results from Figure 30 ••••••••••••••••••• 67 2) Summary of results from Figure 31 ••••••••••••••••••• 69 3) InitIally. all of the measures ••• of equal sfze •••• 89 4) Measure 5 expanded to its minimum sfze ••••••••..•••. 91 5) All measures are reduced by the same amount... • •• 92 6) All measure are reduced agafn ••••••••••••••••••••••. 93
x
Chapter 1
IntroductIon
ENGRAVE is an expert system that understands musical
notation. Its function is that of an engraver who
extracts individual parts rrom a conductor's score. It is
an expert system in that it captures the expertise or an
engraver who must be able to read a score and generate
notational1y correct parts.
ENGRAVE's input is a grey-tone digital image of a
handwritten musical score (one page at a time). Low-level
image processing routines extract primitive features from
the input image. These features are lines, circles,
blobs, and curved lines. Descriptions of them are made
availab1e to a visual expert system which interprets
these features to determine the pitch and duration of
each note as well as some articulation information. The
visua1 expert represents the separate musical parts in a
music language called NOTES. NOTES is interpreted by a
graphical expert that produces high quality graphical
output on a laser printer or other high-resolution
graphics output device. See Figure 1 for a block diagram
of the system. Figure 2 shows part of a page from a
1
2
Ol6lTA.L 1M A6E
LOW LEVEL V IS 'ON
V IS UAL EXPERT
- NOTES
SP"T'~l PLA.NNtR
,
Figure 1 Block Diagram of the
ENGRA VE Iystem.
3
~ ... .... . :-... '1.J.I) 1C "'U.-.. ~ -----T C PAR
loun4 •• v • CLARINE
ALTO SA HORN
TENOR SA
TRUMPE
T
X 5
X
T
I
TRO .. ION E I 2
I AR I TONE
lASSES
li 1'fWO
PERC .1 .. _e {em ••
'''''' I 0" "" "', )C "'&\1
,.. -......... I"'"
loll ., ,. • ...., v'
n~ .. A
I)C'Ill f(1J Utt1 ~lAJldf, l~~ V'
J ~ .. .. • I -.., II ~ n J (.(0 .. t) "--:
I ..... ·
~ • .J. .J.,. ..t .... J. .t l j
In U " • I
I ..... r _I II a-
"" ~S"11I6 f!ill-, " ~.tr:-'1 m rn r.-"\n ~~ .. .. 1& .. , .. ~: t' I ~ \..:! \.U
C, i . ~, -. T e.
Figure 2 Copy of handwritten input
to the ENGRAVE system.
.
-,,-
- - . -
.. - .. --
'" '--""
r .!~ j. .i. - - . -
• - .",
.. - .. - -.r.-;,h 'm r.1
" . t , UW
.. • •
handwritten score
shows the output
music.
4
(the input for ENGRAVE) and Figure 3
of the C-part in the same piece of
A
ENGRAVE
primary motivation behind the development of
is economics. Music copying and engraving is an
art requiring the services or highly trained experts. The
texts for copyists and engravers reveal the intricacies
of producing readable work. For example, Ross [1970]
gives almost three hundred examples of beams between two
eighth notes. That these professionals are members of
unions only compounds the situation. A publisher has a
choice between having his work done in the United States
or sending it overseas to take advantage of cheaper
foreign labor. In Korea, printing shops are run like
assembly lines where each person makes only one kind of
figure. Printing performed in this manner is relatively
inexpensive, but one must pay a price in the time that it
takes to send the inrormation between Korea and the
United States.
In the vfsion phase, ENGRAVE uses knowledge about
musical notation to interpret the input images.
Traditionally, vision systems have used domain knowledge
5
& .... ::s o
6
at high levels while low-level vision processing has
remained general purpose. During the initial development
of ENGRAVE, the plan was to use general-purpose image
processing techniques to detect musical primitives. A
high-level
was to
notation.
expert system that operated on these Features
contain all of the knowledge about musical
Many diFFerent approaches were 'tried at the low
level with little success. Finally, the decision was made
to use knowledge about musical notation during low-level
processing. The results obtained From this approach were
signiFicantly better than any other approach that was
tried. As a result or this experience, we hypothesize
that low-level knowledge is necessary For a vision system
to perrorm successrully. ENGRAVE supports this claim.
ENGRAVE uses
graphical output.
spatial planning to produce its
Spatial planning is a knowledge-based
technique that uses constraints and production rules.
Unlike its predecessors, ENGRAVE is not simply a clever
algorithm that prints music. Algorithmic approaches are
limited in capability, and systems that use them cannot
be easily modiFied or extended. ENGRAVE's capabilities
match or exceed those or its predecessors. More
importantly, any notation that ENGRAVE cannot currently
1
produce can be added with ease. This is true ror two
reasons. First, spatial planning
ror musical output that captures
problem. Any modirications to the
is a natural approach
the structure or the
system will involve
only a rew rules and will not result in changes to the
overall structure or the system. Secondly, ENGRAVE can be
easily modiried since rule-based systems are highly
modular.
Not only is spatial planning a natural approach
Tor printing music, it can be applied to other domains
that involve placement and interactions or objects in
space. Examples oT this class or problems include
production plant layout and rormatting graphics screens
ror human-computer interaction. By applying spatial
planning to a large problem, we have shown it to be a
reasible approach Tor real-world problems.
The ENGRAVE system may be logically divided into
three parts: low-level vision, a visual expert, and a
graphics expert. Each part will be treated separately in
its own chapter. In Chapter 3, we discuss the low-level
processing that supports the visual expert and the use or
domain knowledge at this level. The low-level system
8
detects primitives that are passed to the visual expert
discussed in Chapter 4. This portion or the system makes
use or rrames and production rules as a representation of
knowledge about musical planning and its application to'
musical output. A central element or the system is a
musical representation language called NOTES. Although a
rull description or NOTES is not within the scope or this
thesis, it is described brierly in Appendix D.
Most or the system is written in PROLOG [Roach and
Fowler, 1983] wIth the exception or the low-level
processing which is written in RATFOR [Kernighan and
Plauger, 1976; Krusemark and Elliott, 1981) and runs on
the GIPSY image processing system (Haralick et al, 1981].
An interrace between PROLOG and Extended CORE graphfcs
(SIGGRAPH.1979; Ehrich, 1983] was developed ror producing
graphical output.
This chapter
Chapter 2
Literature revIew
is an overview or previous work
reported in the literature on printing and recognizing
music. Each subsystem or ENGRAVE is discussed in a
separate
background
chapter.
chapter, so discussion of theoretica1
is reserved For the beginning or each
2.1 PrIntIng
Music printing programs have existed at least as
early as 1965 when Hiller and Baker [1965] reported on a
system that used a music typewriter For input. Figure 4
shows a later system by Boker-Heil (1972] is shown in
Figure 5. This system as well as many or the others in
existence uses
Genera11y, these
enter. This can
an alphanumeric language as input.
languages are diFFicult to read and to
1ead to input errors. A natural way For
humans to enter data is needed.
Besides input 1anguage, both or the systems
mentioned have two major downralls spacing and
graphics. The graphics in both systems are crude to the
9
point
is not
10
that some Figures are not recognizable. Although it
the most diFFicult problem posed by this domain,
it is an important consideration since we would like
attractive output. Proper spacing aids a perFormer by
providing a visual clue to note duration. IF a piece is
properly spaced, a performer will not have to expend
extra effort deciphering the notation [Roemer, 1973, p.
v]. Gomberg [1977] proposed a data structure and an
algorithm to provide proper spacing but neither showed
any output nor indicated that he had implemented his
techniques.
The SMUT system [Byrd, 1974] produces the best
results of the early systems. Figure 6 (from [Byrd,
1977]) shows some output. Output from a more recent
version of SMUT is shown in Figure 7 (taken From
[Hofstadter, 1979]). Figure 8 s~ows the output of ENGRAVE
on the same part.
Other printing work has been reported in connection
with musical editors. These systems generally do not
include all aspects of musical notation but leave some
tasks to the user. An early musical editor that produced
some fine output is described in [Smith, 1973] and an
1 1
example can be seen in Figure 9. The MOCKINGBIRD system
[Maxwell and Ornstein, 1984] provides the best graphics
the author has seen but leaves many notational details
For the user. Figure 10 (From [Roads, 1981]) shows some
output aFter editing to speciFy spacing, voicing and
beaming.
Some graphics work has been done For music
transcription systems [Moorer, 1977; Piszczalski and
Galler, 1977]. The emphasis in these systems is on
interpreting a waveform rather than on graphical output.
2.2 Recognition
All previous work relating to the automatic recognition
of musical notation has dealt with printed music on1y.
The best known of these systems is DO-RE-MI [Prerau,
1971;
that
their
where
music,
1975]. DO-RE-MI is a pattern recognition system
classifies the objects in an image according to
size. This technique works well for printed music
figures are uniformly printed. In handwritten
this technique is not reliable since the the
appearance of a given type of note may vary widely within
a single piece.
12
2.3 General fnrormetfon
For inrormation about the rules or conventional
musical notation, (Roemer, 1973] provides motivation for
correct notation and is oriented towards copyists.
[Stone, 19801 is intended to be a standard for unusual
notations that occur in modern music but also covers some
of the basic conventional notation. A book for engravers.
[Ross, 1970], is a clear and well-organized source
presented in a form that is easy to convert to a computer
program.
13
.J., ...... • " r ,,~ ~ J... ~ .. -.. lIP , •
• ... fI' .. -
IP
- ... J...
• ... .. ''/ • IP
._--" ,.. -
.L I · - "::=::; - r:. .. .. .. ./ ~ T ", p .. ..... "-. ,
_-j ::::==t=='=- .~ '"_·c-.. ./ P ;
'- tr. II' ,... I. · "-:::=
./ v p ,
",
I.- I'l .... L..h . · " "D ... D ..
Ip : ./
J. - ." .. .. ...J.. ... ....... c=:.:~; .. ....... , , ,,--~ .. •• • • . .. • ... ' ... . • • .. • • ...
" L L -t::--
-- "" ....
" "-., , , fj j ..., III " fI ~ fj , ijW " ii "
Figure 4 Output from [Hiller and Baker, 1957].
14
15
STRING aUFlR1ET
[!'lEse. DIM.
p (RES( •
1 7 I' II p
:j r it r Ir I
Figure 6 Output from SMUT [Byrd. 1977].
~~
"" tl
01
...
-~-
fI
tl
,;
_-0
.;
:l
II
16
Ii!!!!! -
~
_I-. - ...---I--~----...
...J -I --
- • • .-A-;" .---- "'--
- -- Ioo",J I .. .fIL .... - --
- I
-- .~~ Ii _____ •••
-- - --
~ I
... • , ... tt .. r- •• tt_ ...
( 1685- J750) I
Figure 7 Bach's "The Art of the Fugue"
as displayed by SMUT.
:~
- ~:- : ~--=
- - --".I)!""..~ '-_r .. .~~~~-.. --.......:
--
... -=~
----+ __ .I
;';-III~.? "ff.ii ~>-- .. ii III
I
--..
.!!..~- ... ... .. -
(
(
(
(
17
OBOE
ClU' J N£l IN 8_
BASSOON
~
I'" JI..
,'" 1_
.....
'. --"
I" ..........
,..,..,
"-
18
TRIO FOR WOODWINDS I. FOR "ADELEIN£: ANO DARIUS IHLHAUD
.... DOI.tl ;-1 ~
It 1 -==L i"
~ OOlCE' r -
........ 1
~P I
==t p J .. ~-- 't!/
-- ,~ ... ~ .. - .... .. • TfIlflO
- - 1- .. l IOOtO STtll.~t '11.1 ttfSC, _ .... ~ -'_"'-::- '-~ ......
"IU tlltSC - , F .,\ POtO ST.INl:t - . - '- ..... -
Figure 9 Output from [Smith, 1973].
~
LELAND SMITH
SUlITO ....
I
DIll, I
DIll, I
.IOOtO tUSt.
-1
J .- J ,
,.. j .J ti _
)
rn "11
~ -------:. ~-
3 3
m m
19
.. t
r-- -I""""'"
I
Figure 10
•
m
Edited output from Mockingbird.
Chapter 3
Low Level Vision
Essential to any vision system is some kind or early
processing that can provide inrormation in a usable rorm
to higher level routines. In ENGRAVE, low-level routines
detect primitives that are passed on to the visual
expert. Our primitives are the commonly occurring shapes
in music: circular blobs (ror closed noteheads), circles
(open noteheads), horizontal lines, line segments (not
horizontal), and arcs. The location and orientation data
ror each primitive are made available to the visual
expert system through a rile. Currently, computer vision
1s thought or as being divided into two areas of study:
low-level and high-level computation. Under the current
paradigm, low-level vision consists or general-purpose
operators that extract information rrom images.
High-level processing interprets this inrormation using
knowledge about fts domain [Ballard and Brown, 1982, pp.
2-6].
During the fnitial phase or development, ENGRAVE was
designed under the current paradigm from computer vision.
Many or the standard techniques ror line detection were
20
21
tried with little success. These techniques could only be
made to work consistently for a small class of images.
This is particularly true of handwriting where there is
large variation between one person's handwriting and
another's. When the system parameters are adjusted for
one set of images, we find that the operators do not work
well for another set. It is difficult and often
impossible to adjust the parameters so that a system will
work reliably for all classes of images in a single
domain. Eventually, we concluded that general-purpose
techniques would not work and that explicit use must be
made of domain knowledge. Concerning the segmentation of
curves, Rosenfeld and Johnston speculate that "if the
user has special knowledge about his class of curves, he
might well be advised to devise his own segmentation
procedure taking this knowledge into account" [Rosenfeld
and Johnston, 1973]. An expert system for low-level image
segmentation is described in [Nazif and Levine, 1984].
The knowledge in their system is about low-level image
processing. We propose not only the use of image
processing knowledge, but of domain knowledge as well.
When domain knowledge was used at the low-level in
ENGRAVE, the results obtained were remarkably better than
any results obtained under the current computer vision
paradigm.
The
provided
22
horizontal staff lines in a musical score
a particularly difficult problem which was
solved by incorporating knowledge about musical notation
into the low-level process. Staff lines intersect most
figures in a piece of music. Should the intersection
pixels be labelled as part of the staff line or as part
of a figure? In reality, these pixels are members of both
the staff and the intersecting figure. Most labelling
schemes, however. can only label these pixels with one
label. Prerau [1971] simply removes the staff lines from
the image thereby causing notes and other figures to be
broken into pieces. These pieces can be easily
reassembled because of the regu1arity of printed music.
With handwritten music, we cannot make use of such
regularity. Instead, we make use of knowledge about the
domain first to detect horizontal lines and then to
remove them intelligently so as not to lose any
information.
The Hough transform has been used successfully in
many applications to detect straight lines. circles, and
ellipses [Duda and Hart, 1972; Tsuji and Matsumoto,
23
1978]. However, the Hough transform has only been used to
detect a few figures at a time. On a single page of a
conductor's score, there are more than 50 stafr lines in
addition to the lines for note stems, bar lines, etc.
In' this application, the Hough space is too noisy
for us to be able to detect all of the figures reliably.
Less prominent figures are obscured by the stronger ones.
Watson et ale [1984] have developed a line tracker that
functions well in a similar domain. The tracker uses
grey-tone values to determine whether to a assign a pixel
to a line or to start a new lfne. Because of the many
line intersections, the line tracker tends to turn
corners or to break notes and other musical figures
apart.
Closed noteheads can be detected as convex blobs. A
spoke filter [Minor and Sklansky, 1981] was tried, but it
detected the line intersections as blobs as well as the
noteheads. In some cases, the intersections were stronger
than the notehead. Haralick [1974] gives a circularity
measure that is simple and works we1l when used with a
threshold. The region detection portion or Watson's line
tracker also works well detecting these noteheads.
page
image
24
The input image is a grey-tone dfgitization oT one
oT a score. This image is converted to a binary
using the double-adaptive threshold (OAT) oT Watson
et al (1984]. Figures 11 through 13 show some samples aT
an image aTter thresholding.
ATter thresholding,
perTormed to determine
some preliminary processing is
line angle and thickness. Using
this inTormation and knowledge oT our domain, horizontal
lines are detected and removed Tram the image. Once the
staTT lines are removed, we are leTt with an image where
the notes and other Tigures are all isolated. These
Tigures are broken into their component line segments.
AlloT the inTormation Tram this low-level system is
passed to the high-level expert through several Tiles.
3.2 Line Angle and ThIckness
Line direction and thickness are important
propertfes that need to be computed. Line direction gives
essential inTormation to help determine horizontal line
25
1~~-~~~~1--~~~--~~2----~-~~-3---~-~---4~------~-~1 I J I J J J J J J J J AAAAAA J I AAM,A,.4.AAAAA J J AAAA J J AM J J AA J J A AAA A AA J IAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAI JAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAJ I MMA /lit itA J I AAAAA itA itA J I MAA itA M J J AAAA A AM J J AAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAJ IAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAJ 2 M AAAA 2 I /It AA J J /ItA ,. AA J J AA A AA J I AAA AAA A AAAAAA AAAAAAAAAAAAAAAAAAAAAI JAA~~~AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAJ J AA AAA AA J J AA AA AA J I itA AM M J I AAA /lltAA AA I 3 AAAA AAAAAA /lit AAAAAAAAAAAAAAAAAAAAAA3 JAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAJ J MM /lit A J J J I J I _ I J /lit AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAJ JAAAAA~AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAI I -I J J 4 4 I J 1---~~-~-1~~-----2~--~---~-3--~---~~-4--~---~--5-J
Figure 11
Example thresholded imaae Irom the handwritten Input. An ei&btb Dote beamed to • half Dote. The bottom of tbe eiabth Dote tail coincides with the .taff line.
26
J-~~~~--~-1--~---~--2~~~---~~~3-~~-~--~-4----~--~-S~J J I J I J I J J J J J I I AAA J J AAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAI IAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAl I AAAAAA AAA AA 1 J AAAAA IttA IttA I J AAAM A.AAA J J AAAAAA Itt AAA J J AAAAA AAA AAAA I JAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAI IAAAAAAA AAAAAAAAAAAAAAAAA AA IttA AA Itt AAAAA I J AA AA I J itA AA I J M AA I 2 . M AAA ~ JAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAI JAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA It AAA J J itA "A I I AA AA J I AA AA I I AA AA I JAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAI I ~"JIt. ... ".AAAMAAAAAAAAAAAAAAAAA A It A I I Itt I 3 M 3 J Itt J I I IAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAI IAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAA AI I I I I J I J J J I 4 4 I I 1~-~~~----1--------~2--~-----~3--~---~--4-----~~--'-J
Figure 12
Example tbresbolded Imale from tbe lIaDdwritteD iDput. A quarter Bote followed by • lIalf Dote. Tbe top of tbe balf Dote tOiDtides witb tbe top Italf liDe
27
1~-~~~~~~1----~--~-2~--~--~--3------~--4--~------~~J I M I I AA I I AA I J I J I I A J I AA J I AA I I AA J 1 AA A AAAAAAAAAAAAAAAAI JAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAI JAAAAA AMA A A AAAAAAAA A I J AA AA AA I J AA AAAA AA AAAA J J AAA AA AA AAAAAA I J AA AAA AA AAAAAAA I JAAAAAAAAAA~·AAAAAAAAAAAAAAAMAAAAAAAAAAAAAAAAAAAAAI JAAAAAAAAAA A A AA AAAA AAAAAAA J I A A AAAA AAAAAA I :2 . ItA AMAAA AAAAA:2 J AAAAA AM AAA J J . AAAAAAA AA AA AAA J JAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAMAAAAAAAAAAAAAAAI J MAAAAAAAAAAAAA AAAAAAAAAAAAA AAM AA I J AA AAA AA I J AA AA AA I J AA AA AA J J AAA AA J JAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAI 3AAAAAAAAA.4:AAAA AAAAAAAAAAAAAAAAAAAAA AA AAA 3 I A A J J A I I J J I IAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAI IMAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAA A J I· J J I J I 4 4 J I J-----~-1--~------2--~~---~-3--~--~~~-4--~------'-I
Figure 13
Exam pie tbresholded Imale from tbe baDdwritteD iDput. Quarter rest, Datural, quarter Dote with dot above It. Note break In starr liDe Dext to rest; also, the Datural bas ODe discoDDected tail part, a Itaff liDe throulb its middle, aDd its top borizoDtal IiDe coiDcides witb a .taff IiDe.
28
pixels and thickness can aid in the identification of
figures that cannot be otherwise identified. For
example, many closed noteheads in our fmages are so small
that they cannot be detected reliably by the circularity
measure. Our eyes, though, can discern their "presence
(See Figure 14). These figures can be properly identified
later by observing that one end of the stem is thicker
than the other.
To compute this information, an odd-sized circular
window is passed over the image. If the center pixel is
black, then the window is scanned to find the black pixel
farthest from the center. ,
The angle of this point and
distance from the center define a vector that
approximates the line direction at the center pixel. To
compute line thickness, the window is scanned along a
line perpendicular to the direction vector until a white
pixel or the edge of the window is encountered.
In many cases, a straight line between the farthest
pixel and the center will cross white space. If this
occurs, the angle will not correspond to the line
direction. To prevent this from occurring, some pixels
are deleted from the window before computing the angle.
29
1~~~~~--1--~~~~---2-----~---3--~~~~--4--~~~-~--5-J J AAM"A J J AAAA I I AA I J J I I JA AM I JAA~~~AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAI I It It AAAAAA AAAA AAAAAAAAAAAAAA AAAAAAAAAAAAAAAI J ~A J I A~ 1 I MAA I I ~A I JAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAI JA AA·~AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAI I A.4.AA.6.A AAA J J AMMA MA J I AAAA A AAAA J J It AMAAA AAA I JAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAMI 2AA~~~~~~~AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA2 J Mit AA J I It It itA J J itA itA J I It AA I IltAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA A AAAAA AAI JMAM.,V~.AAAAAAAMAMAAAAAAMAAAAAAAAAAAAAAAAAAAAAI J itA I I A I I A J 3 3 IAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AA A I JAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAI I I J I J I J I I I I I I I 4 4 I I 1~~~--~--1~--~----2---~~~--~3----~~--4-------~-~~1
Figure 14
Example tbresbolded Imale from tile .andwritten loput. UslOI a malDifyiol &Iass, we were able to determine that this result was Dot uDfaithful to the orilioal. Dilitization of oriliDal pale was 1200 x 100 (horizontally, therefore 100 IcaD lines per incb). The blob to the left was meant as a sharp, the filure to the rilht is a quarter Dote. The quality of this iDput is Dot typical.
30
Any black pixel that (1) is not the central pixel, and
(2) is adjacent to a white pixel that is closer to the
center than itselr, is deleted. This process is repeated
until no more pixels are deleted. Figure 15 illustrates
this process. Arter deletion, the line angle and
thickness are completed.
To detect horizontal lines, the radius or the window
should be set to a large value -- at least halr the width
or one starr in the 'image. This will Torce pixels at the
intersection or a staff line and a figure to have a
horizontal line angle. Figures 16 through 18 show the
line angle images computed from the examples.
3.3 Labelling the Background
Many figures such as half notes and accidentals have
"holes" that iF detected can be useFul in identirying the
Figures. To this end, the image background is segmented
by a simple labelling process as follows:
- Ir an unlabelled white pixel has no labelled white a-neighbors, then give it a unique label.
- IF a labelled white pixel has labelled a-neighbor, then label pixel with its neighbor's labe1.
a differently the original
(¢~" ... J.'<V ~.Q.. \ .e,.. \:, ot/\
-
'figure 15 Coll1Vut.ation of line an~e.
/
\.
I
I
1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
32
~~~lf ., ., " '- 'v""~ ~~~--,)'" ,,~~~~ .......... t""t"" ..... ltr r """" • It • \I" -:
~ • -"
, "
CI) ...
" •• ~ """ :::s 0.0
" ~ , """" i.t:
~ "-" 0 -» » i "" 0.0
i i ~ ,. .. itf' it "" r:: ;s
i i~~lI;itll;i tt '" \C C
i t 't~~i-- "" - 0 0.
i i Iti" ii "" CI) CI.l ... CI)
:::s t i i • i""" ii "" 0.00
i T i" i iT '\, i.t: C)
CI.l
t t i" i iT 'l'\ CI)
0.0
l i i'\ it IT '\ co
it i ii~rr i '\ .5 T i it ~ii ~~~ii
CI)
t ~ ~~~~~ ~~"4-Ci;,
~t~~~~~t ~~~~tt ttlt a CIJ
i it i ~ii~ ii c
i i t it»-- iiw ::i i i i+ ii~------ti~ it iJ, i+ ii ~~ i ii iJ, iJ, ii ii ii iJ, iJ. ii ii ii it iJ, ii i it it
++ ii " i ii it it " i ii it if ii 'i
i't it it ii ~ i it ii it ii .\0 i ii it ii it 'i ii it iT ii f i ii ii ii it ., t tf 1'i ft ii t i Ti ti if tT , f if if iT ii 1'i
33
\,J, J, J, J, J, • 'oaIJ, J. J, J, J, ~J, J, J, J, J, ...
~± J, ± : J, J, • J, J, .1 '" .... ..l. ... '.11~ ..l. J, II J, J, '"
...,., .... J,.,. .... '.II .... '" .1 J, .1 '" lIfJ, "'~"'J, J,rr J, J" lItJ, ~~+ )J, J, J, ,
'.1 ~~~~~~~J,±~~~~J,~ ± ! tJ,~ ~~~~~~J, ~~~~J, J,01 ~ )t~ J,J, J, J, ,
J,J,~A~~ ... t~ J,J, J, J, J,J, ~1~ .1..1. J, .1.J, J,J, t J,.1. J, J,J, M J,J, l' • J,J, t · J,J, -J,J, J,J, J,J, G) .... J,J, 4 J,J, J,J, :::s
CO
±± 4 ±t ±± ti:: 0
J,J, J,J,
it ...
I • J,J, i±
co 4 =
J,J, J, ~ r- =
1.1 J, ~ J,J, .1 J,J, _ 0
c. 1J, J, • .1J, J, J,J, G) CIl .... ~ 1J, il J,J, J,
,I t =' I:: coo
Ii J,J, J, ti:: Y
J,~ J,J, ±
CIl G) co
J,l J,J, CI:I
rJ, J,-l. J,J, ±
.5 J,-l, J, G)
I J,-l. J, -;0
s J,~ J,
G)
J,-l. J, = J, , J, :.:J
'" , '" J,1' f t1'
if '.II 1'1' J, lI~\rt f • rr -+ -+-+-+-+-+ t-+ ....... lt'" ~ .... ~~~+-+-+--+ -+-+-+-+--' l~~~"" .... 4f" ........ ~~ .. ..... ~'" , .. "A~~~
, ~ ~ ~ '" t ~ 11\ t ,. ~ t ,. It t ,. It t 1 It t
• ...
t"" :r (II
§ (JQ
(i'
9' I»
~ I:n
n :!t OQiQ .., = (; (; I:n
"0 o -:s co Q.
S' (JQ
... o ~
OQ' = .., ('II -~
j i i
+t tt ii if tt iT tt ii ii
i ~ i po i ,. i
i i i i
· i i i i i i i i
++ ++ ~I · ~+ +~ ~ " ~~ ~~ ~ . ++ ++ ~ ,. .l.J. J,J,'''- ,. ,.., "~"' .. lt4 '''' tit "'t,.,. It''' A4'1t ... 11,." • fot Aot"lIt f ..,~~ ...
" "'''I'' .... t t t t ,.,,. . . ~
ttc ttl' ~tt~~,.~ttl' •
tt""",."ttl' t tt t t
" " " 1- • + ~ t -i J, 1t i i J, ¥ • j f 1 ~ i i 1 ~ i i 1 .. ~Ttlltllt.i 1 t ~~~~t~~~~~i~~~~~ r t .... It illt 4~~f"'~~~ r t ~lIlt i • ~i r ~"~i·i r ""~i 1i T .. It". ......... i ... ~~~~i~ i ~ .... ~". .... ~i~~~~~i if ~~~,.1 i i ii "i i iJ. iii iJ,
.. 1 i f fJ, poi i f iJ,
f t i jJ, iii i' f i i i~ i i ""i~ i~ i i ~"lIti~~ i~ i i"~~~~f~~~ it
~f~~~~~i •• ~~~i~~~ ·i~ ~~~T~~~~~i~~~~~i~~~ it
~i i f~i"''''''' i~ i i fi~'" i~ iii i~ i i f i~
vE
• .. . .. ( .
35
This process is continued until no more labelling is
done.
The difference between the "holes" for which we are
looking and the ,other white areas in the image is that
the holes are significantly smaller and they are located
inside of musical figures.
3.4 Horizontal Lines
The most important figure to deal with in music is
the horizontal line. Nearly every figure in a piece of
music 'intersects a staff line. Our strategy is to detect
horizontal lines and then remove them in a manner that
does not break up the other figures. We use angle
information and pixel adjacency to determine whether a
pixel is a line pixel. The rules are as follows:
A black pixel is marked as a horizontal line pixel if'
it is to the left or right of a line pi xel; or it is horizontal and it 8-adjacent to a 1 f ne pi xe 1 •
If a pixel is horizontal and has no marked adjacent pixels then mark it with a new label.
A white pixel is a horizontal line pixel ,if it has a marked pixel to its left and to'its right. (This rule allows for the possibility that a line
36
may have a i-pixel break.)
Orten, it is impossible to tell by its neighbors
whether or not a pixel fs part or a horizontal line. A
pixel may even be both part or a horizontal line and part
or some other rigure. These pixe1s are given a special
label that indicates that these pixels are
"questionable." That is, it is uncertain whether these
pixels are part or a horizontal line or not. A list or
these pixels is stored in the output rile ror use at
higher
pixels:
levels. Here are the rules ror questionable
A pixel is questionable ir it is not horizontal and 1s above or below a line pixel.
A pixel is questionable ir questfonable pixel and questionable pixel.
it is below
above a another
- a questionable pixel above a line pixel is re-labelled as part or that line ir none or the a-connected pixels above it are black.
- a questionable pixel below a line pixel is relabelled as part or that line ir none or the a-connected pixels below it are black.
As pixels are labelled, they are added to a list or
pixels that all have been marked with the same label. A
list is also kept or questionable pixels.
37
Some pixels may be elements both or a line and the
questionable list. These will generally be pixels that
are both part or a line and some other Figure.
AFter all the line pixels are identiFied, there
usually are adjacent regions that have been labelled
diFrerently. With the exception Or the pixels that have
been labelled "questionable," these regions are part Or
the same line and are merged.
A least-squares Fit is used to determine the
equations Or the lines and these coerFicients are made
available in a file along with the list or questionable
pixels. We can identiFy starr lines by the ract that they
are almost as long as the width of a page and that the
other horizontal lines are much shorter. Figures 19
through 21 show the line pixels that were detected in the
original images.
3.5 Horizontal Line Deletton
Once the horizontal lines have been detected, they
are removed rrom the image leaving the other Figures
38
1~-~---~-1----~--~~2-~--~----3----~---4----~~---S-J 1 J I J I J 1 J 1 J I J 1 J J J I J 1 1 JIB B IS J ICCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCJ ICteeeceeeececccececcccccccccccceceeeecececcccccCCCCI 1 J 1 J I J I 8 I 1 IBBS RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRJ JRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRJ 2 8 B B 2 I J I J J I I B ISB B IB BB8 QQGGQgQQQgQGGGGGQQgGGI IQQQogQQgcQogQggQQQQQQQOOBQgQgQggQQQQQQQQQQQgQgggQgQJ I B J I J 1 I 1 I 3 B BBB I 44444444444444444444443 1444444444444444B48B444444444444444444444444444444441 J BBBB BB J J J J J J I I B '~~~5~5~"~~~~555'55'5""""I JSSS5~SS"SS'55'5"55'5"55"""'5""""""""51 I . I J I 4 4 I I 1~-~--~---1--~------2--~~--~--3~-~-~--~4----~----5-J
Figure 19 Horizontal lines, staff lines,
extracted for image in Figure 11.
39
J~-~---~~1-~-------2------~~~3--~-~---~4-----~-~~5-J J J J J J J J J J J J J J BBB J J CCCCCCCCCCC CCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCI JCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCI 1 8 8 1 J J J J J J J 8 B 8 J JOQQQOQOOOOQQOOOOOOOOOOOOOOOOOOOOOOOQOBOBOOOOOOOOOOOI JBBBBBBB OOQQOQOQOQOOOOOQO IB IB BI I BIB J J J J I J I 2 I 2 10000000000000000000000000000000000000000000000000001 JoooooOOOOOooOOooOOOOooooOOOOooOOOOooOOOOOO I IBB J J J J J J I J J JVVVVYYYYYYYYYVVVVYYVYVYYVYVYYYVVVYVYYVYYVYYYVYVYYYYJ JYYYYYYYYYYVYVVVVYVYYYYYYYYYYY B B B J J J 3 3 J I J J JUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUI JUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUU BI I J I J J J J J J J 4 4 J J J~~~~~~1---------2-~----~~-3---------4-----~~--5-J
Figure 20 Horizontal lines, staff lines,
extracted for the image in Figure 12.
40
1~~~-~-~~1~-------2~--~--~--3---~---~-4-~--~----5-1 I I I I I J I I I J I I I I J J I I 1 A BBBBBBBBBBBBBBBBl JBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBI ICCCCC C C C C c ceaaa C I I I I J I J I D D J IEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEI IFFFFF FFEE F F F EEEEEEE I I F I 2 2 I J J QgQ QQ I JHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHI J HHH-IHHf+IHHI+I I I I I 11 I I I J HHH J r I I J J I I I J I IKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKI 3KKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKK LL L L 3 J L J I J J I I I 1"~KMMMM""MMMMMMMMMMMMMMMMMMMMMMMMMMMMMMM""MMMM"""""r 1""t9V1~MMHJ1M~I1MMI1HMHHM NNN N I I I I J J I 4 4 J J 1-~-~--~--1~~~-~---2--~------3------~--4---~-----5-1
Figure 21 Horizontal lines. staff tines.
extracted for image in Figure 13.
41
intact. This is not done by deleting line pixels, but by
marking the pixels that should remain. The rules for
marking follow:
Mark a black pixel if - it is not a line pixel; - it is not horizontal but the pixel above
or below it is horizontal; - it is a line pixel and is above or below
a marked pixel; or - it has marked pixels to its left and to
its right.
When this process is completed, the musical figures will
have been isolated. Figures 22 through 24 show the
results of line deletion.
3.6 Figure Segmentation
Once the figures have been isolated, they are
segmented into their components and this information is
made available to the visual expert. First, the figures
are thinned using the algorithm given by Arcellf and
DiBaja [1981] in order to obtain chain codes. Their
thinned figures are segmented at junction points.
Junction points are defined as pixels where three or more
segments having the same label meet. After segmentation,
we are left with the chain codes of all of the arcs in
42
J---------l----~~---2--~~~--~-3-----~---4--~~---~-~-J J J J I J J I J I I J MAAAA I J AAAMAAAAAA J J AAM J J AAA J J AA J JAM A J I MAAAM A AA J I AAAAAM A AA I J MAAA A AA J J MAAA AA AA I J MAA AAM I I MAA AAM I I AAAA AAAAAAA J I MAA AAAAAAA I 2 ItA ItAAA 2 I A AA I J M A AA J J M A AA J J AM AAA AA J I AAA AAAA AA J I AA AAA AA J J AA AA AA I I M AAA AA I I AM AM AA J 3 M" MA litA 3 J MA AM AA J I M J I J I J I I I J I J I J I J 4 4 I J 1~~-~-~~1----~~--2-~-----~3--~----~-4-------~~~-1
Figure 22 Musical figures remaining once the
horizontal staff lines have been intelligently removed from the image in Figure 11.
43
J~--~-~~~-1~--~~~~-2-~-------3--~~~-~-~4-----~---5-1 J J I J J J 1 1 J J 1 J 1 'A J I AAAAA A AA J J AAAAA AA AA J I AAAAA AA AA 1
·1 AAAM AA AA J I AAAAA A AAA J I AAAAAA A AAA J I AAAAA AA,AAAA J I AAAAA AAAAAAA J J AMA AltAA 1 I AA AA J J M M J J AA AA J i2 AA AA 2 J AA itA J J AA itA J I AA AA J J itA itA J J itA itA J J AA itA J 1 AA itA J 1 AA It J J A J 3 M 3 J A J J J J J 1 J I J J J J J J J J J 4 4 J J 1-~--~~--~1--~--~~--2~~-~---~-3--~~~---4~--------S-J
Figure 23 Musical figures remaining once the
horizontal staff lines have been intelligentl y removed from the image in Figure 12.
44
J- --J---------~---------~---------4---------~-l J J 1 AA J 1 AA ) 1 1 J 1 1 A J J AA 1 1 AA 1 1 AA J J AA A 1 J AA AA AA 1 J AA A~ AA J I AA AA AA J I AA MAAA AA AAAA J J AAA AA AA AAAAAA J J AA AA AA AAAAAAA J J ~AA AA AAAA A~AAAAA 1 J AAA AA AAAA AAAAAAA 1 J A A AAAA A~AAAA J ~ AA AAAAAA AAAAA 2 I AAAAA AAA AAA 1 1 AAAAAAA AA AA AAA J J AAAAAAA AA AAA AAA I J AAAAAAA AA AAA AA 1 J AA AAA AA J J AA AA AA 1 1 AA AA AA J ) AA AA J ] ' ... A AI\ J 3 AA AA 3 J A A 1 1 A J J J J J J 1 J J ] J J 1 Jl ~ 4 J J J---------1---------2---------3---------4---------~-J
Figure 24 Musical figures remaining once the
horizontal staff lines have been intelligently removed from the image in Figure 13.
45
the image. Each individual arc is Further segmented by
breaking the arcs at curvature maxima as described in
RosenFeld and Johnston [1973]. Figures 25 through 27 show
the results of thinning and segmentation.
Descriptions of all the features detected are output
to a file. This rile is loaded by the expert system which
uses it to identify the figures.
3.7 Summary
The results obtained using these techniques were
quite good
cases, the
segment the
available to
-- certainly far above expectations. In most
system was able to correctly isolate and
figures, and enough information was made
identify them correctly. In portions of the
score where handwriting was sloppy, some information was
lost. Figures 28a through 28d show successive results of
processing a sloppy image.
Originally, our plan was to use general purpose
image processing techniques as early processing for a
visual expert system. It was round, however, that
general techniques could not be made to work we1l.
46
J~~~~~-1~--~-~--2-~-~-----3----~----4---------'-1 J J J I I I J J I J J J J AAAAAAAA J J itA J J It I I A 1 J A J I It D J IDE J I B D E J J B 'D E I J B D E J J B D E J J B D E J J B DDU J 2 B F ;l J B F J J B C F J I B C F J J B C F I J B C F I J B C F J J B C F J I B C F I J B C F I 3 F 3 J F J J I J J J I J J -I J J I J I I I 4 4 J J 1~-~-~~~-1~~-~----2--~---~--3-~--~----4--~----~-S~J
Figure 25 The final, thinned image corresponding to the image in Figure 11. Other information
including where blobs (noteheads) exist and their extent~ candidate horizontal lines connecting
music figures. etc. are passed to the high level expert.
47
1~-~~-~~~-1~---~---2~-~-~-~~-3---~~~--4~~---~~--S-J I I I I I J I J J J J J J J J J J A J 1 DBA 1 J DBA J J DBA J J DBA J J D 8 A I I EF BAAA J J F C J J F C J J F C J J F C J 2 F C 2 J F C I J F C I J F C J J F C J J F C J J F C I J F C J J F I I F J 3 F 3 J J J J I J J J J J J J J I J J J I 4 ~ J J J~----~---1~-----~-2-----~---3~--~~-~-4~----~---5~J
Figure 26 The final, thinned image corresponding to the image in Figure 12. Other information
including where blobs (noteheads) exist and their extent. candidate horizontal lines connecting
music figures, etc. are passed to the high level expert.
48
1--~~---1~~~~----2----~~---3--~--~---4---------'-J J I J A I J A J J I J I J I J I J B J J B I I B 1 J B J J F Bel J F B C J J F Bel J F Bel J F Bel I F Bel J FIe Q I I F B C Q I 2 F DeC Q 2 J F D g I J FF . D Q I J FFF Q I J F E Q I I F E Q I I F E Q I I F Q I J F Q I I F Q I 3 F Q 3 I Q I J I I J I I J J J I I I J I J J 4 4 J I J~~~~~~1~~---~-2--~-~-~--3--~~-~~--4-~---~~--'-I
Figure 27 The final. thinned image corresponding to the image in Figure 13. Other information
including where blobs (noteheads) exist and their extent,. candidate horizontal lines connecting
music figures. etc. are passed to the higb level expert.
49
Domain knowledge had to be incorporated at the low level
to achieve satisfactory results. This is an important
result since vision systems typically do not explicitly
incorporate knowledge
Traditionally, world
during
knowledge
low-level
is used
processing.
only ror
"high-level vision" and low-level processing is done
using general purpose mathematical models.
Much or the lower-level knowledge in this system is
procedura1 not declarative. Where it was used. the
rule-based approach gave the system the flexibility it
needed to do its Job. Several of the processes in this
low-level system were implemented under the old
general-purpose paradigm. The system would benefit rrom
also making these processes rule-based and rlexible. The
importance of this approach is not that we have achieved
the ultimate system but that a dramatic improvement
resulted from adding world knowledge to low-level
processing. This suggests that a vision system must
incorporate world knowledge at all levels to work
successfully.
50
1---------1---------2---------3---------~---------5---------6---------7---------1
I A AAAA I I AAAAAAAAAAAAAAAAAA I I AAAMAM. I M~A J I AAAAA • I AA AAAA I I AA AAA J I fIlA MAA J I A' A N!AA " I I ,,~~ :~ :! I I "AM AA A AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAJ IAAAAAAAAAAMAAAAAMAAAAAAAAAAAAAAAAAAAAAAAAAAAAMAAAAAAMAAAAAAAAAAAAAMAAA AA I I A AAAM M AAAAA AA I I AAAA AA A AAAAA MA I I AAAA M"" A AAAAAA A AAAM I • AAA A AA MAA AA M AMAA I a . AAA AAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAMAAAMA~ IAAAAMAAAAMAAAAAAAA""MAAAAAAAAAAAAAAAAAAAAAAAAAAMAAAAAAA""AAAAAAA AAAAAMAAAA I • A AA AAAAA AAAAA AA AAA AAAA I I AA MAAA AAAAA MAAA AAA 1 I" AAAAAA AAAA AAAM AA I I A "AAAA A AA AA AA I J A AAAAA AA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAMAJ IAAAAAMAAAAAAAAMAAAAAAAAAAAMAAAAAAAAAAAAAAAAAAMAAAAAAAAAAMAAAAAMAAAAAAAAAAJ 1M AA All. AA AM J 1 A AA A AAAA I 3 AA AA A AAAMA 3 I AA AA AA' A AAAAAAAA I IAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAMAAAAAAAAAAAAAAAAAAAAAAAAI I AAAAA M MAA M A /ltAAA AAA AA " I I ItA AA AA • I M AAA AA I I MA AAAA A A I J MA AAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAMAJ JAAAAAMAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAMAAAAAAAAAAAAAAAAAAAAAAJ 1 AA AA MA AAAA I ~ All. MAA AAAAAM ~ I AAA AAAAAAAAAAAAAAMA J J AAA A MAMAAMAA I J AAA AA I I AAAA MA I J AAAAAAAAAAAAAAA I I AAAAAAAAAA I J J I I I J 5 5 I j
I J 1------l------iZ-------3-----~--------S------6-----7-----1
Figure 28a
A sloppily writteD IDput. A couple of slurred dotted ei&htb .. sixteeDtb Dote pairs. Tbe secoDd sixteeDtb Dote bas a Ibarp accideDtal to Its left. Tbe secoDd tail OD eacb sixteentb Dote coincides witb a staff line; the visual cue sbowin& that it is • sixteentb Dote Ii tberefore abliterated.
51
1---------.---------2---------3---------4---------$---------6---------7---------1
I • IDDD I 1 +.+++.++++++.+.... I I .+++++++ I I + ••• +++ I I II I I 1 J J , 1 I I I I I I
I I 8 2~~~~~~~~~~~~~22~~~2222~22~~~~~22~22~1 1~2~~~22~~2~22~22222~22~222~2222~~2~22~222222222~222222222222222~2222222222~ DD I I a a I I I I I I J R I II I 2 .v9999 .9999999999999999999999999999999999999999999999999~ 199999999999999999999999999999999"'99 •• 9999'9999999999999999999999999999999~99 I
- J a • D J I I I I I I J 80 +.+++++ •• ++++.+++++++ •••••••••••• + ... . 1 .............. + •• ++.+++ •• +.+ ••• ++ •• ++++++.+++++++.+ •• +++++++.++++.++.+ ......... , 188 a • I 1 I I 3 ID 3 I II. 11$... ••• I I ••••••••••••••••••••••••••••••••••••••••••••••••••••• • ••• ••••• •• • •••••••••••••• 1 188888 10 IB 8B • aBBB • I J I I I I 1 8 I I I IDD8BBEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEFEEEEEEEEEEEEEEEI IEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEE£EEEJ 1 10 I • I 4 4 I IDDDIDBDB J I MAAAMAMA I I J J an n I I ,....."''''tttt1rIM .. ,rttft'tf1Mr1 I I BUUUUUDBBH I 1 I J I J I ~ 5 • I 1 I 1------1------2---------3----4-----$------6------7-----J
Figure 28b
Horizontal liael .etected la tbe Imale
52
J---------I--------::!--------3---------4---&------6-----7-------. • I • I I I I I I AAA I • AA AAAA I I AA AAA I I M A~ I J A AAA I I A AM A I I AM AA AA I J MM A AA J I AAM A A Afl.fl.AA AA I J AAAM A A AMAA AA I I AAAAA A "'-AAAA M I I "'-AAA M A AAAAA AAA I J AAAA MA A AAMM A AAAAA I J AAA A AA AAA "'-A A AAMA I 2 AAA AAAAA MAAA AAAAAA AAAAAAA ::! I AAA AAAAA MAAA MAAM AAAAAM I I AA "'-AAAA AAAAA AA AI\ AIIAA I I AA AAAAA AAAAA MAAA MA I I A AAAAAA AAAA AAAAA AA I I A AAAAA A AA AA AA I I A AI\AAA A MAAA AAA J I A AAAAA A AAAM AM I I A AA A AAA I '. A AA A AAAA I 3 All. AA A AAAA 3 I Alit AA AA AMA • I AAAA A... AA MAA I I ItA AA All. A I 1 AlIt It... AA I 1 /ItA AAA ItA I I MA AAA A I J MA AAA A M AAA I I MA MA A ItA MA I I AA A AA AAA J 41 AA MAA AAAAAAA 41 I AAA MA AAAAA I J AAA A A A I J AAA AA J J All. AA I I A I I I I I J I I I , , • • J I 1-------1-------2--------3-------4--------'-----6------7-----1
Figure 28c
HorizoDtal .tart IIDes removed from tbe Im8lt.
53
1--------1-------:1--------3---------4-----1----6--------7------1 I ) I I I I I I I J I A • I AAA I I A AA J J A MA I I A M I t A J J It A I I A It J I A A A A J I A A A It A I 1 A It A A I I A A A A J I A AlIt It AA A A J J A A A itA It It 1 :I A A A AA A A :I I A A A A MA J 1 A A AlIt AA A J J A A A AA A J I A A A A A J I A A A AA A I I A A A AA A I I A A A AA A I I A A A A I J A A A A J 3 A A A A 3 I A A A A J I It A A A I I A A A 1 I A "A J J A A It J I A A A J I A A It I I A A A A A I 1 A A A I 4 A A A .. I It A AAM J I A A A I I A A J J 1\ A I J A J I I I ) I J I I ~ ~
• • I. I 1------I-------;!--------3----4-;.----a-----6-------7-----)
Figure 28d
The fiDal tbiDDed result. Tbe opeD Dolebeads on the second dotted eilbtb-sixteentb will be easily eliminated by the hilb level expert system. Deter-mininl tbat tbe second Dote or eacb couple is a sixteenth Dote, however, can only come from contextual Inference and backlbround knowledle.
Chapter 4
VI sua) Expert System
AFter low-level processing has extracted the
primitive Features From a piece of music, this
inFormation is passed to a visual expert system. The
expert system, written in PROLOG, uses knowledge about
musical notation to identiFy each of the Figures in the
score based on the primitives and the spatial
relationships between them. These primitives are the
lines, holes and blobs that are detected by the low-level
system. These Features are made avai1able to the visual
expert through several Files which are loaded by PROLOG.
For each primitive, there is information about its type,
its location and its orientation. The goal of the system
is to determine pitch and duration For each Figure in the
image. A "figure" is a collection of primitives that have
some relationship to each other. Usually, these
primitives are connected. When pitch and duration have
been determined, the information 15 output in a musical
representation language called NOTES (brieFly described
in Appendix D).
The visual expert's problem is to classiFy the
54
55
various Figures in handwritten music in a manner
consistent with the rules of musical notation. This has
been reFerred to as a "consistent labelling problem", and
tree-searching methods have been proposed to solve these
problems [Haralfck and Elliott, 1980]. Music presents a
complex domafn and a very large search space. Standard
search
bl ind
techniques,
and may
even with clever speed-ups~ are still
spend time investigating unlfkely
possibilities. Use of Frames provides a means or limiting
the search space and directing the search to produce a
more Focused problem solution.
ENGRAVE bu i 1 ds a hierarchy or rrames that
corresponds to the structure of musical notation. Lines
contain measures and measures contain notes. Slots of
higher level Frames in the hierarchy may contain the
names of lower level Frames. For example, line Frames
have a list of measures as one of their slots.
Communication among levels of the hierarchy is
achieved through production rules. These rules are
"attached" to the slots or the rrames and to the Frames
themselves. One or more "IF-NEEDED" rules are associated
with each slot and are invoked when a value is needed For
that slot. When an IF-NEEDED rule succeeds, the value
56
that was computed is added to the slot. "TO-FILL" rules
are associated with a frame and contain the slots that
are required to be filled for a frame. TO-FILL rules
automatically fire when a frame is created. Both TO-FILL
and IF-NEEDED rules are triggered "on demand." No values
are computed until they are needed. Once they are
computed, they are stored in slots so that they need not
be computed again. The visual expert is patterned after
CENTAUR [Aikins, 1980], a lung disease expert that uses
frames with attached rules to make diagnoses about lung
diseases based on data from tests.
Each measure frame must satisfy a constraint that
the sum. of the durations of the notes in 'a measure must
add to the number of beats in the measure. If this
constraint is not satisfied, the system will backtrack
and try new rules for various figures. Each time the
constraint Is checked for a measure, an evaluation
function is run. If the sum of the note durations for
that measure is equal to the number of beats in the
measure, then the evaluation function returns zero.
otherwise, the evaluation function returns a value that
indicates the degree that the constraint is violated.
When a set of notes is suggested that does not satisfy
57
the constraint, ENGRAVE wi 1 1 backtrack until the
constraint is satisFied or until all solutions for the
measure have been exhausted. IF no solution is Found that
completely satisFies the constraint, the solution with
the lowest value returned From the evaluation Function
(the solution that is "closest" to being correct) is used
for the measure.
The system provides the capabi 1 ity for the
application of constraints to any frame. These
constraints' are checked when the TO-FILL rule for a frame
has comp 1 eted runn f ng. ·Wh f 1 e ENGRAVE on 1 y conta i ns one
constraint, other constraints may be added to measures
and to other objects as the knowledge base grows.
4.1 ImplementatIon
The Frames and production rules in this system are
implemented in PROLOG using the techniques described in
[Chester, 1981]. Production rules are all of the form:
( (RULE ••• args ••• ) IF (condition-i) (conditfon-2)
(conditfon-m) (act 10n-l)
58
(action-m)
To retrieve values From a Frame slot, we use a Prolog
clause called RETRIEVE. RETRIEVE First to get the value
directly From the slot. IF the slot does not contain a
value, then RETRIEVE calls the IF-NEEDED rules that are
attached to the
the slot or iF
slot. IF there are no rules attached to
they Fail when they are called, then
RETRIEVE will return the deFault value For the slot. It
is expected that the IF-NEEDED rule will add the value it
computes to the slot.
The goal of the system is to Find the Figures in a
piece of music and to label them correctly using their
Features as inFormation. Frames play an important part in
this process because they speciFy the Features needed to
give a Figure a certain label. In addition, the presence
of a Frame suggests plausible combinations of Features
and thereby provides a Focus For program execution. The
program
Features.
begins
IF
associated with
possible- objects
by checking For important distinguishing
one or more of these Features is Found
a Figure, this suggests a class of
For the Figure and causes a frame to be
59
instantiated. Now the system begins to Fill out the other
slots in the Frame. Each Frame is assigned a label based
on the Features that are detected. In cases where there
is ambiguity or where a Frame is not completely Filled
out. there may be several possible labels For a frame.
When there is more than one possible label, these labels
are resolved using global constraints. Figure 29 shows a
Frame For a dotted eighth note.
4.2 An Example
This section shows the processing sequence of
ENGRAVE as it operates on a one-measure image. A dIagram
of the image is shown in Figure 30. The rules used in
this expert system are listed in Appendix A.
AFter loading
started simply by
the TO-FILL rule
the primitives file, the program is
creating a staFF Frame. This triggers
for staves which tries to fill the
"lines" slot and "measures" slot for the staFf. Since
neither slot has a value, the IF-NEEDED rules for these
slots will fire.
The rule that fills the "lines" slot of a staff
(note G45 notehead GI2) (note G45 stem GIS) (note G45 flags (G22) ) (note G45 dots (GI3) )
(note G45 duration 2.666)
(note G45 pitch (C S) )
60
blob for a notehead stem connected to notehead one flag connected to stem one dot fn the vicinity of and following the notehead Durations are represented by the denominator of the note-type, in this case, a 1/8 note has been modified by the dot.
(blob G12 SO 103 23) ; notehead (curve GIS «51 106)(52 106)(52 107)(52 108) .•• (51 122»)
; stem (curve G22 «52 122) ••. (40 130»)
Figure 29 Example of a note frame
;flag
61
E Q)
-&J (/)
> (/)
+J L-Q) Q X Q)
ttJ :J (/)
> 0 (T) Q)
J: Q) +J L-:J L. Oi 0 - "-
I.L.. co
+J (tJ "0
+J :J Q C
"-0
E CO L-0: q)
0
62
looks ror rive starr lines that are adjacent (no starr
lines between them). Starr lines are distfnguished during
low level processing rrom other horizontal lines by the
ract that staFr lines are very much longer. In this
example, there are only Five starf lines in the image, so
"finding them is trivial. In a larger image, ENGRAVE would
start with the Five highest staFr lines in the image.
When we say that a rule "looks "for" a starF's lines,
we mean that it simply perrorms a RETRIEVE operation.
This is how "computation on demand" works. When the value
or a slot fs needed, it is simply computed. I"f the value
has been computed, it is simply returned. IF no value has
been computed, the IF-NEEDED rule ror the slot is
triggered and computes the value. In our example, the
TO-FILL rule attempts to retrieve the value of the lines
slot. Since there is no value For the slot, the IF-NEEDED
rule fires and makes a conclusion about which primitives
are the starf's lines.
The TO-FILL rule for staves also looks for the
measures that are part of the staff. The IF-NEEDED rule
for measures repeatedly creates a measure frame until it
can Find no more measures in the starf. Each time a
63
measure Frame 15 created y the TO-FILL rule For measures
is run. This rule looks For a measure's bar lines and its
notes and then checks the notes against a constraint. In
our example, ENGRAVE labels the primitives 1 and 25 as
bar lines.
The First primitive that is Found is the hole
numbered 13. The hypothesis is made that this primitive
is the notehead of the current note. The "notehead" slot
in the note frame now contains a reFerence to h01e 13.
The system knows there are only a few kinds of musical
Figures that have open noteheads: whole notes y half
notes y sharps and naturals. The First primitive that the
TO-FILL rule looks For is a stem. IF there is a stem,
then we know that the note is a halF note. Otherwise, the
system must go on and try another rule. In this case, the
note does have a stem: the Figure numbered 16. The system
needs only to determfne whether there are any dots
modiFying the duration of the note. None are Found and
the IF-NEEDED rule for dots Fails and ENGRAVE must
include that this note is a halF note with the usual two
beat duration.
The next primitive that ENGRAVE Finds is a blob (3).
64
This is most likely a note with a closed notehead (such
as a quarter note or an eighth note) or a whole or half
rest. The TO-FILL rule. iF it Finds a blob that has not
already
and then
IF-NEEDED
notehead,
been labelled, assigns the blob as the notehead
retrieves the note's duration. This triggers an
rule for note duration. If a note has a closed
ENGRAVE looks for a stem; The notehead numbered
3 does have a stem so we know that the note is not a
rest. At this point, the system has to look for flags and
or dots to determine the duration of the note. The
IF-NEEDED rule For Flags fails because there are no flags
attached to the stem. The rule for dots concludes that
there is a dot (7). Therefore, ENGRAVE concludes that
this note is a dotted quarter note (1-1/2 beats).
The blob numbered 8 is the primitive used for the
next note. The reasoning process proceeds exactJy as
above until ENGRAVE looks for flags. This note has a flag
(15) attached to its stem. From the figure, it may not
seem that 15 and the note's stem are connected but
ENGRAVE has knowledge to account for the fact that some
of the connecting pixels were included as part of the
stafF line. There is a rule that understands that these
two features are connected because they both are
65
connected to a starr line and they are placed in a
relationship that suggests a stem and a rlag.
None or the remaining primitives in the image have
the correct relationships to each other to be labelled as
musical rigures. (One exception is the tie formed by 10
and 11 no knowledge of ties is currently present in
the system.) Each or these rigures receives a NIL label
indicating that no interpretation has been assigned.
At this point the notes are checked against a
constraint. The constraint requires that the sum of the
durations
or beats
time (a
durations
contains
of the notes in a measure must equal the number
in the measure. Since this measure is in 4/4
fact specified by the user), the sum of the note
must add up to four beats. This measure
a dotted quarter note (1-1/2 beats), an eighth
note (1/2 beat). and a half note (2 beats). The sum or
these durations is indeed four beats so the constraint is
satisfied. At this point all or the notes in the current
measure are known and the TO-FILL rule for the measure is
satisried.
there are
The rule for the staff is also satisfied since
no more measures in this image. On a larger
image, the TO-FILL rule ror a staff would create a new
66
measure 'frame and proceed to process the next measure.
The 'final conclusions in this example are summarized
in Table 1. ENGRAVE has built a smal1 'frame structure
showing a line that contains a single measure. The
measure contains three notes. The 'first note is a dotted
quarter note (1-1/2 beats) with a pitch oT "Aft. The
second is an eighth note "E" and the third is a hal'f note
"En. Figure 31 shows a messier measure. ENGRAVE's
conclusions are summarized in Table 2.
4.3 Conclusion
ENGRAVE can understand most conventional notation iT
it is reasonably legible. Even when the handwriting is
somewhat sloppy, as in Figure 31, ENGRAVE can understand
the notation. In Figure 31, ENGRAVE mistakes the natural
berore the second note Tor a sharp. This is due to the
'fact that the key signature is not known to the system.
ENGRAVE can tell when a horizontal line is being obscured
by a star'f line. In Figure 30, the system determines that
the second note fs an eighth note even though the 'flag is
closer to another note. Because ENGRAVE knows that star'f
lines obscure horizontal lines and because the system
67
Table 1 Summary of results from Figure 30
fDlnm .s.un valuers) Staff A Lines 20, 21, 22, 23, 24
Measures B
Measure B Staff A Bar-lines 1, 25 Notes C,D,E
Note C Measure B Dotted quarter note Notehead 3
Stem 2 Dot 7 Duration 1 1/2 beats Pitch A natural
Note D Measure B Eighth note Notehead 8
Stem 9 Flag 15 Duration 1/2 beat Pitch E natural
Note E Measure B Half note Notehead 13
Stem 16 Duration 2 beats Pitch E natural
.----------------------------Unaccounted for: 10, 11
· · · · r :
l"· I · · · · ·
68
~<
~
r ·f I
! J.
I · · I · · ~~~~~~~~~<~~~~~~~~~~<~«~ !
· · · · · · · • I
~-"""~H"""_H""H"""HH~_"HH""HH"~H __ "_hH_M~"_"""_""_~~
('I")
Q) s... :J 0)
La..
"'0 c to
+J rt)
'-4J "'0 C :J
0 +J
4J s... :J rt) (0 Q) E
+J -:J 0
~ ~
"0
Q) s... ~
C(
Table 2 Summary of Results from Figure 31
Er.aJru:. Sl21 Yahu~(&} ~ SJJn Yalue<&} Starr A Lines 1, 2, 3, 4, 5 Note E Measure B
Measures B Eighth note Notehead 14 Accidental note D
Measure B Staff A Stem 16 Bar lines 6. 35 Flag 17+18+20 Notes C,E,F,G Duration 1/2 beat
Pitch D sharp Note C Measure B Quarter rest Upper leg 7+8 Note F Measure B 0\
Lower leg 9 Eighth note Notehead 24 \0
Middle bar none Stem 23 Duration 1 beat Flag 17+18+20
Duration 1/2 beat Note D Measure B Pitch D sharp Sharp Center 11
Left leg 10 Note G Measure B Right leg 12 Half note Notehead 28, 29 Upper bar none Stem 30 Lower bar none Duration 2 beats Modifies note E Pitch D sharp
-----------------------------Unaccounted for: 19. 21, 22, 25, 26
10
knows the kinds of notes that should have Flags, ENGRAVE
is able to correctly determine the duration of both the
second and the third notes. ENGRAVE is also able to use
its knowledge to determine that the two curves at the
beginning of the measure in Figure 31 form a quarter rest
in spite of the Fact that the middle bar is obscured by
the staff line.
In the present implementation, the visual expert
system is separate From the low-level system. Once the
low-level system has run, it produces Files For the
visual expert and control is passed to the expert system.
As mentioned in Chapter 3, knowledge is used in the
low-level system as well as in the the visual expert
system. Ideally, these two levels should be integrated.
In this way, all of the knowledge in the vision system
could be available to every level of the system. In
addition,
backtrack
necessary_
First, a
when a high level rule Fails, the system could
into the lower levels of the system iF
Some work is needed to build such a system.
control structure is needed to handle
communication between the diFferent levels of the system.
Secondly, knowfedge has diFferent meaning at different
levels. A method must be determined For representing
71
knowledge at these different levels and for converting
between them.
Chapter 5
Spatial Planning For "usfcal Output
An important kind of planning involves the
management of objects in space and their interactions and
is called spatial planning. Spatial planning is a
resource allocation problem. It is diFFerent From the
usual robot planning in ArtiFicial Intelligence, however,
because it deals with geometric problems. Typically, A.I.
planners have dealt with dependencies or tasks and with
task sequencing, not with reasoning about geometry.
Spatial planning must also address the interactions of
objects in space and the methods needed to place them
properly.
Creating printed music is a spatial planning problem
characterized by a large solution space. To get an idea
of the potential size of the problem see Figure 3. Figure
3 shows a simple piece of music that has 139 notes and
50 measures. Each note has a wealth of inFormation
associated with ft including location, pitch, duration,
stem direction, and spacing inFormation. All of these
properties must be computed For every note, measure and
staFF so that each object is properly placed on a
72
73
sheet of paper. Ross gives over three hundred rules just
for beaming two eighth notes together [Ross, 1970, p.
104]. Because or the size or the space, some planning
must take place in order to avoid having to redo large
amounts or work. This planning activity must be guided by
knowledge or musical notation and by knowledge or how
musical rigures interact in space.
This chapter reports the investigation or spatial
planning ror printing music. The output stage or ENGRAVE
takes a computer representation of a piece of music and
generates a high qual ity output. ENGRAVE is a
hierarchical, spatial planner that uses a process called
"constraint guidance" to plan the spatial layout and
correct notation or a piece or music.
5.1 Plarvafng ahead
Berore producIng a manuscript, a copyist must plan
ahead to insure that a 11 or the measures fit on a page
and that . they 1 ine up properly on the stafr [Roemer,
1973] • Engravers plan by actually drawing sketches on
their plates to mark the positions or the notes [Ross,
1970]. Notice in Figure 3 that the last measure on each
74
line (other than the last) ends exact1y at the end of the
line. In addition, the notes have diFFerfng amounts of
space between them dependfng on their duration -- longer
notes take more space. To determine the placement of
notes on .a line so that they are properly spaced and
st i 11 Fi 1 1 the line, ENGRAVE plans ahead using a
technique called "constraint guidance"
ENGRAVE's equivalent to the engraver's rough sketch
is a hierarchical plan. The hierarchy represents the
hierarchical structure of music. At the highest level are
pages. To print a page, one must print the staFF lines
,that it contains. Likewise, to print a staFF line, its
measures must be printed. ENGRAVE computes the structure
of the plan and computes only the details that are
needed to make sure that the plan will work. Once this
inFormation is determined, all of the other parameters
needed to print the music can be calculated when the
music is actually printed. See Figure 32 For an example
of a plan the notatfon that it will eventually produce.
5.2 ConstraInt guIdance
Spatial planning can be viewed as a constraint
75
..... . -
<
76
satisraction process. A good indication or this can be
seen by noting the similarity between Gaschnig's [1974]
rormalization of constraint satisraction problems and
Eastman's [1973] characterization or the spatial planning
problem.
ENGRAVE uses constraints to guide its spatial
planning process. A constraint is derined as a relation
between plan variables [Stefik, 1981] or a requirement
that must hold true of the variables. In ENGRAVE, the
constraints are
figures' that, when
the required
satisfied,
relationships between
will result in correct
musical notation. We use the term "constraint guidance"
because the constraints are the goals of the system.
Because the constraints constitute goals, it can be said
that the constraints guide the spatial planning process.
Constraint guidance is a generate and test
procedure. (Sterik suggests that all planning is generate
and test.) ENGRAVE starts with an initial plan and tests
the plan to see whether it satisfies the constraints. If
any constraint is violated, a corrective action is taken
that modifies the plan, and the cycle is repeated untiJ
all constraints are satisfied.
71
Every constraint has a corrective action associated
with it that is taken if the constraint is not satisfied.
Each of these constraint-action pairs forms a
forward-chaining production rule with each having the
form IF (NOT constraint) THEN corrective-action. Using
this format, it is possible to build a knowledge base
about any domain that requires resource allocation.
Later, we will demonstrate this by showing that
constraint guidance can be used to format screens for
human-computer interaction.
5.3 Description
Execution of ENGRAVE proceeds in three stages:
initialization, constraint guidance and rendering. During
the initialization stage, a skeleton plan is built that
the constraint guidance stage uses. Measures are assigned
to staff lines at a rate of ten measures per line. The
constraint guidance stage uses this skeleton plan for the
constraint guidance process. When no more constraints are
violated, the rendering stage computes the values
necessary to dfsplay the music as indicated in the plan.
Constraint guidance and rendering are knowledge based
78
processes driven by rules.
There are really two kinds or rules in ENGRAVE. They
are called rendering rules and constraint guidance rules
arter the stage or the system that uses them. The main
dirrerence is that constraint guidance rules guide the
planning process and rendering rules compute the details
or a particular plan.
Constraint guidance rules are the constraint-
corrective action pairs mentioned above. These rules are
concerned with planning how to layout the piece or music
on a sheet of paper. They deal with the higher levels or
the plan hierarchy mainly starr lines and measures.
The constraint guidance rules are used by ENGRAVE listed
below:
1) Ir there exist two adjacent measures containing a whole measure rest or a multiple-measure rest, then combine them into a single measure containing a multiple measure rest.
2) Ir the last page in the sequence contains more than its maximum number or lines then create a new page following it.
3} Ir there exists a page with more than its maximum number of lines, then move the last line on the page to the next page.
4) If, for the last line in the piece, the sum or the minimum sizes of its measures exceeds the amount of space available for the line then
79
create a new line rollowing it.
5) Ir there exists a line such that the sum or the minimum sizes of its measures exceeds the amount or space available for that line then move the last measure down to the next line.
6) If there exists a 1 i ne, not the last 1 i ne, such that the sum or the sizes or its measures is less than the amount of space available on the 1 i ne then expand the measures uniformly to fill the 1 i ne.
7) If there exists a line such that the sum of the minimum sizes of its measures is less than or equal to the space available but the sum of the actual sizes is greater than the space available then reduce all of the measures uniformly so as to make the sum of their sizes equal to the space ava i 1 ab 1 e.
8) Ir there exists a measure whose size is less than its minimum size then expand the measure to its minimum size.
9) Ir there exists a beat-unit that is not full and there exists a flagged note that is adjacent to one of the notes in the beat-unit but is not a member of any beat-unit itselr then add the note to the beat-unit. (Note: A beat-unit is a group Or notes that are beamed together and treated as a unit. Typically, the notes in beat-unit will sum to one beat. By "flagged note" we mean that the note has duration less than a quarter note and will have a tailor a beam attached to its stem. )
10) If there exists a flagged note that is not a member Or any beat-unit then create a beat-unit with that note as its only element.
11) If there exist two adjacent sub-beat-units that are of the same level and are members or the same beat-unit then combine the two sub-beat-units.
80
These rules are sufficient to insure that. when the
constraint guidance process is finished. the plan places
all or the starr lines and allOT the measures correctly
and that all notes smaller than a quarter note are
grouped properly into beat-units.
Rendering rules are
stage of ENGRAVE. They
mainly used in the rendering
are used to compute properties
such as stem direction and note locations that can be
determined once a plan has been computed. Some oT these
ru1es are used during constraint guidance to compute
properties (such as
planning and which
Rendering rules are
frame slot, when
available.
5.4 An Example
measure size) which are needed Tor
may change as the plan changes.
invoked to compute a property. or
the value is needed but is not
To illustrate how the constraint guidance procedure
works. we will trace through a small example. The piece
of music is shown in Figure 8 as ENGRAVE produces ft.
First. the initialization section produces a
81
skeleton plan: ten measures to a line. Diagrammatically,
the skeleton plan is shown in Figure 33. Because the
initialization section blfndly assigns ten measures to
be necessary. This is each line, some
where constraint
adjustment
guidance comes in. Notice that the sum
or the minimum sizes or the measures in the first line is
210.75 spaces. (A space is the width Or a quarter note.)
Since lines are 160 spaces long on the paper being used,
the First line is much too Full. Ir we tried to print the
line with so many measures it would be completely
unreadable because there is not enough room on the line
for all or the notes in the measures. This is accounted
For by one of the constraints that ENGRAVE uses to plan
the layout or thfs piece or music. When constraint
guidance begins, the first rule that Fires is rule 5
which contains a constraint concerning the sum or the
minimum sizes of the measures in a line. When. the rule
fires. it moves measure 10 down to the beginning or the
next line. The new plan is shown in Figure 34.
BeFore we go on. we need to discuss brieFly the way
that a measure's minimum size is computed. Any two
adjacent figures in a piece of music must be at least
one space apart for the music to be legible.
line:
measure
minimum size:
1 2 3 4 5 6 7 8 9
16.2 25.2 34.65 45.3 31.8 10.2 26.4 10.2 4.2
Figure 33 Skeleton plan for Figure 8.
10 11
6.6 6.6
12 13
16.2 26.4
14
30.6
Q) N
line:
measure
minimum size:
1 2 3 4 5 6 7 8 9
16.2 25.2 34.65 45.3 31.8 10.2 26.4 10.2 4.2
Figure 34 Measure 10 has been moved
to the second line.
10 11
6.6 6.6
12 13
16.2 26.4
14
30.6
CD w
Add i tiona 1 1 y.,
duration oT
following it
84
the distance between notes varies with the
the notes. Each note has an amount oT space
that is a Tunction of its duration.
Consequently, the minimum size is determined from the
duration of the smallest note in the measure. Consider,
Tor example, the measure shown in Figure 35. The smallest
note in this measure is the first (a 32nd note) so the
distance between it and the next note can be no less than
one space. The third note is twice as long as the first
two but it does not, as we might suppose, get twice the
space. Instead, it gets one and one half times the space
oT the shorter note. As it turns out., music engravers do
not use a strict proportionality but tend to "skimp" on
their
this
space in order to fit more notes into a line. Using
"one-and-one-half" method, we can compute the
minimum amount oT space needed For a measure, adding one
space for the distance between the bar line and the Tirst
note.
Going back to the constraint guidance process and
Figure 34, we Find the sum of the min<imum sizes of the
measure in the First line still exceeds the amount of
space available For the line so constraint guidance rule
number 5 fires again moving measure 9 to the next line.
85
Figure 3S Notes with longer durations
take up more space.
Rule 5
moved to
36. Now
86
will continue to rire until measure 5 is rinally
the second line. The plan now looks like Figure
the second line is in violation or the line size
constraint. Since it is currently the last line in the
piece, rule 4 wi1l rire creating a new line with no
measures in ft. Because no measures were moved by rule 4,
the second line is still too rull and rule 5 rires moving
measure 14 down to line 3. Now, the line constraint is
satisried ror all lines and the plan looks like Figure
37.
At this point, we know that all Or the measures will
rit on their lines. However, the measures must also rit
exactly. That is, the bar line or the last measure must
line up exactly with the end or the starr. The constraint
guidance process continues.
Since it is desirable to have all or the measures in
a line be nearly the same size, they are initially
assigned actual sizes such that they are a1l the same
size and the sum or their sizes is the length or the
line. In the case Or line 2, each or the nine measure is
given 17.78 spaces. When we look at Table 3, however, we
see that some or the measures are smaller than their
line:
measure 1 2 3 4 5 6 7 8 9
minimum 16.2 25.2 34.65 45.3 31.8 size:
10.2 26.4 10.2 4.2
Figure 36 All of the measures in the
first line now fit on the line.
10 11
6.6 6.6
12 13
16.2 26.4
14
30.6
CD ....,
line:
measure 1 2 3 4
minimum 16.2 25.2 34.65 45.3 size:
5 6 7 8 9
31.8 10.2 26.4 10.2 4.2
Figure 37 The final configuration tbat ful1y
satisfies the line constraint.
10 11 12 13 14
6.6 6.6 16.2 26.4 30.8
T = 138.6
CD m
Table 3 Initially, an the measures on
a line are of equal size.
measure 5 6 7 8 9 10 11 12 13 minumum size 31.8 10.2 26.4 10.2 4.2 6.6 6.6 16.2 26.4 size 1 7 .78 1 7 .78 1 7 .78 1 7 .78 17 .78 1 7 .78 17 .78 1 7 .78 1 7 .78 T = 160
(]) '-'>
90
minimum sizes. This causes rule 8 to rire and adjust the
size Or measure 5. In Table 4. we now have measure 5
correctly sized but the sum or the measure sizes is
larger that the line size. This causes rule 7 to Fire and
all of the measure sizes are reduced.
To save some computation, only those measures that
are larger than their minimum size are considered
reducible. There are six reducible measures in this
example, so the amount of reduction is (174-160)/6=2.33.
IF any reduction would result in a measure being smaller
than its minimum size, the measure is leFt at that size.
The result or this reduction is shown in Table 5.
Because of measure 62. the sum of the measure sizes
is still too large. Rule 7 fires again. This time there
are Five reducible measures and the amount of reduction
fs (178 - 160)/5 = 3.6. When this reduction is applied to
the measure. we get the result shown in Table 6 and all
of the measures Fit properly on the line. This procedure
is applied to every line in the piece until all measures
are correctly sized. The only exception is when there are
only a few measures on the last line (as in this
example). In this case, no actions are taken and the
measure 5 minimum size 3 1.8 size 31.8
Table 4 Measure 5 expanded to its minimum size.
6 7 8 9 10 11 12 13 10.2 26.4 10.2 4.2 6.6 6.6 16.2 26.4
17.78 17.78 17.78 17.78 17.78 17.78 17.78 17.78
\0
T = 174
measure minimum size size
5 31.8 31.8
Table 5 An measures are reduced by the same
amount but not below their minimum size.
6 10.2
15.45
7 26.4 26.4
8 9 10 11 10.2 4.2 6.6 6.6
15.45 15.45 15.45 15.45
12 16.2 16.2
13 26.4 26.4
\D N
T = 178.05
measure minimum size size
5 31.8 31.8
Table 6 All measures are reduced again.
6 7 8 9 10 11 10.2 26.4 10.2 4.2 6.6 6.6
11.85 26.4 11.85 11.85 11.85 11.85
12 16.2 16.2
13 26.4 26.4 T = 160
\D w
94
measure size set to the minimum size.
5.5 Beet Unfts
Rules 9, 10, and 11 deal with groupings of notes
called "beat-units." A beat-unit is a group of notes
connected by a beam. See, for example, Figure 38. Each
note is an eighth note or a shorter note. The "sub-beat-
unit," a construct that was devised for ENGRAVE,
indicates grouping within a beat-unit. The beams on a
beat-unit indicate the sub-beat-units. In every beat-unit
there is a "major beam" connecting all of the notes in
the beat-unit. If any of the notes are shorter that an
eighth note, there are also "minor beams" to indicate the
duration of each note. This will become clearer as we go
through the example.
When
beat-units
ENGRAVE creates
are indicated in
its skeleton
the plan. A 11
plan, no
notes are
attached directly to their measures. The part of the plan
for Figure 38 is shown diagrammatically in Figure 39.
Each of the notes is a flagged note indicating that
its duration is that of an eighth note or less. Rule 10
95
Figure 38 A beat-unit.
line
notes 1 2 3 4 5
Figure 39 The same beat-unit as it
is represented in the skeleton plan.
6 beat units
\D 0'1
97
will create beat-units or sub-beat-units to change the
plan to look like Figure 40. One level is added to the
plan For each Flag that a note would require iF it were
notated by itselF. Rule 9 will group the beat-units'
together to Form a single beat-unit as shown in Figure
41. Rule 11 combines adjacent sub-beat-units of the same
level. When we combine at sUb-beat-unit levell, we get
Figure 42. All of the notes at level 1 are adjacent, so
they get combined into a single sub-beat-unit. Combining
at level
beaming
will get
another
1, which
2, we get Figure 43. The plan now contains the
structure of our example. The beat-unit itselF
one beam. Underneath the major beam will be
beam, corresponding to the sub-beat-unit at level
also spans the entire beat-unit. Below the
second beam there are two beams connecting notes 1 and 2
and connecting notes 4 and 5. These beams correspond to
the sub-beat-units at level 2.
5.6 RenderIng
Once all of the constraints have been satisFied, the
constraint guidance process is Finished and the rendering
phase begins. During thfs phase, all of the properties
are computed that are needed to print the layout as
measure
beat units
sub lui 1
sub lui 2 \0 en
notes 2 3 4 5 8
Figure 40 Beat·units and sub-beat·units created.
measure
beat units
sUb-beat-unit lui 1
sub lui 2
notes
1 2 3
Figure 41 Beat-units combined.
4 5 6
\D \D
measure
beat units
sub lui 1
lui 2
notes 1 2 3 4 5 6
-o o
Figure 42 Combining level-l sub-beat-units.
101
N
! .~ c::
::::a ::::a CI) - N
CI)
CU CD CD as -E .! :2 :2 0
c::
102
planned during constraint guidance.
Since the nodes in a plan represent objects, it is
natural For them to be represented as Frames. rendering
rules are "iF-needed" rules that are attached to slots
and that Fire when a slot is accessed which has no value.
In this way, all slot values are computed on demand. As a
result, the program needs only to display each object in
the the plan. As each object is accessed all of the
values needed to display the object are computed iF they
have not already been computed.
When a key signature or a time signature must be
displayed, there are rendering rules that determine the
between them. These rules are used proper spacing
whenever a change of key, time, or cleF occurs in the
music and at the beginning of the piece. Figure 44 shows
a diagram of proper spacing between each of these kinds
of figures.
Slot
propertfes
but that
values that are computed by rendering rules are
that are not essentfal to the planning process
are needed to display the figure. Some of these
properties are stem length, pitch, and key signature. A
103
......--o-~
Figure 44 Conventions for spacing.
104
list of the rendering rules can be round in Appendix A.
ENGRAVE is capable or producing most conventional
music notation. Figures 45 through 47 show some or its
capabil ities. An evaluation or other systems that produce
musical notation can be round in Chapter 2. We reel that
ENGRAVE compares ravorably with these systems and that it
has an added advantage because or the flexible nature of
its representation. ENGRAVE does not currently output
multi-line parts or music ror voice as these capabilities
require new rulesets. Appendix E contains the details or
the ENGRAVE's graphics including the methods used to
compute ties, slurs and beam slants.
5. 7 Constra' nt Gu f dance f n a Another Dana in
Constraint guidance can be applied to any domain
that involves the placement or objects in space. To
illustrate its use, we will present a second example of
constraint guidance as applied to the formatting or
graphics screens For human-computer interaction. We wi11
use the afrcraft carrier controller's screen discussed in
[Reilly and Roach, 1984]. Instead of two screens, we will
assume that we have one large screen on which we place
105
106
107
~
~ ~
~ •
, ~ t-
n ~ cG
c rL ·s
«I
!- .8 I- "C c b as
t WJ -~
·c r-- ::s ~.:... as u u ... .0 ::s
• tIOtaO .- C ~ --il
0 ..c
WJ
u Q. .. e "" l< U
I c <
I
~
,
108
all or the objects. These objects are: a plan view or the
aircrart carrier showing the rlight path and the
locations or the planes,
displaying the aircraFt
horizontal perspective or
a vertical perspective Tor
altitudes (the stack), a
the rinal landing phase (the
glide path), the marshall status board containing textual
inFormation about the aircrart, and a message area ror
system messages and user input. We want to place each or
these objects on the screen while rollowing good visual
design technIques so that no inrormation is obscured.
As in music, we start with a skeleton plan. In this
simpliFied version or the domain, two plans will seldom
be very dirrerent. The major blocks will be ordered in
the plan according to their relative importance as
speciried by the user. A three-level hierarchy exists in
this example because each block on the screen is composed
or several parts. Figure 48 shows a typIcal plan.
Initially, the nodes in the plan will not have location
values. The constraint guidance system will compute and
manipulate these location and size.
Constraints ror this example involve the concepts or
109
'i c: ..!. u n... f u
Cf.I
Cf.I .... ~ -0 ... -c 0 u
00 ... u
~ .-... u :a ... u ::I .!!l~ '" as ... u ... . ;
{; ... 0 ~
-¥s -1 ~ / .. '- /
~/
fa -a <
/
110
visual design and the requirement that inFormation must
not be obscured. We will use the following set of rules:
1) IF the placed, screen.
highest priority block then place it in the
has not been center of the
2) If the stack has not been placed, then place it to the leFt or right of the carrier.
3) IF the second highest priority block has not been placed, then place it so the Flight path points to it.
4) IF it
the glide path has not been placed, then place so that it points towards the aircraft
carrier.
5) If the marshall status board has not been placed, then place it in the largest free block of space.
6) If the message area has not been placed, then place it in the largest Free block or space.
1} IF two opposite sides of a block intersect an object or are off the screen, then scale the block down so as to clear at least one side.
8) If a side of a block is off the screen, then move the block away from that edge of the screen.
9) If a side of a block obscures another object, then move the block away from that object.
The blocks in this example are shown separately in
Figure 49. There are five planes in the flight path.
Three of these have their characteristics displayed on
the marshall status board and one is in the final landing
phase. Assume that the blocks are ranked in the following
111
o
o
MSB
Figure 49 The blocks found in the controller's screen.
1 12
order
stack.,
or importance:
gl ide path.,
plan view, marshall status board,
and message area. Rule 1 will place
the plan view in the center or the screen. Rule 2 places
the stack directly to the left of the carrier. The second
highest priority block, the marshall status board, is
placed by rule 3 below and to the right Or the plan view.
Rule 4 places the glide path above and to the right or
the plan view. Finally, the message area is placed in the
free area to the right of the screen by rule 6. The
result of these placements is shown in Figure 50. All of
the block have now been placed so rules 1 through 6 will
no longer trigger. Rules 7, 8 and 9 deal with the blocks
that overlap or
marshall status
the screen. A
go off the screen. Ru 1 e 8 wi I l' move the
board upward so that it is entirely on
plane is obscured however., by this move.
Before this is dealt with, rule 8 will trigger once more
to move the glide path onto the screen. The result is
shown in Figure 51. Now rule 9 triggers to move the
marshall status board to the right and uncover the fifth
plane. The glfde path still overlaps the plan view but no
information is obscured so the screen is lert as it is.
Figure 52 shows the final screen.
have given a simple example to show how
113
o
o
o
MSB
Figure SO Initial placement of objects.
114
o o
o
Figure 51 Resolution of objects off the screen.
!:J.
o
o
o
115
Figure 52 Final screen
116
constraint guidance can be applied to another domain. In
reality, the rules For visual design are not nearly as
well-derined as in music and they are not as rigid [Roach
et al., 1982; Roach et al., 1986]. A realistic system ror
visual design would have to capture this rlexibility and
the subjectivity of the domain. Spatial planning is
central to the realization or such a system.
5.8 Literature Review
Planning is an important part of problem solving.
Work on planning systems refers back to some of the
earliest work in Artificial Intelligence. GPS (Ernst and
Newell, 1969] and STRIPS [Fikes and Nilsson, 1971] are
especially notable. Both
essentially linear planners
level of detail. Multiple
or
and
levels
these systems were
could only plan at one
of abstraction were
provided by hierarchical planners, the first of which was
LAWALY [Siklossy, 1972; Siklossy & Dreussi, 1983]. Other
hierarchical planners are ABSTRIPS [Sacerdoti, 1974],
NOAH [Sacerdoti, 1977], and HOLGEN [Stefik, 1981]. It is
worth noting that HOLGEN used a technique called
"constraint posting" to'handle some or problems caused by
Interacting subgoals.
117
Constraint satisraction is also an important topic
in Artiricial Intelligence literature. In some domains.
it is desirable to derine constraints that limit the
search space or a system. This is especially true or
resource allocation problems [Wilkins, 1984]. ISIS [Fox,
1983], a
constraints
Job shop scheduling system, makes use or to build schedules ror machines and products.
As noted above, MOLGEN used constraints to reduce
sub-goal interaction in planning tasks. DEVISER [Vere,
1983] is a time-based planner that uses constraint
propagation to adjust start and end times ror tasks in a
plan. While TEX [Knuth, 1979] is not an AI system,
Knuth's "glue" is a kind or constraint.
Eastman's [1973] spatial planning system planned the
layout or rurniture in a room. The used a digitized
representation or space and a set or procedural
operators. Grason [1970] used a graph theoretic approach
ror arranging rooms in a building. In both cases, the
objects that are befngarranged had simple shapes and
could be arranged using a straightrorward state space
search. We reel that we have been able to build on this
roundation to allow for spatial planning using complex
118
objects.
The emphasis in this work has been on planning in
spatial domains rather than on musical output. There are
some musical output systems, however, that merit
attention. Perhaps the best known of these systems is
SMUT [Byrd, 1974]. This system can handle most
conventional music notation and can print multiple
voices. Other systems are mentioned in Chapter 2. All of
these systems use algorithmic methods to format their
output. We feel that much flexibility and power can be
gained by using heuristic methods from Artificial
Intelligence instead of algorithmic methods.
5.9 Conclusions
We have described a system that produces quality
musical output. The main thrust of this chapter however,
was not to describe how to produce correct musical output
but to describe constraint guidance by showing how it
operates in a musical domain. We have endeavored to do
this by discussing the principles behind constraint
guidance and by stepping through some examples.
119
There are some lfmitatfons to constraint guidance:
its dependence on an initial skeleton plan and the lack
of ability to relax constraints. Constraint guidance as
we have implemented it relies on an initial skeleton plan
to begin its operation .. This makes ENGRAVE a
generate-and-test system in a sense. One of the lessons
of DENDRAL [Lindsay et al., 1980, p. 148] is that a good
generate-and-test system must have a good generator. It
is important therefore, that a skeleton plan be a good
"first cut." This will reduce the amount of extra work
that the system must do. We could have had ENGRAVE
initially assign all of its measure to one line. Had we
done thfs, ENGRAVE would have spent a lot of time
creating new lines and moving individual measures to the
next line. Thfs could be a very serious performance
problem for a large piece of music. A system that uses
constraint guidance ought to have a good set of
heuristics for generating a skeleton plan.
It is
over-constrained
possfble that, fn some
situations could occur.
domains,
In our
implementation this would cause ENGRAVE to run forever.
In an over-constrained problem, there will always be a
violated constraint. It is desirable to be able to detect
120
over-constrained situations and intelligently relax
constraints to arrive at a best solution [Fox, 1983].
Barring this, the rule set must be designed such that
there is always a so1ution that satisfies all of the
constraints.
We believe that constraint guidance is applicable to
three dimensional domains. The chief requirement is that
objects can be decomposed into sub-parts. This allows Tor
the multiple levels of detail in a plan. It is not
required that the decomposition conform rigidly to a tree
structure but there must be a distinction between levels
of detail. A good example is a factory floor layout. Many
objects on a factory floor have parts that can be
configured in different ways. For example, a robot
station may have a controller that must be located near
the robot and a parts bin that must be within reach of
the robot arm. Additionally, machines may be combined
into larger entities such as an assembly line using
constraints about their location on the line. We believe
that this kind of spatial planning problem is a suitable
application for constraint guidance.
In summary, ENGRAVE is a spatial planning system
121
that uses a technique called constraint guidance to plan
the spatial layout of music. This technique is useful not
only ror musical layout, but for other spatial domains as
well. We believe that constraint guidance extends
existing spatial planning techniques to provide more
utility and power.
6.1 VIsIon
Chapter 6
Dfscussfon
The rirst section or ENGRAVE is a vision system that
can process handwritten musical notation. In the first
attempt to
general-purpose
reatures. These
expert system
these 'features.
it depended on
but 1 d the system, we tried using
image processing
reatures were
techniques
to be the
to detect
input to an
that used musical knowledge to understand
The expe'rt system worked qu i te we 1 1 but
an unreliable low-level system 'for its
input. Much time was spent changing parameter va1ues 'for
the low-level operators to get better results. However,
a set or parameters can only be tuned to work for a small
set of images and consistent results cannot be guaranteed
in general'. The solution to this prob1em was to use
domain knowledge at the low level. When this was tried,
there was
suggests an
be employed
successFu 1 •
a remarkable improvement in the results. This
important conclusion: domain knowledge must
at all levels ror a vision system to be
122
There are
consideration.
123
. several issues
The system as
that merit Further
it stands is still
separated into a low-level section and an expert system.
feel that these two leve"ls need to be integrated into a
single system. This would enable Full control over
knowledge at all levels. In this way, failure or a
high-level rule could result in backtracking into
low-level rules causing a reconsideration of pixel
relationships. When we build such an integrated system,
we will rind that several levels of knowledge exist.
Knowledge about image-processing, low-level
relationships, musical notation and control exist
simultaneously in the same system. This has been
discussed in [Clancey, 1982] and a Framework is proposed
to represent diFferent levels of knowledge and their
relationships. The use of Frames [Minsky, 1975] is
another method that is used to explicitly handle
diFferent levels of knowledge.
Another issue is the Flow of information between
levels. While ENGRAVE was being developed, it was found
that information From low-level processes was useFul at
the higher levels even though no conclusions could be
made earlier using the same inFormation. Low-1evel
124
inrormation, however, arter has a dirrerent meaning at
higher levels. The problem that needs to be addressed is
how inrormation is passed between levels and what forms
it should have to be most errectively used at any level.
6.2 GraphIcs
ENGRAVE uses spatial planning to produce its output.
This methodology is appropriate ror domains that involve
placement or objects in space. This work and previous
work have only dealt with two dimensional domains. An
interesting and useful erfort would be to extend these
ideas to three dimensions.
One of the worst shortcomings or the spatial planner
is speed. The system is slow partly because it is
implemented in an fnterpreted language and because it
must repeatedly search a large database. (There are
approximately 250 nodes in the plan for the full piece
shown in Chapter 4.) One or the 1essons of DENDRAL is
that the erriciency or the generator is important in
generate-and-test systems [Lindsay et al, 1980]. In our
case, this points to the corrective actions or the
production ru1es. What is needed for improvement is
corrective actions
Ideally,
125
that correct a situation more
when an unsatisFied constraint is completely.
detected, the
the constraint.
corresponding action should fully satisfy
Whether this is possible is an open
question, but there certainly is room for improvement.
6.3 Conclusion
The development of ENGRAVE provided a good vehicle
to study some interesting questions in artificial
intelligence, particularly expert systems. The use of
world knowledge for low-level vision computing and the
use of spatial planning for graphical output are the two
most important concepts that came out of this study.
While ENGRAVE, as it stands now, is not ready for
commercial use, it would require only a Few additions to
reach that point. Development of a music editor is the
largest effort that would be required. More effort is
also needed to make the system execute faster. However,
with the state of affairs in the music publishing
industry, a batch job that ran overnight still wou1d be
an economic improvement over current practices.
REFERENCES
J. S. Aikins, "Prototypes and Production Rules: A Knowledge Representation For Computer Consultations", Ph.D. dissertation, Deptartment of Computer Science, Stanford Univ. 1980.
C. Arcel11 and G. Samitl Di Baja, "A Thinning Algorithm Based on Prominence Detection," Pattern Recognition 13, no. 3, pp.225-235, 1981.
D. H. Ballard and C. M. Brown, Computer Vision, Englewood ClifFs, New Jersey: Prentice-Hall, 1982.
D. Byrd, "A System ror Music Printing by Computer," Computers and the Humanities, vol. 8, pp. 161-172, 1974.
D. Byrd, "An Integrated Computer Music Sortware System," Computer Music Journal 1, no. 2, pP. 55-60, 1977.
D. Chester, "Implementing Production Systems In Prolog," Internal Report, Deptartment or Computer and Inrormation Sciences, University Or Delaware, 1981.
W. J. Clancey, "The Epistemology of a Rule-Based Expert System: A Framework ror Explanation," Artiricial Intelligence, vol. 20, no. 3, pp. 215-251, 1983.
R. o. Duda and P. E. Hart, "Use or the Hough Transrorm to Detect Lines and Curves in Pictures," Communications of the AC M ., no. 1 5 , pp. 1 1 - 1 5 , 1 972 •
C. M. Eastman., "Automated Space Planning," ArtiFicial Intelligence 4, pp.41-64, 1973.
R. W. Ehrich, "A User's Guide to the EXTENDED CORE Graphics System ror the VAX 11/780," Internal Report, Department or Computer Science, VPI & SU, 1983 •.
G. W. Ernst and A. Newell, GPS: A Case Study in Generality and Problem Solving, New York: Academic Press, 1969.
I. D. Faux and M. J. Pratt, Computational Geometry For Design and ManuFacture, Chfchester,.West Sussex, England: Horwood, 1979.
126
127
R. E. Fikes and N. J. Nilsson. "STRIPS: A New Approach to the Application of Theorem Proving to Problem Solving," ArtiFicial Intelligence 2, pp.189-208, 1971.
M. S. Fox, "Constraint-Directed Search: A Case Study of Job-Shop Schedu 1 I ng, 'f Ph. D. d i ssertat i on , C.omputer ~cfence Department, Carnegie-Mellon University, 1983.
J. Gashnfg, ttA Constraint SatisFaction Method For InFerence Making," Proceedings of the 12th Annual Allerton ConFerence on Circuit and System Theory, University of Illinois, Urbana-Champaign, pp.866-874, 1974.
D. A. Gomberg., "A Computer-Oriented System Printing," Computers and the Humanities, pp.63-80, 1977.
For Music vo 1 • 1 1 ,
J. Grason, "A Dual Linear Graph Representation For Space-Filling LocatIon Problems of the Floor-Plan Type.," in Emerging Methods in Environmental Design and Planning, G. Moore (ed.), Cambridge, Mass.: The MIT Press, 1970.
R. M. Haralick, "A Measure For Circularity of Digital Figures," IEEE Transactions on Systems, Man and Cybernetics., vol SMC-4, pp. 394-396., 1974.
R. M. Haralick and G. L. Elliott, "Increasing Tree Search EFFiciency For Constraint SatisFaction Problems," ArtiFicial Intelligence 14, pp.263-313, 1980.
R. M. Haralick, S. Krusemark Introduction," Internal Report, Laboratory, VPI & SU, 1981.
and K. NeikIrk, "GIPSY: Spatial Data Analysis
L. A. Hiller, Jr. Printing," Journal
and R. of
A. Baker, "Automated Music Music Theory 9, no.l,
pp. 129-162, 1 965.
D. R. HoFstadter, Godel, Escher, Bach: An Eternal Golden Braid, New York: Basic Books, 1979.
B. W. Kernighan and P. J. Plauger, SoFtware Tools, Reading, Mass.: Addison-Wesley, 1976.
D. E. Knuth, TEX and METAFONT: New Directions in Typesetting, BedFord, Mass: Digital Press., 1979.
127
128
S. Krusemark and G. Elliott, Documentation," Internal Report, Laboratory, VPI & SU, 1981.
"RATFOR Spatial
Preprocessor Data Analysis
R. K. Lindsay, B. G. Buchanan, E. A. Fiegenbaum, J. Lederberg, Applications or Artiricial Intelligence ror Organic Chemistry: The Dendra) Project, New York: McGraw-Hill, 1980.
J • T • Maxwe 1 1 I I I and S. M. Ornstein, "Mockingbird: A Byte, Vol 9, no. 1, pp. 384-410, Composer's Ameneunsfs,"
1984.
L. G. Minor Segmentation or Transactions on 1 94 - 20 1, 1 98 1 •
and J. Sklansky, "The Detection Blobs in Inrrared Images," Systems, Man and Cybernetics,
and IEEE pp.
M. Minsky, "A Framework ror Representing Knowledge," in The Psychology or Computer Vision, ed. Winston, New York: McGraw-Hill, 1975.
J. A. Moorer, "On the Transcription or Musical Sound by Computer," Computer Music Journal 1:4, pP. 32-38, 1911.
A. M. Nazir Segmentation: An Pattern Analysis 555-577, 1984.
M. Piszczalski Transcriptions," 1977.
, and M. D. Levine, "Low Level Image
Expert System," IEEE Transactions on and Machine Intelligence, no. 6, pp.
and B. Computer
A. Galler, "Automatic Music Music Journal 1:4, Pp. 24-31,
D. S. Prerau, "Computer Pattern Recognition of Printed Mus i c , " Proceed f ngs of the Fa 1; 1 Jo i nt Computer ConFerence, AFIPS ConFerence Proceedings, 1971.
D. S. Prerau, "Do-Re-Mi: A Program that Recognizes Music Notation," Computers and the Humanities, vol. 9, pp. 25-29, 1975.
S. S. Reilly and J. W. Roach, "Improved Visual Design ror Graphics Display," IEEE Computer Graphics and Applications, vol. 4, No.2, pp. 42-51, 1984.
J. Roach, J. A. Pittman, S. Reilly and J. Savarese, "A Visual Design ConSUltant," International Conrerence on
129
Cybernetics and Society. Seattle. Wash •• 1982.
J. W. Roach and G. S. Fowler. "The HC Manual: Virginia Tech Prolog." Internal Report. Deptartment of' Computer Science. VPI & SU. 1983.
J. W. Roach. S. S. Re i 11 y and J. E. Tatem, "I nte 11 i gen't Tools: An Automated Advisor f'or Visual Design," in HumanComputer Dialog Design, eds. Ehrich and Wil leges, 1986, pp. 146 - 163.
C. Roads, "A Note on Music Printing by Computer," Computer Music Journal 5, no. 3. pp. 57-59, 1981.
C. Roemer, The Art of' Music Copying. Sherman Oaks, Calif'.: Roerick Music Co., 1973.
A. Rosenf'eld and E. Johnston, "Angle Detection on Digital Curves," IEEE Transactions on Computers, pp. 875-878, 1973.
T. Ross, The Art of' Music Engraving and Processing, Miami: Hansen Books, 1970.
E. A. Sacerdoti, "Planning in a Hierarchy of' Abstraction Spaces," Art if' i cia J I nte 1 1 1 gence 5, 1974, pp. 1 15 - 135.
E. A. Sacerdoti, A Structure f'or Plans and Behavior, New York: Elsevier Publishing, 1977.
SIGGRAPH, "Status Report Of the Graphics Standards Committee," Computer Graphics (13), August 1979.
L. Siklossy, "Modelled Exploration by Robot," Technical Report 1, Computer Sciences Dept., University of' Texas, Aust in, 1972.
L. Siklossy and J.Dreussi, "An EfFicient Robot Planner Which Generates Its Own Procedures." Proceedings of' the Third International Joint Conf'erence of Artif'fcia1 Intelligence, Stanf'ord, Ca1if., pp. 423-430, 1973 reprfnted in Tutorial on Robotics, eds. C. S. Lee, R. C. Gonzalez and K. S. Fu, 1983.
L. Smith, "Editing and Printing Music by Computer," Journal of Music Theory 17, no. 2, pp. 293-309, 1973.
M. Stef'ik, "Planning with Constraints (MOLGEN: Part 1),"
130
ArtiFicial Intelligence 16, no. 2, pp. 1-39, 1981.
K. Stone, Music Notation in the Twentieth Century, New York: Norton, 1980.
s. Tsuji and F. Matsumoto, "Detection of Ellipses by a ModiFied Hough TransFormation," IEEE Transactions on Computers, vol c-27, pp. 777-781, 1978.
s. A. Vere, "Planning In Time: Windows and Durations For Activities and Goals," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 5, pp. 246-267, 1983.
L. T. Watson, K. I. Arvind, R. W. Ehrich, and R. M. Haralick, "Extraction of Lines and Regions From Grey Tone Line Drawing Images," Pattern RecognItion 17, No.5, pp. 493-507, 1984.
D. E. Wilkins, "Domain-Independent Planning: Representation and Plan Generation," ArtiFicia1 Intelligence 22, #3, 1984, pp. 269-301.
Appendix A
V I .ue 1 Expert Au Ie.
This appendix and Appendix B contain listings or the
rules used by ENGRAVE. The most important product or an
expert system is the knowledge that has been codiried.
Thererore, these rules are listed in English so that
someone else may make use of the knowledge.
A. I Ru 1 es about note durat fOIls.
Note durations are expressed as the reciprocal or
the mathematical value or the note's duration. Hence the
duration or a quarter note would be expressed as 4. The
duration or a dotted quarter note (one and one-halr
beats) would be 2.6667 (or 8/3).
Ir a note has a closed notehead, then its duration
is computed using the the formula 2(F+D+2)/3D• Where
F is the number or rlags and D is the number or dots
associated with the note.
Ir a note has an open notehead and there ,are two
curves that intersect the notehead and one curve is to
131
132
the leFt or the notehead and the other curve is to the
right or the notehead and one or the curves rises higher
than the other then the note is a natural and its
duration is NIL.
If a note has an open notehead, its duration is
computed by the rormula 2oT/3o where T is the type or
note (2 ror a halF note, 1 For a whole note) and 0 is the
number of dots that modify the note.
If a note has no notehead and there are two curves
such that both intersect the middle line of the stafF and
one runs from the middle line highe~ and one runs rrom
the middle line lower and both nearly vertical then the
note is a quarter rest and has pitch R (for rests).
All other notes have duration NIL
A.2 Rules about measures
To compute
create a note
Then add the
measure.
the notes of a measure, recursively
frame until the create operation fails.
list of notes to the NOTES slot of the
133
Ir a measure has a previous measure, then the rirst
bar-line is the second bar-line or the previous measure.
IF a measure has no previous measure, then its First
barline is the lertmost curve that is nearly vertical and
fntersects all or the starr lines.
To compute the second bar-line or a measure, rind a
curve that is nearly vertical and intersects all or the
starr lines and has no similar curve between it and the
rirst barline.
IF a measure is in the top line on the page, then
the top boundary or the area covered by the measure is
the top or the image.
IF a measure has another starr line above then its
top boundary is halr the distance between the top line of
its starr and the bottom line or the starr above it.
Ir a measure is in the bottom line or the page, the
measure's bottom boundary is the bottom Or the image.
134
Ir a measure has another starr line below it then
its bottom boundary is halr the distance between bottom
line or its starr and the top line or the starr below it.
The lert boundary or a measure is the co1umn
coordinate or the measure's rirst bar line.
The right boundary of a measure is the column
coordinate or the measure's second bar line.
A.3 Ru I as about notes
A.3.1 TO-FILL rules 'For notes
Ir there is an unused hole in the image. then treat
it as a notehead and retrieve the note's duration and
pitch.
Ir there is an unused blob in the image. then treat
it as a notehead and retrieve the note's duration and
pitch.
Ir there is an unused curve in the image. then
assign it to a note and retrieve the note's duration and
135
pitch.
A.3.2 IF-NEEDED ru1es For notes.
Ir a note has an open or a closed notehead and there
is a set or connected near1y vertical curves one or which
intersects the notehead, then conclude that these curves
are the note's stem.
Ir a note has a closed notehead and the note has a
stem and there is a curve that intersects the stem on the
opposite end rrom the notehead, then add the curve to the
list or rlags ror the note.
Ir a note has a closed notehead and the note has a
stem and there is curve that. intersects a starr at one
end and the note's stem fntersects that same starr line
and there are no curves between the stem and the curve,
then add the curve to the note's list of Flags.
IF a note has a closed notehead and the note has a
downward pointing stem and there is a curve that is below
and to the right or the note's stem and there are no
curves between the stem and the curve being considered,
136
then add the curve to the note's list or rlags. (This
allows ror stems that are drawn too short. One end or the
rlag should be almost directly below the stem.)
Ir a note has a closed notehead and the note has an
upward pointing stem and there is a curve that is above
and to the lert or the note's stem and there are no
curves between the stem and the curve being considered,
then add the curve to the note's list or rlags.
A note's starr is the starr that contains the
measure that contains the note. (This one is a litt1e
obvious but it is a necessary piece or knowledge.)
Ir a note has a stem and the end or the stem ,
opposite its intersection with the notehead, then the
note has a downward pointing stem.
Ir a note has a stem and the end or the stem
opposite its intersection with the notehead, then the
note has a downward pointing stem.
Ir a note has a stem which does not intersect the
notehead and the stem is above the notehead, then the
137
note has an upward pointing stem.
IF a note has stem which does not intersect the
notehead and the stem is below the notehead, then the
note has a downward pointing stem.
Note: In this implementation, a note may have only one
dot or none.
IF a note has a downward pointing stem and there is
a dot in the measure which is to the right and above the
notehead, then add this dot to the list of dots For this
note.
IF
dot in
notehead,
note.
IF
a note has an upward pOinting stem and there is a
the measure which is to the right and below the
then add this dot to the Jist of dots For this
there is a dot in the same measure as a note. and
the dot is directly to the right of the notehead and
there are no other noteheads between the dot and the
notehead, then add this dot to the list of dots For this
note.
138
A.4 Rul.s about stav.s.
To rill a stave rrame rind its lines and rind its
measures.
The lines or a starr consist or rive "starr-lines"
that are equal dfstances apart and that have no other
starr lines between them.
The lert end or a starr is the average or the lert
end coordinates or all the starf lines.
The right end of a staff is the average of the right
end coordinates of all the starr lines.
To rind the measures or a starf, create a measure
rrame repeatedly until the TO-FILL rule rails.
Appendix B
SpatIal Planner Rule.
B.l Constraint GuIdance Ru1es
1) Ir there exist two measures such that: - they are adjacent; and - they both contain only a whole rest;
then combine them into a single multiplemeasure rest.
2) IF there exists a line such that: - it is the last line; and - the sum or the minimum sizes of the
measures is greater than the line size; then create a new line arter this one.
3) Ir there exists a line such that: - it is not the last line; and - the sum of the minimum sizes or the
measures is greater than the line size; then move the last measure or the line down to the next line.
4) Ir there exists a line such that: - it is not the last line; and - the sum or the minimum sizes or the
measures is less than the line size; then expand all the measures in the line by (L-S)/n where L is the 1 ine size, 5 in the sum or the measures mlnlmum sizes, and n is the number or measures in the line.
5) IF there exists a line such that: - the sum or the measure sizes is greater
than the line size; and - the sum or the minimum sizes is 1ess than
or equal to the line size; then Find the least complex measure in the line with (size > min) and reduce it by min(sum_sizes - linesfze, size - min).
6) IF there exists a measure such that its size is less than its minimum size then expand the measure by (min - size).
139
140
1) IF there exists a beat-unit such that it is not Full and there exists a note such that:
- it is a Flagged note; and it is adjacent to one of the notes in the beat-unit; and
- it is not in another measure or beat-untt; then add that note to the beat-unit.
9) IF there exists a note such that: - it is Flagged; and - it is not already in a beat-unit;
then create a beat-unit with that note as its only element.
10) IF there exist two sub-beat-units such that: - they are the same level; and - they are adjacent;
then they can be combined.
B.2 Rendering Rules
IF the pitch language), the ( I mp 1 i ed tie)
IF the NOTES is not a rest,
IF the NOTES and the note notehead.
of a pitch
duration then the
note is not deFined (in the NOTES is the same as the note beFore it.
value For a note is 1 and the note note is a whole note.
duration value For a note starts with a 2 is not a rest, then the note has an open
IF the NOTES duration value For a note starts with 4 or greater and the note is not a rest, then the note has a closed notehead (as in quarter notes).
IF a note is a rest and has a duration value of 1, then the note is a whole rest.
IF a note is a rest and has a duration value that starts with 2, then the note is a halF rest.
IF a note is a rest and has a duration value of 4, then the note is a quarter rest.
141
IF a note is greater., then sma 11 er) •
a rest the note
and has a duration va1ue of 8 or is a eighth note rest (or
The vertical coordinate For rests is the same as the center line of the staFF.
Notes in the treble cleF are oFfset vertically from the center of the staFf by
(3.5 * a * S) - 17 spaces
where a is the octave number For the note and 5 is the oFfset From C for the letter name of the note.
Notes in the bass cleF are offset vertically from the center of the staff by
The 2.5.
The
The
The
The
The
The
(3.5 * a * 5) - 11 spaces
where a is the octave number for the note and 5 is the ofFset from C For the letter name of the note.
sequence number (distance in spaces from C) for A is
sequence number For B is 3.
sequence number for C is o. sequence number for 0 is 0.5.
sequence number for E is 1 •
sequence number for F is 1 .5.
sequence number for G is 2.
Full measure rests (whole rests) are placed in the center or the measure.
The First note in a measure always starts one unftsize from the start of the measure.
IF a measure contains a front repeat (leFt-handed)., then the first note in a measure is placed 3 spaces away from the bar 1 fne.
142
IF a measure is the First in its line and the line contains a time signature at its beginning, then the rirst note in the measure is placed 2-1/4 spaces aFter the time signature.
IF a measure is the rirst in a line, and the line contains a key signature at its beginning, then the First note in the measure is placed 2-1/2 spaces past the last accidental in the key signature.
IF a measure is the First in its line, and the line contains no key or time signatures at its beginning, the First note in the measure is placed 1-1/2 spaces From the beginning of the line.
IF a measure note in the signature.
has a time signature change, then the First measure is placed 2-1/4 spaces From the time
IF a measure has a key signature change, then the First note in the measure is placed 2-1/2 spaces From the last accidental in the key signature.
IF a measure has no time or signature changes, then the First note in the measure is placed 1-1/2 spaces From the First bar line of the measure.
IF a note is not the First note in horizontal coordinate depends on measure location or the last note as Follows:
where:
p * U * (5 - E) + L (M - E)
its measure, size and
P is the space allocated to the last note; and U is the "unltsize" of the measure; and 5 is the size of the measure; and
i'ts the
- E is the amount of s~ace needed In the measure For signature changes and repeats; and
- M is the minimum size of the measure.
IF a note has an accidental sign, the note should be moved 1 space to the right and the sign should be placed 1 space beFore the note.
143
The "space" property or a note (the amount or space the note gets within a measure) is computed by:
where:
5 .. T V
5 is the spacing value or the note from the measure's prorile; and T is the type or the note (4 ror quarter notes); and V is the reciprocal or of the notes duration in beats.
Note that this space is computed as ir the measure were at minimum size.
The stem direction ror notes in a beat unit is the same as the stem direction of the note in the beat unit that is rarthest rrom the center line of the stafr.
IF a note is part or a beat-unit, then its stem direction is the same as that or the beat-unit.
If a note's horizontal location is below the center line of the staFF, then its stem direction is "up".
Ir a note's horizontal coordinate is above or on the center line of the staFr, then its stem direction is "down".
Ir a note is part of a beat-unit, then its stem is long enough to intersect with the major beam of the beat-unit.
Ir a note is more than one ledger line outside of the staFr, then the stem is long enough to intersect the middle line of the starF.
IF a note has more than two flags, then the stem length is computed (in spaces) by:
F + 1.5 where F is the number or Flags.
In general, stems are one octave (3-1/2 spaces) long.
IF a note is not a whole note, and not a rest, then the note has a stem.
144
If a note is dotted and is on a line. then the first dot is 1/2 space above and 1-1/2 spaces to the right of the notehead.
If a note is dotted and is on a space, then the rirst dot has the same vertical coordinate and is 1-1/2 spaces to the right of the notehead.
Dots ror the same note are placed one space apart horizontally and all have the same vertical coordinate.
If a note is above or below the staff., then the number of ledger 1 i nes is computed by:
0-2
where 0 is the absolute value or the orrset or the note rrom the center 1 i ne or the sta"Fr in spaces.
The slope or a beat-unit is the linear regression slope or the line rormed by the noteheads in the beat-unit.
For the primary beam. the note closest to the middle line of the staff determines the location of the beam.
For sub-beat-units., the point on the beam is one line lower/higher than the primary beam for each level past 2.
Ir a note is the only element or a sub-beat-unit and is the last note or its beat-unit. then it has a "Fractional beam that points to the left.
If a note is the only element or a sub-beat-unit and is the first note of its beat-unit, then it has a fractional beam that points to the right.
Ir a note is the only element of a sub-beat-unit and is in an odd position in its beat-unit., then it has a fractional beam that pOints to the left.
If a note is the only element of a sub-beat-unit and is in an even position in its beat-unit. then it has a fractional beam that points to the right.
If a beat-unit or sub-beat-unit does not have a fractional beam., then the X-coordinates of the endpoints or its beam are the same as the X-coordinates or the stem ends of the first and last note in the unit.
145
IF a beat-unit has more than one note, then it has a beam.
IF a note has Flags, then the First Flag is located at the end of the stem and subsequent Flags are each located one space closer to the notehead.
IF a note has an accidental, it is located 1-1/4 spaces to the lert of the notehead and has the same V-coordinate.
The key signature of a line is the same as the key signature of its first measure.
If a line is the first in the piece, then print its key signature when the line is printed.
IF a line has a diFFerent key than the previous line, then we print its key signature.
No key signature is printed for lines other than the first unless the key has changed since the last line.
No key signature is printed iF the line is in the key of C.
The cler of a line is the same as the clef of its first measure.
The time signature of a line is the same as the time signature of its first measure.
IF a line is the first on its page, then its X position is
120 120 -------
n + 1
in spaces where n is the number or lines on the page.
A line's column coordinate is computed by
120 L -------
n + 1
in spaces where L is the location of the previous location and n is the.number of lines on the page.
IF a measure is the last in its line, the line is not the
146
last in the piece. and the next measure does not have a key change, then the measure's bar line is placed at the end or the line. (Note: the bar line that is associated with a measure is the one at the end or the measure.)
Ir a measure contains a rest ror its entire duration. and it is not the rirst measure or the line, then the location or the bar line is computed rrom the location of the previous bar line added to the size or the measure.
A bar line's horizontal coordinate depends on measure size and the location of the 1ast note in the measure as follows:
where:
p * u * (S - E) + L (M - E)
P is the space a1 located to the last note; and U is the "unitsize" or the measure; and 5 is the size or the measure: and E is the amount of space needed in the measure for signature changes and repeats; and M is the minimum size or the measure.
The vertical coordinate of a bar line is the same as the center line or the staff.
If a measure is the first in its line and has a starting repeat, then its starting point is the same as the horizontal coordinate or the repeat sign.
Ir a measure point is 3/4 signature.
is the first in its line, then its starting spaces past the last sign in the lines time
Otherwise. a measure's starting point is the same as position of the second bar line of the previous measure.
In a given measure. ir there exists a note such that: it is not the first note in the measure; and
- it has an accidental; and - its predecessor is of Jess than or equa1 duration; then the measure's unitsize is three times the size· of a horizonta1 space
otherwise it is twice the size of a horizontal space.
147
IF a measure is in the 1ast line on a page and the sum of the minimum sizes of the measures on the line takes up less than 1/3 of the space on the line. then set the size of the measure equal to the minimum size of the largest measure on the 1 ine.
The size of a measure is computed by the space available For a line divided by the number of measures in the line.
IF a measure is the First or last measure in the piece (These measures are oFten incomplete), then the size of the measure is computed by
where
o *. 5 * L
Num it N
o is the denominator of the time signature; and Num is the numerator of the time signature; and 5 is the sum of the durations of the notes in the measure (this is equal to the time signature For complete measures); and L is the amount of space in the 1ine; and N is the number of measures in the line.
IF a measure is the First in the piece, then its time signature is the same as the time signature of the piece.
IF a measure has a time signature change, then its time signature is the same as the new time signature.
IF a measure is not the First in the piece and does not have a time signature change in it, then its time signature is the same as that of the previous measure.
IF a measure is the First in the piece, then its key signature is the same as the key signature of the piece.
IF a measure has a key signature change, then its key Signature is the same as the new key signature.
IF a measure is not the First in the piece and does not have a key signature change in it, then its key signature Is the same as that of the previous measure.
IF a measure is the First in the piece, then its cleF is
148
the same as the clef of the piece.
If a measure has a clef change, then its clef is the same as the new clef.
If a measure is not the first in the piece and does not have a clef change in it, then its clef is the same as that of the previous measure.
If a clef is the clef at the beginning of a line, then it is located 2-1/2 spaces past the beginning of the line. Its vertical coordinate is the same as the center line of the staff.
If a clef is part of a measure (because of a cleF change), then it is located 2-1/2 spaces to the left of the previous bar 1 ine ..
If a key signature is part of a measure (a key change) then the horizontal position of its first sign is 1-1/2 spaces past the bar line.
IF a key signature is part of a line, then the horizontal location of the first sign is 2 spaces past the c1ef sign.
If a time signature is the in the same measure as a key signature, 'then the time Signature is positioned 2-1/4 spaces past the last sign of the key signature.
If a time signature is part of a line, then its horizontal is 3-1/4 spaces past the clef sign.
If a time signature is part of a measure and there is no key signature in the measure, then its horizontal position is 1-3/4 spaces past the bar line.
Note: In the following, "front repeat" re'fers to the starting repeat figure in a repeated section of music and "end repeat" reFers to the terminating repeat Figure.
If a 'front repeat is in the first measure of a line, then its horizontal position is 2-3/4 spaces past the time Signature.
If a 'front repeat is in the same signature change, then its horizontal spaces past the time signature.
measure as a time position is 2-3/4
149
IF a front repeat is in the same measure as a key change, but the measure has no time change, then the horizontal position of the repeat is 2-3/4 spaces past the last sign in the key signature.
If a front repeat is in a measure that is the first in its line, and there are no other figures at the start of 'the line (cleFs, etc.), then the horizontal location of the repeat is the same as the start of the staff line.
If a Front repeat is in measure that is not at the start of a line and the measure has no other key, time, or clef changes, then the horizontal position of the repeat is the same as that of the previous bar line.
If an change, sign.
end repeat occurs in a measure preceding a clef then locate the repeat 2-1/2 spaces before the clef
IF an end repeat occurs in a measure with no clef change following, then the horizonta1 location of the repeat is the same as the location of the measure's bar line.
If a line does not contain a repeat in the first measure, then the amount of space on the line for notes is lessened From the normal length of staff line by 6-1/2 spaces plus one space for each sign in the key signature.
I f ali ne has a new key signature, a new time signature', and a repeat in the first line, then the space on the line for notes is lessened from the normal length of a staff line by 10 spaces plus one space for each sign in the key signature.
If a line has a new key signature and a new time signature but no repeat in the first measure, then the space on the line for notes is lessened from the normal length of a stafF line by 6-1/2 spaces plus one space for each sign in the key signature.
If a line has a new key signature and a repeat in the first measure then the space on the line for notes is lessened from the normal length of a stafF line by 7-3/4 spaces plus one space For each sign in the key signature.
IF a line has a new time signature and has a repeat in the First measure then the space on the line For notes is lessened From the normal length of a starr line by 6
150
spaces.
IF a line has a new key signature only, then the space on the line For notes is lessened From the normal length of a staFF line by 4-1/2 spaces plus one space For each sign in the key signature.
IF a line has a new time signature only, then the space on the line For notes is lessened From the normal length. of a staFF tine by 2-1/2 spaces.
IF a line has a repeat in the First measure and no changes in key or time, then the space on the line For notes is lessened From the normal length or a staFF line by 1-1/2 spaces.
IF a line has no changes and thererore no special Figures to allow space ror then the space on the line ror notes is equal to the normal length or a stafr line.
IF a line has a diFFerent key signature than the line preceding it then both the clef and the key signature must be printed at the beginning of the line.
If a line has a diFferent clef than the line preceding it then both the cleF and the key signature must be printed at the beginning of the line.
IF a line has a different time signature that the line preceding it then the time signature and clef must be printed at the beginning of the line.
The mInImum size of a measure is determined by the following Formula:
where:
S * U + E
S is the sum of the durations of the notes in the measure; and
- U is the unitsize For the measure; and E is the amount of extra space needed for signatures, cleFs, etc.
'Allow 3 spaces in a measure for a Front repeat.
IF a rront repeat is followed by a time signature, then
151
there are 2~3/4 spaces between them.
IF a Front repeat is Followed by a key signature. then there are 2-3/4 spaces between them.
IF a Front repeat Follows a bar line. then they have the same location.
Allow 2-1/2 spaces For a time signature.
Ir a time signature is rollowed by a key signature. then there are 2-1/4 spaces between them.
IF a time signature rollows a bar line, t~en there are 1-3/4 spaces between them.
Allow 4 spaces plus one space For each sign but one Tor a key signature.
Allow one space iF the First note in a measure has an acc i denta 1 .
IF the next measure has a cleF change. then allow 5 spaces to print it in the current measure.
Allow 1-1/2 spaces iF the measure has an end repeat.
IF two notes are tied or slurred together and the notes are not on the same line, then create two "slur" nodes 'for the Figure. One node is From the first note to the end of the line and the other is From the beginning of a line to the second note.
IF two notes on the same line are slurred or tied together, then create "slur" node For the slur or tie.
IF a tie is a short tie and the notes are on a line, then the left end point is 3/4 spaces to the right of the notehead of the First note and 1/4 space above or below the notehead (in the direction or the tie).
IF a tie is a short tie and the notes are on a space, then the leFt end point is 1/2 space to the right of the First notehead and 3/4 spaces above/be10w it.
If'a tie is a short tie and the notes are on a line, then the right end point is 3/4 spaces to the lert or the second notehead and 1/4 space above/below it.
152
IF a tie is a short tie and the notes are on a space, then the right end point is 1/2 spaces to the leFt of the second notehead and 3/4 spaces above/below it.
IF a tie is a short tie and the notes are on a line, then the peak is located halFway between the noteheads and 1/2 space above or below them, depending on the direction of the tie.
IF a tie is a short tie and the notes are on a space, then peak of the tie is located halfway between the noteheads and 1 space above or below them.
If a tie is not a short tie and has both notes on a line, then the left endpoint is located 3/4 spaces to the right of the first notehead and 3/4 spaces above or below it (in the same direction as the tie).
If a tie is not a short tie and has both notes on a space, then the left endpoint is located 1/2 spaces to the right of the first notehead and one space above or below it.
If a tie is not a short tie and has both notes on a line, then the right endpoint is located 3/4 spaces to the right of the second notehead and 3/4 spaces above or below it.
If a tie is not a short tie and has both notes on a space, then the right endpoint is located 1/2 space to the right of the second notehead and one space above or below it.
If a then
tie the
noteheads them.
is not a short tie and has both notes on a line, peak of the tie is located halfway between the of the tied notes and 1-1/2 spaces above or below
If a tie is not a short tie and has both notes on a space, then the peak of the tie is located halfway between the noteheads of the tied notes and 2 spaces above or below them depending on the direction of the tie.
A curve is a short tie if it is a tie and the first and last notes are less than 3 spaces apart.
IF a slur or a tie has any connects with a downward stem, concave down.
note between the notes it then the slur or tie is
153
If the left note or a slur is on a line and the stem direction of the note is the same as the direction or the slur, then the lert end point or the s1ur is 1-1/4 spaces directly above or below the notehead (in the opposite direction of the stem).
If the leFt note of a slur is on a space and the stem direction or the note is the same as the direction of the slur, then the left end point of the slur is one space directly above or below the notehead (in the opposite direction or the stem).
If the left note of a slur has an upward stem. and the slur is concave down. then the left end point of the slur is 2 spaces to the right of the notehead and 1/2 space above the top of the stem.
If the right note of a slur is on a line and the stem direction of the note is the same as the direction of the slur, then the right end point of the slur is 1-1/4 spaces directly above or below the notehead (in the opposite direction of the stem).
If the right note of a slur is on a space and the stem direction of the note is the same as the direction of the slur. then the right end point of the slur is one space directly above or below the notehead (in the opposite direction of the stem).
If the right note of a slur has an upward stem. and the slur is concave down. then the right end point of the slur has the same Y coordinate as the n6tehead and is 1/2 space above the top of the stem.
If a slur is concave down, then the peak of the slur is one space above the highest obstacle between the left and right notes or the slur.
If a slur is concave up. then the peak of the peak of the slur is one space below the lowest obstacle between the left and right notes of the slur.
Appendix C
Diagrams or Husfcal Characters
This appendix diagrams or the graphical characters
used by ENGRAVE to produce musical notation.
154
155
Figure C-l Whole Note
10
156
10
Figure C-2 Closed Notehead
'0
157
Figure C-3 Open Notehead
'0
158
10
Figure C-4 Sharp
5
159
Figure C-5 Natural
:s
160
Figure C .. 6 Flat
5
161
8 ia~
/ ~
.........
... =-
- -----~ r"
...,
$ ~ \ I . I
$
162
Figure e·g Repeat
I i~
10
_ S
. I
.5 1.0
I-
-
i-
,.
l
1
....
10
r
r -
-t-
.
- r-
l
163
$ .
$
Figure C*9 Repeat (beginning)
. "20
5
164
Figure C-IO Flag
10
165
I/O
Figure C-l1 Treble Clef
20
166
Figure C .. 12 Bass Clef
'0
161
Figure C-13 Figure '3' for time signatures
168
Figure C-14 Figure '4' for time signatures
169
Figure C-15 Quarter Rest
5
170
Figure C-16 Eighth Rest
Appendix 0
NOTES
Input
expressed
to the
in NOTES
printing system is a piece or music
-- a language ror representing music
on a computer. We describe NOTES brieFly to acquaint the
reader with it and to aid in the understanding or the
system.
NOTES is a list-structured language suitable For use
with PROLOG or LISP. It has the advantage that it is also
reasonably easy For human beings to read.
A score is
elements or the
initializations
represented
1 i st are
such as the
as a list. The rirst
the composer, the title and
key signature and the time
signature. The elements Following are lists For each
movement or section. Each movement is Further decomposed
into parts For each instrument. It is the parts that
contain the important inFormation For printing.
To represent a note, we must First represent both
pitch and duration. A note is a list or at Jeast two
character strings. The First string represents
171
172
duration. Duration is a sequence of digits Followed by
zero or more "d"'s which may be Followed by an optional
"t" and more digits. The First digits Form an integer
that represents the type of note. A halF-note would be
represented by a "2" and a quarter-note by a "4". The
optional "d"'s are used For dotted notes one tid" For
each dot. The "t" shows that the note is part of an
n-tuplet group with the integer Following being n. Pitch
is represented by a three- or Four-character string. The
First character is the letter name of the note. The
second is the octave that the note is in and the third
and Fourth characters. iF present. represent an
accidental. IF the pitch is simply the letter "R". then
the note is a rest. Any other strings Following the
pitch will be modiFiers that represent articulation
markings. Where possible. these are literal indicators of
the Figure.
Measures are simply lists of notes. At the head of
the 1 i st is the word "meas" Fo 1 lowed by any mod i F i ers to
the measure such as a repeat or a key change. See Figure
0-1 For an example of a Few measures expressed in NOTES.
The Following is the NOTES representation of the
173
piece shown in Figure 8.
(part tenor (meas (4 F3)(4 EN3)(4){4 G3» (meas (4)(4 F#3)(8)(8 G#3) (4 A3» {meas (2)(8)(16 G#3)(16 F#3)(4 G3» (meas (4){8 A3)(8 E2)(8 A3)(16 Bb3)(16 A3)
(8 GN3)(8 A3» (meas (8 B3)(8 04)(8 G3){8 A3)(8 B3)(8 G3)
(8 E3)(16 FN3)(16 G3» (meas (4 A3)(2 G3)(4 F3» (meas (8)(16 E3){16 03)(4 A3)(4)(4 G3» {meas (4 A3)(4 R)(2 R» {meas (1 R» (meas (2 R)(2 83» (meas (2 A3){2 C4» (meas (4d BN3)(8 C#4) (2 04» (meas C4d)(16 C#4)(16 BN3)(2 C4» (meas (8 04)(8 E4)(8 04)(8 CN4) (8 BN3) (8 A3)
(8 83)(8 04»)
174
(meas (ad D5) (16 B4) (4 A4) (2 R) )
{meas (4t3 B4) (4t3 C5) (4t3 D5) (2 E5)
(meas (4 R) (4 r#5) (2 E5) )
Figure D-l NOTES Eumple
Appendix E
SpatIal Planner GraphIcs
E. 1 Character Symbo 1 s
Many of the symbols in musical notation are
characters. That is, they have a constant shape and
orientation. These characters were hand digitized and
made available as a sequence of points. A Figure is
speciFied by a list of triples (X Y OP) where X and Yare
spatial coordinates relative to the center of the Figure
and OP is either "move" or "line" indicating whether the
operation
location
should be MOVE_ABS_2 or LINE_ABS_2. When the
of a Figure is calculated, these values are
added to X and Y For each point and the proper operation
is perFormed. Drawings of these Figures are in Appendix
c.
E.2 Beaning
Among the non-character symbols are the beams
between adjacent eighth notes and smaller notes. These
beams vary in length and slope For diFFerent combinations
of notes. The slope of the beam must follow the general
175
176
slope of the notes it connects. We use linear regression
to compute the slope of the notes in a beat-unit and use
this slope as the slope of the beam. The stern directions
of all the notes in the beat-unit are determined by the
-stern-direction that the note Farthest From the middle of
the staff would have if it were not beamed. The rule for
locating the beam'dictates that no stem in the beat-unit
should have a length of less than a sixth (2 1/2 spaces).
We position the beam one octave (up or down
according to stern-direction) from one of the notes in the
beat-unit. This note is chosen so that all of the other
notes in the beat-unit have acceptable stem lengths. For
example, if the notes have up sterns and the beam has a
positive slope. then the note chosen is the highest note
in the beat-unit with ties going to the First note. 5ee
Figure 47 For some examples of beaming.
E.3 TIes and Slurs
Ties and slurs are modeled as conic sections using
the techniques outlined in [Faux and Pratt. 1979, pp.
17-36]. We find that the intersection of two conic
sections, 51 = 0 and 52 = 0 specifies a family of
177
conics given by
(1 - q)Sl + qS2 = 0
In particular,
= 0 be degenerate conics where 1 l' 12 and 13 are
straight lines. The equation
(1 - q)l112 + q13 = 0 (1)
speciFies a Family of conics that are tangent to 11 and
12 at their intersections with 13· See Figure E-l.
For a tie or slur we let the point (xl'Yl) be the
leFt-most point of the curve and -let (x2'Y2) be the
right-most point. We speciFy some peak by (x3'Y3)·
For ties, this peak is the midpoint of the two
noteheads. For slurs, it is slightly higher than the
highest obstacle that must be cleared_ The height is a
Function of the distance of the obstacle From the
midpoint of 13- The closer the obstacle is to the end
of the curve, the smaller the height becomes. To speciFy
the curve uniquely, let ali ne through
and perpendicular (speciFied by
118
~--~
-----,
Figure B-1
The illterseCtioll of two degenerate Collies
179
intersection of' 13 and 14· Now let (Xt,Yt) be
another point of' 14 such that (x3'Y3) is midway
between (xb'Yb) and (Xt ,Yt) • If' 1 1 is a 1 i ne
through (xI'Yl) such that 1 1 and 12 intersect at
(Xt,Yt) then we have speciFied our geometry as shown
in Figure E-2. The equation of 13 is:
13 = a3x + b3Y + c3
= (Y2 - Yl)x + (Xl - x2)Y + x2Yl - xIY2
Since 14 is perpendicular to 13' its slope is
-1 -x2 - xl xl - x2 m4 = = ------- = -------
m3 Yl - Yl Y2 - YI
and its equation is
14 = a4x + b4Y + c4
= (x I - x2)x + (Yl - Y2)Y
+ x3(x2 - X I) + Y3(Y2 - Yl)·
Solving the intersection of 13 and 14 ' we obtain
b3c 4 - b4c 3 xb = -----------
a3b 4 - a4b3
a4c 3 - a3 c 4 Yb = -----------
J33b4 - J34b3
Then
Xt = 2x3 - xb
and
ISO
Pi'Ure B.2
tbe geometry tor c~CU1atillg slurs IIIld lies
181
Yt = 2Y3 - Yb
by our geometry. UsIng these values, we find that
11 = alx + b1Y + c1
= {Yt - Yl)x + (xl - Xt)Y + XtYl - xIYt
and
12 = a2x + b2Y + c2
= (Y2 - Yt)x + (Xt - x2)Y + x2Yt - XtY2
Substituting into (I), we get
(1 - q) 1 11 2 + q 1 32
= Ax2 + Bxy + Cy2 + Ox + Ey + F (2)
aFter some manipulation where
2 A = a 1 a2 ( 1 - a) + a3 q
B = (alb2 + a2b l)(l - q) + 2a3b3q
C = bI b 2(l - q) + b3 2q
0 = (aI c 2 + a2c I)(l - q) + 2a3c3Q
E = (blc2 + b2c I)(1 - q) + 2a3c3q
F = clc2(1 - q) + C32q.
To produce this curve, we take advantage of the fact
that it is relatively Flat. Starting at Xl we
increment X and compute Y unt f 1 we reach x2.
Rearranging (2) to solve for Y we get
Cy2 + (Bx + E)y + (Ax2 + Ox + F) = 0
182
and
-(Bx + E) ± (Bx + E)2 - 4C(Ax2 + Ox + F) Y =
2C
and when C = 0
Ax2 + Ox + F Y =
Bx + E
This geometry creates an ellipse so we use the positive
square root when the curve is concave down and the
negative square root when the curve is concave up.
The geometry we have used works quite well in most
cases. Figure 46 shows an example o~ the ties and slurs
that can be produced.