HAL Id: hal-02355473https://hal.archives-ouvertes.fr/hal-02355473

Submitted on 8 Nov 2019

HAL is a multi-disciplinary open accessarchive for the deposit and dissemination of sci-entific research documents, whether they are pub-lished or not. The documents may come fromteaching and research institutions in France orabroad, or from public or private research centers.

L’archive ouverte pluridisciplinaire HAL, estdestinée au dépôt et à la diffusion de documentsscientifiques de niveau recherche, publiés ou non,émanant des établissements d’enseignement et derecherche français ou étrangers, des laboratoirespublics ou privés.

Methodology for evaluating automated mapgeneralization in commercial software

Jantien Stoter, Dirk Burghardt, Cécile Duchêne, Blanca Baella, Nico Bakker,Connie Blok, Maria Pla, Nicolas Regnauld, Guillaume Touya, Stefan Schmid

To cite this version:Jantien Stoter, Dirk Burghardt, Cécile Duchêne, Blanca Baella, Nico Bakker, et al.. Methodologyfor evaluating automated map generalization in commercial software. Computers, Environment andUrban Systems, Elsevier, 2009, 33 (5), pp.311-324. �10.1016/j.compenvurbsys.2009.06.002�. �hal-02355473�

Computers, Environment and Urban Systems 33 (2009) 311–324

Contents lists available at ScienceDirect

Computers, Environment and Urban Systems

journal homepage: www.elsevier .com/locate /compenvurbsys

Methodology for evaluating automated map generalization in commercial software

Jantien Stoter a,*, Dirk Burghardt b,1, Cécile Duchêne c, Blanca Baella d, Nico Bakker e, Connie Blok a,Maria Pla d, Nicolas Regnauld f, Guillaume Touya c, Stefan Schmid b

a ITC, Enschede, The Netherlandsb University of Zurich, Switzerlandc IGN, Franced ICC, Catalonia, Francee Kadaster, The Netherlandsf Ordnance Survey, Great Britain, UK

a r t i c l e i n f o

Keywords:NMA requirements for automated map

generalizationEvaluation of generalized dataConstraint-based generalizationState-of-the-art of generalization

0198-9715/$ - see front matter � 2009 Published bydoi:10.1016/j.compenvurbsys.2009.06.002

* Corresponding author. Present address: GISt, OT2600 GA Delft, The Netherlands.

E-mail address: j.e.stoter@tudelft.nl (J. Stoter).1 Present address: Institute of Cartography, Dresd

Helmholzstr. 10, Dresden, Germany.

a b s t r a c t

This paper presents a methodology developed for a study to evaluate the state of the art of automatedmap generalization in commercial software without applying any customization. The objectives of thisstudy are to learn more about generic and specific requirements for automated map generalization, toshow possibilities and limitations of commercial generalization software, and to identify areas for furtherresearch. The methodology had to consider all types of heterogeneity to guarantee independent testingand evaluation of available generalization solutions. The paper presents the two main steps of the meth-odology. The first step is the analysis of map requirements for automated generalization, which consistedof sourcing representative test cases, defining map specifications in generalization constraints, harmoniz-ing constraints across the test cases, and analyzing the types of constraints that were defined. The secondstep of the methodology is the evaluation of generalized outputs. In this step, three evaluation methodswere integrated to balance between human and machine evaluation and to expose possible inconsisten-cies. In the discussion the applied methodology is evaluated and areas for further research are identified.

� 2009 Published by Elsevier Ltd.

1. Introduction

Research in automated map generalization has yielded manypromising results (Mackaness, Ruas, & Sarjakoski, 2007). At thesame time, vendors face difficulties in implementing automatedgeneralization solutions in commercial software (Stoter, 2005),which occurs for several reasons.

First, a formal definition of map specifications is lacking.Although a satisfying generalization solution can be defined in gen-eral terms—e.g., as a map that reduces the details and discerns re-gional patterns, that is aesthetically pleasant, and enables users tosucceed in a given task (Mackaness & Ruas, 2007)—it is difficult tospecify specifications into such a format and knowledge level insuch a way that they can steer the automated generalization pro-cess. Second, software vendors need map specifications that areshared by several map producers such as National Mapping Agen-

Elsevier Ltd.

B, TU Delft, P.O. Box 5030,

en University of Technology,

cies (NMAs) to justify their investments. Such shared generaliza-tion specifications are not easy to formulate because ofdifferences in data models, level of detail of initial data, landscapesto be mapped, scales to be produced, etc. A final reason for the dif-ficult implementation of automated map generalization is thatgeneralization is a subjective process in which more than one idealgeneralization result is often possible. This subjectivity in solvingcartographic conflicts is a challenge to automate.

To address these difficulties, we conducted a study on the stateof the art of automated map generalization in commercial soft-ware. Specifically, through the study we aimed to learn more aboutgeneric and specific map specifications of NMAs, to encourage andsupport vendors in implementing these specifications in commer-cial software, and to identify areas for further research. The studytook place in the framework of EuroSDR (European Spatial Data Re-search), where NMAs, research institutes, and private industrywork together on research topics of common interests.

The present paper focuses on the methodology that we devel-oped to evaluate complete maps, generalized by different systemsand different testers, taking into account the differing map specifi-cations of several NMAs. The methodology had to consider all kindsof heterogeneity to guarantee independent testing and evaluation ofavailable generalization solutions. To meet these heterogeneities,

312 J. Stoter et al. / Computers, Environment and Urban Systems 33 (2009) 311–324

the methodology consisted of two main steps: requirements analy-sis for automated map generalization and evaluation of generalizedoutputs.

Our paper starts with an overview of previous research related todefining specifications for automated map generalization in Section2. This section also defines the scope of the current study. Section 3describes the first main step of the methodology, i.e., the require-ment analysis. This step consisted of sourcing representative testcases, defining map specifications as generalization constraints, har-monizing constraints across the test cases, and analyzing the typesof constraints defined. Section 4 presents the second main step ofthe methodology, i.e., the evaluation of generalization outputs. Thisstep included developing and integrating three evaluation methods:expert evaluation, automated constraint-based evaluation, andqualitative comparison of outputs. The paper concludes with anevaluation of the methodology, sharing insights obtained duringthe tests, and identifying areas for further research (Section 5).

2. Background

2.1. Previous research related to specifications for automated mapgeneralization

An overview of previous studies on formalizing map knowledgefor automated generalization can be found in Sarjakoski (2007).Various researchers have studied specifications for automatedmap generalization (Foerster, Stoter, & Kraak, 2009). Müller andMouwes (1990) examined existing map series to conclude that‘‘superficial” generalization knowledge exists in the form of mapspecifications written down for interactive generalization. Comple-mentary to this ‘‘superficial” knowledge, cartographers use ‘‘deep”generalization knowledge to interpret superficial knowledge. Thisdeep knowledge is much harder to automate. Rieger and Coulson(1993) carried out a survey among a group of cartographers per-forming interactive generalization and concluded that a commonview on the classification of generalization operators does not ex-ist. Nickerson (1991) and Kilpelaïnen (2000) acquired knowledgefrom experts to define rules for knowledge-based map generaliza-tion. Various studies used reverse engineering to collect general-ization knowledge by comparing map objects across scales(Buttenfield (1991), Leitner and Buttenfield (1995), and Weibel(1995)). Other studies generated rules from interactive generaliza-tion carried out by a cartographic expert (Weibel (1991), Weibel,Keller, and Reichenbacher (1995), McMaster (1995), and Reichenb-acher (1995)). Several studies applied machine learning techniquesto convert expert knowledge into map specifications for automatedgeneralization, e.g., Weibel et al. (1995), Plazanet, Bigolin, and Ruas(1998), Mustiere (2001, 2005) and Hubert and Ruas (2003). Brewerand Buttenfield (2007) ran map exercises with students, on differ-ent datasets at various scales, to provide guidelines for generaliza-tion processes.

Our study builds primarily on the research by Ruas (2001),which took place within the European Organization for Experimen-tal Photogrammetric Research (OEEPE; the predecessor of Euro-SDR) and investigated the state of the art of generalization byevaluating different interactive generalization software. Ruas’sstudy aimed to obtain insight into generalization processes for car-tographic purposes—not to evaluate generalization packages orcomplete generalized output. The OEEPE study tested five plat-forms on three generalization cases for a selection of themes. Gen-eralization operators on individual objects or groups of objectswere triggered by testers’ interaction. Because of a lack of writtenspecifications, the target maps served as examples. Templatesdeveloped for the project included lists of cartographic conflicts,operations, and algorithms.

Several of Ruas’s recommendations are relevant for the method-ology presented in our paper. First, a formalized description ofspecifications for the output maps should help to obtain bettersolutions. Furthermore, tests should be evaluated by a more flexi-ble and digital method, since the manual tracing of all testers’ out-put in Ruas’s study was extremely labor-intensive. Finally, testsshould use symbolization information to standardize the outputs.In our study we have implemented all of these recommendations.

2.2. Scope of the current study

The two main questions of our study were:

(1) What are the possibilities and limitations of commercialsoftware systems for automated generalization with respectto NMA specifications?

(2) What different generalization solutions can be generated forone test case and why do they differ?

Several aspects defined the scope of the study.First, the aim of the study was to obtain knowledge on different

aspects of automated map generalization with respect to NMAspecifications, and to discover how these are implemented in com-mercial software. The potential and limitations of individual sys-tems were therefore not relevant.

Second, our study focused on map specifications of NMAs. Thestudy did not consider specifications of map end-users, becausesurveys performed by NMAs among their customers showed a con-tinuous need for traditional, paper maps representing topographyat different scales. This implies that NMAs still have to produce tra-ditional map series, and justifies our focus on NMA map specifica-tions. Although this study is driven by large volume (paper) mapproduction at NMAs, one should realize that the results are highlyrelevant for other map producers and for web mapping.

Third, our study focused on large- to mid-scale generalization,since the involved NMAs considered this the most time-consuminggeneralization task of current production lines.

Fourth, our study focused on complete maps, rather than onspecific situations. Therefore, the generalization processes shouldnot be a sequence of operations triggered by conflicts on individualobjects or a group of objects as in Ruas’s OEEPE research, but betriggered by object class (theme) or spatially indicated areas(partitions).

A final focus of the study was to limit the tests to commerciallyavailable versions of software to allow us to conclude on generali-ties. Consequently, research team testers, either experienced orinexperienced with the systems, were not allowed to customizethe software nor to program new algorithms. This did not mean thatthe implementation of specifications was straightforward: all testedsystems—ArcGIS (ESRI), Axpand/Genesys (Axes systems), Change,Push, Typify (University of Hannover) and Clarity (1Spatial) —pro-vide considerable flexibility to deal with the specifications. Conse-quently, many decisions on how to express the specifications wereleft to the testers. In some systems testers had to decide on the orderof addressing the specifications; in other systems they had to decidewhich algorithms and parameters values to use. Therefore, all testsrequired considerable effort to align the functionality of the systemswith specific test cases. To enable vendors to show all the potentialsof their system, they performed parallel tests in which they were al-lowed to customize and develop new algorithms.

3. Requirement analysis

This section presents the results of the requirements analysis.Section 3.1 describes the selection of test cases representing map

J. Stoter et al. / Computers, Environment and Urban Systems 33 (2009) 311–324 313

generalization problems. Section 3.2 describes the formalization ofNMA specifications for automated map generalization. Section 3.3reports on the harmonization that was carried out to produce onegeneric set of formal map specifications within the context of ourstudy. Section 3.4 analyzes the defined specifications to learn moreabout similarities and differences between map specifications ofNMAs.

3.1. Selecting the test cases

The first step in the requirement analysis was the selection oftest cases representing problems for automated map generaliza-tion. To meet this objective, we generated a list of outstandingmap generalization problems based on the OEEPE research com-pleted with the research team’s own experience. Examples of theseproblems are building generalization in urban zones, mountainroad generalization, solving overlapping conflicts in locally densenetworks, pruning of artificial networks, and ensuring consistencybetween themes in particular areas such as coastal zones. Some ofthese problems have been tackled in research, resulting in at leastpartial solutions. However, we wanted to evaluate complete solu-tions in commercial systems, and, therefore, these problems werealso identified as representative map generalization problems.We selected four test cases that included all these problems (seeTable 1) provided by Ordnance Survey Great Britain (OSGB), Insti-tute Geographique Nationale, France (IGNF), The Netherlands’ Kad-aster (Kadaster) and Institut Cartogràfic de Catalunya (ICC).

The NMAs of the test cases modified their datasets to preparethem as input for the generalization tests, e.g., details such as richclassifications were removed from the datasets and the datasetswere translated into English. In addition, to be able to define spec-ifications of the output maps with respect to symbolized objectsand to assure uniform outputs, the NMAs defined symbols forthe outputs. Fig. 1 shows cutouts of the source datasets.

3.2. Formalizing NMA specifications for automated map generalization

In the task of formalizing map specifications for automated gen-eralization, we can distinguish between two stages. The first stageis to describe the specifications in a way that a user (in our case thetesters of the systems) fully understand what (s)he should try toobtain with the system. The second stage is to translate these spec-ifications in a format understandable by the generalization system.The first stage was completed by means of cycles between the dataproviders and the research team. The second stage was completedby the testers during the test process.

To implement research theories, we formalized map specifica-tions of NMAs as a set of cartographic constraints to be respected.In previous research on generalization, the use of constraints is acommon method to define specifications and to control and evalu-ate the automated generalization process. Examples are McMasterand Shea (1988), Beard (1991), Bard (2004), Barrault et al. (2001),Ware, Jones, and Thomas (2003), Burghardt and Neun (2006), andSester (2000). Constraints express how generalization outputshould look without addressing the way this result should beachieved, e.g., by defining sequences of operations.

Table 1Test cases selected for the EuroSDR research.

Area type Source dataset Target dataset (k) Provided by

Urban area 1:1250 1:25 OS Great BritaMountainous area 1:10 k 1:50 IGN FranceRural area 1:10 k 1:50 Kadaster, NLCoastal area 1:25 k 1:50 ICC Catalonia

We developed a template for a uniform way to define con-straints in the four test cases. In the template specific propertiesof the constraint can be defined such as condition to be respectedand the geometry type and feature class(es) to which the con-straint applies (see Appendices A–C and Table 3). The template dis-tinguishes between constraints on one object, on two objects, andon groups of objects. An importance value indicates the impor-tance of satisfying the specific constraint in the final output. Thisvalue does not indicate in what sequence the constraints shouldbe solved (Ruas, 1999). Satisfying less important constraints firstmay be necessary to satisfy more important constraints later. Forexample, generalization of buildings should start with reducingdensity before trying to cope with overlaps, even though non-over-lapping constraints are more important than density constraints.NMAs could also propose an action to support the tester in findingthe most desired generalization solution. This is because in somecases NMAs know what action should be taken to meet the con-straint optimally, e.g., the action ‘‘exaggerate detail” for constraint‘‘minimal depth of protrusion of a building.”

3.3. Harmonizing constraints

NMAs defined their map specifications for automated general-ization in the developed template by analyzing text-based mapspecifications, software code, and cartographers’ knowledge. Ini-tially a large number of constraints were defined for the four testcases (about 250), which often covered similar situations.

In the next step we harmonized the constraints, which wasneeded for two reasons. Harmonization, resulting in the same con-straints for similar situations, unified the tests. Once a tester hadexpressed the constraint for one test case, (s)he could performthe same actions to express a similar constraint for a second testcase. Second, harmonization enabled us to compare results for sim-ilar constraints across the test cases.

For the harmonization, similar constraints across the four testcases were identified by carefully comparing the four constraint sets.The harmonization resulted in a list of generic constraints. A few con-straints were so specific that they remained as a specific constraint.Examples are OSGB constraints addressing how buildings should beaggregated depending on the initial pattern. The harmonization pro-cess resulted in 45 generic constraints: 21 generic constraints on oneobject (see Appendix A), 11 constraints on two objects (see AppendixB), and 13 constraints on a group of objects (see Appendix C). Theharmonized constraints describe those properties of the constraintsthat are generically applicable. These constraints contain blank en-tries to be completed by NMAs to define their constraints as specifi-cation of the generic constraints. The columns in the harmonized set(e.g., class, action, importance) only contain values when the value isapplicable for any case, except for the column ‘Condition to be re-spected’ which is always filled, mostly with non-specified parametervalues. In all other cases NMAs can specify their classes, actions,parameter values and importance values to define their constraintsas specification of the generic constraints.

Table 2 shows examples of generic constraints on one object,two objects, and a group of objects (the constraint type will beintroduced in Section 3.4).

No. of feature classes Main feature classes

in 37 Buildings, roads, river, relief23 Village, river, land use29 Small town, land use, planar partition74 Village, land use (not mosaic), hydrography

Fig. 1. Cutouts of source datasets in the EuroSDR generalization study. Maps are reduced in size.

314 J. Stoter et al. / Computers, Environment and Urban Systems 33 (2009) 311–324

After all four NMAs agreed on the harmonized constraints, theyredefined their initial constraints as generic constraints using theirown feature classes, thresholds, parameter values, and preferredactions, see Table 3 for an example of ICC (all NMA specific infor-mation is indicated in red).

3.4. Analyzing the test cases

To obtain more in-depth knowledge on NMA specifications forautomated map generalization, the final step of the requirementanalysis was the comparison of constraints across the four testcases.

For this comparison, one should realize that the constraint setsdo not reflect all generalization problems of NMAs. First, the NMAshad to limit their constraints to those describing the main prob-lems in the test area and to constraints that were more or less

straightforward to formalize. Second, the constraints were definedwithout running any automated generalization process that wouldhave shown both missing and unclear constraints. Last, the amountof time allocated to the testers would never enable them to set upthe equivalent of a complete generalization production line, han-dling all specifications for one given map scale; therefore, NMAslimited their efforts on constraints that could be tackled withinthe context of the tests.

For the comparison of constraints among the four test cases weused three criteria: (1) the number of objects taken into account inthe constraints, (2) the type of the constraints, and (3) the featureclass for which the constraints were defined.

For the constraint type we distinguished between two main cat-egories: legibility constraints and preservation constraints (Burg-hardt, Schmidt, and Stoter (2007)). Preservation constraints arecompletely satisfied at scale transitions. These are constraints

Table 2Examples of harmonized constraints.

Constraint type Property Condition to be respected

Constraints on one objectMinimal dimension Area Target area > x map mm2; target area = initial area ±x%

Width of any part Target width > x map mmArea of protrusion/recess Target area > x map mm2

Length of an edge/line Target length > x map mmShape General shape Target shape should be similar to initial shape

Squareness [Initial value of angle = 90� (tolerance = ±x�)] target angles = 90�Elongation Target elongation = initial elongation ±x%

Topology Self-intersection (Initially, no self-intersection) no self-intersection must be createdCoalescence Coalescence must be avoided

Position/orientation General orientation Target orientation = initial orientation ±x%Positional accuracy Target absolute position = initial absolute position ±x map mm

Constraints on two objectsMinimal dimensions Minimal distance Target distance >x map mmTopology Connectivity [Initially connected] target connectivity = initial connectivityPosition Relative position Target relative position = initial relative position

Constraints on a group of objectsShape Alignment Initial alignment should be keptDistribution & statistics Distribution of characteristics Target distribution should be similar to initial distribution

Density of buildings (black/white) Target density should be equal to initial density ±x%

Table 3Example of ICC map specifications defined as constraints that extend the EuroSDR harmonized constraints.

Item in constraint template Example on one object Example on two objects Example on group of objects

Constraint ID ICC-1-22 ICC-2-21 ICC-3-18Geometry type Polygon Polygon–line PolygonsFeature class 1 Quay_adjacent_to_sea Building BuildingCondition for object being concerned with this

constraintDepth of protrusion>1 map mm

Distance between building and road<0.5 map mm

Constrained property Width of protrusion/recess

Orientation Density of buildings (black/white ratio)

Condition depends on initial value? No Yes YesCondition to be respected Target width

>0.2 map mmBuilding must be parallel to road Target density should be equal to initial

density ±20%Action Collapse to a lineImportance of constraint (1–5, 1 is less

important)3 3 3

ExceptionSchema to illustrate if neededAdditional for constraints on two objects:Feature class 2 RoadCondition for both objects being concerned with

this constraintObjects are parallel (±15�)

Additional for constraints on group of objects:Kind of group Urban blockKind of objects of the initial data composing the

groupBuildings surrounded by minimal cycle ofroads (in urban areas)

J. Stoter et al. / Computers, Environment and Urban Systems 33 (2009) 311–324 315

prescribing topology, position, orientation, shape, and distribution/statistics. Preservation constraints may be violated when opera-tions are applied for ensuring legibility (minimal dimensions andgranularity). Legibility can be investigated independently of thesource dataset, while preservation always has to be evaluated incorrelation with the source data. Besides legibility and preserva-tion constraints, we identified ‘‘model generalization” constraints.These refer mainly to constraints for removing certain featuretypes from the data (e.g., ‘‘cycle path” in the Kadaster test case or‘‘wall” in the ICC test case). These constraints are also for avoidingaggregation of objects with different attributes; e.g., different typesof buildings in the OSGB test case should not be aggregated.

Table 4 shows the results of comparing the four constraint setsusing the three criteria. Several conclusions can be drawn from thistable. First, the ICC test case contains a large number of constraintscompared to the other cases. This can be explained by the largenumber of feature classes (see Table 1) resulting in several similar

constraints for different types of roads. Second, most constraintsare defined for one object in all four cases, whereas the fewest con-straints are defined for groups of objects, most likely because itwas difficult to define constraints on groups of objects. Third, con-straints for ensuring minimal dimensions are important in all fourtest cases, showing the importance of these constraints in the car-tographic generalization process. Another observation is that topo-logical constraints are defined on a more general level such as‘‘preserve topological consistency and connectivity,” ‘‘self-intersec-tion not allowed,” or ‘‘keep adjacency.” It is notable that there areonly a few shape constraints defined by Kadaster. Position and ori-entation constraints are sparsely specified by all NMAs, and theyrefer only to buildings. One explanation could be that buildingsare expected to be displaced more often than other objects duringthe generalization process. A final conclusion of this analysis con-cerns the feature classes that were included in the constraint def-initions. All four test cases contain many constraints on buildings,

Tabl

e4

Ana

lysi

sof

cons

trai

nts

ofth

ete

stca

ses,

clas

sifi

edon

vari

ous

crit

eria

.

Test

case

Tota

ln

um

ber

ofco

nst

rain

ts

Nu

mbe

rof

obje

cts

Con

stra

int

type

Feat

ure

clas

ses

invo

lved

On

one

obje

ct

On

two

obje

cts

On

grou

pof

obje

cts

Mod

elLe

gibi

lity

Pres

erva

tion

Bu

ildi

ng

Lan

du

seR

oad

Wat

erR

elie

fC

oast

alfe

atu

res

An

yO

ther

Mod

elge

ner

aliz

atio

nM

in.

dim

ensi

onan

dgr

anu

lari

ty

Posi

tion

Ori

enta

tion

Shap

eTo

polo

gyD

istr

ibu

tion

/st

atis

tics

Oth

er

ICC

137

8623

2812

800

419

125

539

2016

258

199

1K

adas

ter

5227

214

1118

10

16

015

1013

233

00

03

IGN

F61

3215

142

152

415

122

933

212

92

02

1O

SGB

4924

1312

216

10

08

022

241

81

80

25

Tota

l29

916

972

5816

129

48

3538

751

106

3659

3818

1913

10

316 J. Stoter et al. / Computers, Environment and Urban Systems 33 (2009) 311–324

land use, and roads. The reason for the importance of these classesin the constraint sets is most likely because these are the most fre-quently occurring objects and the most significant for users of themap and therefore most (interactive) generalization is applied tothese objects. The variation of constraints among other featureclasses is a result of the relative importance of certain feature clas-ses within the four chosen test cases; e.g., constraints on coastalfeatures are dominant in the ICC case.

Every system was tested two to three times on all four test casesby generalization experts, who were both skilled and unskilledwith the systems. In every test, the tester tried to translate all de-fined specifications into a form understandable by the specific soft-ware. After the testing, the outputs were evaluated using amethodology that is explained in the next section.

4. Evaluating generalized outputs

Evaluating generalized data can serve three main tasks: evalua-tion for tuning the generalization system prior to generalization,evaluation for controlling the generalization process during general-ization, and evaluation for assessing the quality of generalized dataafter generalization (Mackaness & Ruas, 2007). The purpose ofevaluating generalized data in our study falls in the last category.However, the evaluation serves a second, more specific aim, whichis learning more about generalization processes.

The methodology that we developed to evaluate the generalizedoutputs of the tests was driven by an observation by Mackanessand Ruas (2007). They stated that an adequate evaluation frame-work should be able to handle the notion that the final output isa compromise among a set of sometimes competing map objec-tives. Such a framework should balance between human evalua-tion and machine evaluation to meet the complexity ofevaluation; e.g., machine evaluation can direct the user to thoseparts of the solution that are deemed to be unsatisfactory.

Based on this observation and motivated by the constraint-based approach of the requirement analysis of our study, we devel-oped three integrated methods for evaluating the generalized data:

1. qualitative evaluation by cartographic experts,2. automated constraint-based evaluation, and3. evaluation, which visually compared different outputs for one

test case

The integration was accomplished by directing experts on situ-ations that were well, badly, or differently solved according to theautomated constraint-based evaluation. In addition, the results ofthe visual comparison of outputs were discussed with the expertsof the test cases. Conclusions of one method are also comparedwith results of the other two methods to identify inconsistent mea-suring tools.

All 34 outputs produced by the tests were evaluated. Thesewere 27 outputs delivered by research team testers and seven out-puts delivered by vendors.

The three evaluation methods are explained in Sections 4.1–4.3.More details can be found in Burghardt et al. (2008).

4.1. Expert evaluation

For the expert evaluation, a survey was developed that extendsthe earlier experts’ survey of the AGENT prototype (AGENT, 2000).The survey, completed by cartographic experts of the four NMAs,focused both on global indicators and on individual constraints.The global indicators used to assess the outputs are shown in Table5. For the assessment of the outputs on individual constraints, itappeared to be impossible to visually assess whether a threshold

Table 5Global indicators used in the expert survey.

Global indicators

Level of manual editions required to meet the constraintsDeviation from initial (undergeneralized) dataPreservation of the geographic characteristics of the test area (urban,

mountainous, rural, or coastal area)LegibilitySeriousness and frequency of major detected errorsNumber of positive aspectsInformation reduction (ungeneralization/overgeneralization)

J. Stoter et al. / Computers, Environment and Urban Systems 33 (2009) 311–324 317

value, as often used in the definition of the constraints, was met.Therefore, we summarized the original constraints in a set of con-straints that could be visually assessed (see Table 6). Cartographicexperts assessed how these derived constraints were solved: eithervery badly, badly, well, or very well.

At the end of the survey, experts annotated the output mapswith examples of good (g), bad (b), and differently solved general-ization solutions (d) (see Fig. 2).

4.2. Automated constraint-based evaluation

The automated constraint-based evaluation compared the mea-sured final value (e.g., ‘‘size’) for a constraint with an ideal final va-lue. For this evaluation an OpenJump prototype (OpenJump, 2008)was developed (see Fig. 3). This prototype implemented the auto-mated evaluation of two legibility constraints: ‘‘target area > x mapmm2” (for one object) and ‘‘target distance > x map mm” (betweentwo objects). The outcome of these evaluations is either 0 (perfectsolution) or 1 (violated constraint).

Table 6Individual constraints used in the expert survey.

Constraints on one object Constraints on two objects

Minimal dimensions Spatial separation between features (distance)Granularity (amount of detail) Relative position (e.g., building should remain atShape preservation Consistencies between themes (e.g., contour line

Fig. 2. Generalization output of the Kadaster tes

Although the implementation of automated evaluation of thesetwo constraints was more or less straightforward, the implementa-tion for most other constraints appeared to be difficult and wastherefore not realized. The reason for this is that the definition ofconstraints mainly aimed at being unambiguously clear for testers.Therefore, we did not endeavor to make them as formal as possible.Although for some constraints (e.g., shape and spatial distribution)it is known that the definition and the measurement are complex, ahigher level of formalization could have been achieved. A con-straint such as ‘‘initial and generalized shape should be similar”is less formal than the constraint ‘‘preserving width–length ratio.”For this reason specifically, the constraints defined for group of ob-jects appeared to be very difficult (if not impossible) to evaluate inan automated manner; examples are constraints on networks, pat-terns, and spatial distributions.

To show to what extent automated constraint-based evaluationis appropriate to identify the quality of generalized data, we ap-plied the prototype to interactively generalized data of Kadaster,scale 1:50 k (the target dataset of the test case of Kadaster). In thistest we assumed that the interactively generalized data, which iscurrently in production, is a good generalization result.

We evaluated two constraints: minimum area of buildings andminimum distance between buildings. The results for the firstconstraint show that 27% of the buildings are smaller than thethreshold (0.16 map mm2) and are therefore evaluated as bad(see Fig. 4). However, when examining the data in more detail,we found that many ‘‘too small buildings” are just a little belowthe threshold size. The difference in minimum size, as mentionedin the written specifications (main source for the constraints) andas used in interactive generalization, can be explained in twoways. First, it is not possible for humans to distinguish betweenthe threshold and the threshold plus/minus a flexibility range,and, therefore, cartographers use the thresholds with a notion of

Constraints on a group of objects

Quantity of information (e.g., black/white ration)the same side of a road) Spatial distributionand river)

t case, annotated by a cartographic expert.

Fig. 4. Results of analyzing minimal building areas in interactively generalizeddata, scale 1:50 k.

Fig. 5. Results of analyzing minimum distance between buildings constraint oninteractively generalized data, scale 1:50 k. The non-violating buildings are notshown in this graph.

Fig. 6. Minimal distance constraint identifies unacceptable situations (a). Acc

Fig. 3. Screen shot of prototype for automated constraint-based evaluation.

318 J. Stoter et al. / Computers, Environment and Urban Systems 33 (2009) 311–324

flexibility (Bard, 2004; Ruas, 1999). Second, in specific situa-tions the cartographer may have chosen to relax the sizeconstraint to meet a more important constraint, e.g., ‘‘keep impor-tant buildings.”

The automated evaluation of the constraint on minimal dis-tance (2 map mm) in the interactively generalized dataset alsoshows many violations of the constraint. 46% of the buildings aretoo close to each other (Fig. 5). The violations can partly be ex-plained by the notion of flexibility and by deliberately violatingconstraints to meet more important constraints, as discussedabove.

However, because of the high number of violations, we exam-ined the violated situations in more detail and encountered manysituations assessed as ‘‘bad,” as shown in Fig. 6b and c. To be able todistinguish between Fig. 6a, on the one hand (in which the mini-mum distance constraint does identify a cartographic conflict),and Fig. 6b and c (which may be acceptable solutions), minimaldistance between buildings should be further refined in constraintdefinitions.

The conclusion of this automated evaluation of interactivelygeneralized data is that constraint-based evaluation requires fur-ther research to be able to describe the quality of generalized data.Future research should aim at better definition of constraints withrespect to automated evaluation and better understanding of theimpacts and dependencies of several constraints.

Section 5 (discussion and conclusion) contains several recom-mendations on how constraint-based evaluation can be improvedto become more appropriate for assessing generalized data.

eptable generalization solutions violate the distance constraint (b and c).

Fig. 7. Focus zone on generalization of buildings in suburban areas use to compare outputs for one test case. ICC initial data (a) and seven generalization outputs ((b)–(h)).

J. Stoter et al. / Computers, Environment and Urban Systems 33 (2009) 311–324 319

4.3. Visual comparison of outputs

The objective of the visual comparison of generalized data wasto describe the differences between outputs for one test case froma qualitative point of view and to explain the differences. The eval-uation carefully examined three to five zones per test case, whichwere identified by the NMAs as being of particular interest. Exam-ples are buildings and streets in cities and suburban areas, coast-lines, road interchanges, parallel roads, mountainous roads,vegetation, and dense channel networks. Fig. 7 shows an exampleof such a focal zone (buildings in suburban area) in the outputs ofone test case. This evaluation obtained insights into the interde-pendencies between different constraints, the completeness andclarity of constraints, and the influence of testers’ experiences withboth the systems and data on the generalized output.

5. Discussion and conclusions

In this paper we have presented the evaluation methodology wedeveloped to assess generalization outputs produced by varioussoftware packages and different testers, taking into account thediffering specifications of the participating NMAs. From the devel-opment and application of the methodology, several conclusionscan be drawn that identify issues for further research.

5.1. Defining map specifications as constraints

The definition and harmonization of constraints formalizingNMA map specifications provided a common view on require-ments for automated map generalization. Although very time con-suming, defining map specifications as a set of constraints was agood experience for the NMAs, because it highlighted the impor-tance of explicitly defining NMA data and mapping specificationsfor automated processes.

The harmonized list of constraints as a result of our study is,however, not complete. The NMAs had to limit their constraintsto those describing the main problems within the selected testareas and to constraints that were more or less straightforwardto formalize. In addition, the constraints were defined without run-ning any automated generalization process, which would have

shown both missing and unclear constraints as well as how specificconstraints work in practice. Nonetheless, the resulting set of con-straints is a first attempt to define a ‘‘full” set of constraints asimplementation of research theories.

5.2. Formalizing and evaluating preservation specifications

The preservation specifications were more difficult to formalizeand to evaluate than the legibility specifications. Therefore, betterunderstanding of preservation specifications is required to im-prove their formalization in constraints as well as the measure-ment of constraint violation. This includes a better understandingof the concepts involved (i.e., how to mathematically describe‘‘shape”) and of the changes allowed (how to mathematically de-scribe accepted modifications). Harrie (2001) obtained such infor-mation by studying existing maps at different scales.

Another problem in evaluating preservation constraints is that acorrespondence is required with the initial data. This is not an is-sue in 1:1 relationships; however, because of operators as selec-tion, typification, amalgamation, and aggregation relationshipsmay become complex, which makes it difficult to compare outputdata with the initial data.

The difficulty of evaluating preservation specifications was alsoencountered in the expert survey: it was often unclear whether apreservation constraint was assessed as ‘‘good” because the systemhad carefully accounted for it, or because the system had simply ig-nored it and at the same time had not much altered the data duringthe process.

5.3. Generalizing through constraints

Our methodology used constraints mainly to determine to whatextent the outputs met the specifications. Our evaluation, whichintegrates three methods, has shown that this approach has animportant limitation: the results for individual constraints arenot always a good indicator for the quality of the overall solution.This has various explanations. First, some constraints may havebeen violated deliberately to enable good results for other con-straints, e.g., by allowing (slightly) more displacement to avoidoverlap. Second, as was observed in the automated constraint-based evaluation of interactively generalized data, one should

320 J. Stoter et al. / Computers, Environment and Urban Systems 33 (2009) 311–324

assess not only if a constraint was violated but also if the violationyields an unacceptable cartographic conflict. Third, very good re-sults for one specific constraint (e.g., minimal distance betweenbuildings) may coincide with bad results for another constraint(e.g., building density should be kept). Fourth, a non-satisfied con-straint can be due to missing functionality in a system, but can justas well be due to imprecise constraint definition. And finally, asHarrie and Weibel (2007) observed, results of constraint-basedevaluation heavily depend on the defined test cases: is the con-straint set complete and evenly balanced, or does it contain manyconstraints for very specific situations (as in the OSGB case)?Therefore, future research should aim to:

(a) revise the threshold values of constraints copied from mapspecifications, because their use differs in interactive andautomated processes. This would require introducing thenotion of flexibility in the formalization and evaluation ofconstraints for automated processes.

(b) evaluate the legibility constraints that account for this flex-ibility as a satisfaction range between 0 and 1, instead of aBoolean outcome. Boolean values may more appropriate toidentify cartographic errors. They may, however, be lessappropriate for assessing the evaluation output, becausethey do not provide information on the degree to whichthe threshold is ignored.

(c) improve operators (algorithms) in generalization systems byapplying the notion of satisfaction ranges.

(d) validate the constraint approach by considering how toaggregate ‘‘constraint-by-constraint” assessments for globalindicators of map quality, specifically by better understand-ing their interdependencies and impact. This also raisesquestions on the domain of constraint satisfaction and viola-tion values and on their weighting and prioritizing tomake different constraints comparable and to enable aggre-gating them to global indicators. These issues have previ-ously been addressed in the domain of constraint-basedoptimization (see Bard, 2004; Ruas, 1998, and Mackaness& Ruas, 2007).

5.4. Improving the constraints

In addition to our recommendation to incorporate the notion ofparameter value flexibility in improved versions of the constraints,our results suggest three specific recommendations for improvingthe constraint-based definition of map specifications. First, theconstraints should be as formal as possible to support the general-ization process as well as the automated-constraint-based evalua-tion. This implies that general concepts, such as shape, pattern, andurban and settlement structures, should be described formally.Second, constraints that were missing as observed from the out-puts should be added. Finally, constraints that appeared to be un-clear need refinement to distinguish, e.g., cartographic conflictsfrom acceptable solutions (compare Fig. 6a with Fig. 6b and c). Cur-rently constraints are usually defined for geometric or thematicproperties. Improvements could come from cognitive science.

5.5. Evaluating generalization software beyond constraints

Our study concentrated on the question of whether commer-cially available solutions could meet the map specifications ofNMAs defined as constraints. However, during our tests severalother aspects were encountered that are also relevant for assessingcommercial generalization systems. For example, our testers found

that in some cases topological errors were introduced during thegeneralization process, and that links between generalized andungeneralized objects, required for automated evaluation, werelost in most of the outputs. Also conflict detection tools are miss-ing. These aspects should be addressed in future tests.

Furthermore the tests highlighted difficulty to parameterize thecomplex algorithms and the lack of default tools, for instance de-fault algorithm sequences or default constraints. Appropriate toolsto optimally parameterize existing algorithms for a specific testcase would highly improve the applicability of commercial soft-ware for a specific test case. Therefore a next research could ad-dress parameterization possibilities.

In addition, a future test should address aspects not amenableto constraints. The constraint approach is based on the conse-quences of scale changes. According to Mackaness and Ruas(2007), this bottom-up approach might work better for small-scalechanges. In contrast, a top-down approach that meets the conse-quences of (large-) scale reduction by choosing appropriate repre-sentations for phenomena might work better over larger scalechanges where changes are much more fundamental. A future testcan provide more insights into the appropriateness of both ap-proaches for automated map generalization. Indeed, it appearedthat constraints on the final result are sometimes not sufficientto fully express without ambiguity what is expected. In some cases,specifying the expected transformation can help if this transforma-tion is always the same and if it is well known. However fuzzy andincomplete constraints resulted in very different interpretationsand solutions among the testers, which may ask for a different ap-proach in defining the requirements for automated generalization.Furthermore, because the limited sizes of the four test cases pre-cluded addressing the problems of dealing with large amounts ofdata (computational complexity, potential memory overflows thatnecessitate data partitioning, presence of numerous and variousparticular cases that make some algorithms fail, etc.), future testsshould define criteria as well as measuring tools to assess scalabil-ity of systems.

And finally, future tests should quantify customization possibil-ities. The most realistic way to address NMA specific requirementsmay be to customize existing software. This requires facilities forwriting extensions or for allowing integration with other systems.

In conclusion, our comprehensive study and new methodologyare a significant contribution to generalization research, specifi-cally to better defining map specifications and evaluating general-ized maps. Future generalization research can extend ourmethodology and make use of our findings, applying improved ver-sions of the constraint sets and re-using our carefully sourced gen-eralization test cases.

Acknowledgements

We would like to thank all participants of the EuroSDR research,in particular we thank Peter Rosenstand (formerly KMS, Denmark),Karl-Heinrich Anders (formerly University of Hannover, Germany),Xiang Zhang (ITC, Enschede), Maarten Storm (formerly Kadaster),Annemarie Dortland and Harry Uitermark (Kadaster, The Nether-lands), Magali Valdeperez, Francisco Martínez and Francisco Dávila(IGN Spain). Also we express our gratitude to our colleagues whojoined us in carrying out the tests: Patrick Revell, Stuart Thom,Sheng Zhou (Ordnance Survey), Willy Kock (ITC), and Patrick Tai-llandier (IGN, France). We are very grateful to the vendors partici-pating in this research for their very important contributions. Wealso thank the anonymous referees who reviewed an earlier sub-mission and gave insightful suggestions for its revision andrestructuring.

Appendix A. Appendix Harmonized constraints for one object

GENERIC-constraint ID

Constraint type Geometrytype

Class Condition for object beingconcerned with thisconstraint

Constrained property Conditiondepends oninitial value?

Condition to be respected Action Importance of constraint(1–5, 1 is less important)

EuroSDR-1-1 Minimal dimensions Polygon Area No Target area >x map mm2 IF finalarea <x mapmm2

THEN {action}EuroSDR-1-2 Minimal dimensions Polygon Width of any part No Target width >x map mmEuroSDR-1-3 Minimal dimensions Polygon Initial area > <= x map mm2 Area Yes Target area = initial area ± x%EuroSDR-1-4 Minimal dimensions Polygon Polygon contains a hole Area of any hole in

a polygonNo Target area of hole >x mm2

EuroSDR-1-5 Minimal dimensions Line/polygon Length of an edge/line No Target length >x map mmEuroSDR-1-6 Minimal dimensions Line/(polyline) Width No Target width >x map mmEuroSDR-1-7 Minimal dimensions Line Vertices density No Target vertices distance >x map mmEuroSDR-1-8 Minimal dimensions Polygon Width of protrusion/recess No Target width >x map mmEuroSDR-1-9 Minimal dimensions Polygon Depth of protrusion/recess No Target depth >x map mmEuroSDR-1-10 Minimal dimensions Polygon Area of protrusion No Target area >x map mm2

EuroSDR-1-11 Shape Any General shape Yes Target shape should be similar toinitial shape

EuroSDR-1-12 Shape Any 1:n Relation(amalgamation)

General shape Yes Target shape should be similar toinitial shape

EuroSDR-1-13 Shape Polygon Initial value of angle = 90�(tolerance = ±x�)

Squareness Yes Target angles = 90�

EuroSDR-1-14 Shape Polygon Initially high concavity Concavity Yes Target shape remains concaveEuroSDR-1-15 Shape Polygon Elongation Yes Target elongation = initial

elongation ±x%EuroSDR-1-16 Topology Line and

polygonInitially, no self-intersection Intersection Yes No self-intersection must be created

EuroSDR-1-17 Topology Line andPolygon

Coalescence No Coalescence must be avoided

EuroSDR-1-18 Orientation Any General orientation Yes Target orientation=initialorientation ± x%

EuroSDR-1-19 Position Any Positional accuracy Yes Target absolute position =initial absoluteposition ± x map mm

EuroSDR-1-20 Model generalization Any Class Yes Target class = initial classEuroSDR-1-21 Model generalization Any Symbolization value Yes Target symbolization value = initial

symbolization value

J.Stoteret

al./Computers,Environm

entand

Urban

Systems

33(2009)

311–324

321

GENERIC-Constraint ID

Constrainttype

Geometry typecombination

Class1

Condition for objectin class 1 beingconcerned with thisconstraint

Class2

Condition for object inclass 2 being concernedwith this constraint

Condition on both objects(in the initial data) forthem to be concernedwith this constraint

Constrainedproperty

Condition dependson initial value?

Condition to be respected Action Importance ofconstraint(1–5, 1 is lessimportant)

EuroSDR-2-1 Minimaldimensions

Any–any Minimaldistance

No Target distance >x map mm IFdistance<x mapmmTHEN{action}

EuroSDR-2-2 Minimaldimensions

Polygon–polygon

One class must be insidewithin another class

Minimalarea

No Target area >x map mm2

EuroSDR-2-3 Orientation Line/polygon–line/polygon

Objects are parallel (±x�) Orientation Yes Object (class 1) must beparallel to object (class 2)

EuroSDR-2-4 Topology/position

Any–any Relativeposition

Yes Target relative positions =initial relative positions

EuroSDR-2-5 Topology Line/polygon–line/polygon

Within a single featureclass

Intersection No No other-intersectionsmust be created

EuroSDR-2-6 Topology Line–any Object (class 1) leadsto the object (class 2)

Accessibility Yes Target accessibility = initialaccessibility

EuroSDR-2-7 Topology Line–any Initially connected Connectivity Yes Target connectivity = initialconnectivity

EuroSDR-2-8 Topology Any–any Object (class1)overlaps object (class2)

Object (class2) is underobject (class 1)

Overlapping No Target overlapping = initialoverlapping

EuroSDR-2-9 Topology Any–any Object (class 1)contains object (class2)

Object (class2) is insideobject (class 1)

Topologicalconsistency

Yes Target topology relations =initial topology relations

EuroSDR-2-10 Topology Line/polygon–line/polygon

Minimal distance <x mapmm and objects areparallel ±x�

Adjacency Yes Target objects must beadjacent

EuroSDR-2-11 Topology Line/polygon–line/polygon

Objects are topologicallyadjacent (sharing an edge)

Adjacency Yes Target topology relation =initial topology relation

Appendix B. Appendix Harmonized constraints on two objects

322J.Stoter

etal./Com

puters,Environment

andU

rbanSystem

s33

(2009)311–

324

GENERIC-Constraint ID

Constrainttype

Geometrytype

Class Kind ofgroup

Kind of objects of theinitial datacomposing the group

Condition (in the initialdata) for group beingconcerned with thisconstraint

Constrained property Conditiondepends oninitialvalue?

Condition to be respected (donot forget the units)

Action Importance ofconstraint (1–5,1 is lessimportant)

EuroSDR-3-1 Minimaldimensions

Any Any Any Minimal distance andminimal area

No Distance between objects >xmap mm AND area of eachobject >x map mm2

IF distance <x mapmm AND area <mapmm2 THEN {action}

EuroSDR-3-2 Minimaldimensions

Any Any Any Minimal distance No Distance between objects >xmap mm

IF distance <x mapmm THEN {action}

EuroSDR-3-3 Orientation Point/polygon

Alignments Alignment orientation Yes Target orientation should besimilar to initial orientation

EuroSDR-3-4 Topology Line andpolygon

Any Any Intersection No No other-intersections mustbe created

EuroSDR-3-5 Topology Line andpolygon

Any Any Connectivity Yes Connectivity must remain

EuroSDR-3-6 Shape Any Shape Yes Target shape should besimilar to initial shape

EuroSDR-3-7 Shape Polygon Buildingalignment

Buildings aligned Spatial distribution Yes Target distribution should besimilar to initial distribution

EuroSDR-3-8 Shape Polygon Urbanblocks

Buildings surroundedby minimal cycle ofroads (in urban areas)

Spatial distribution Yes Target distribution should besimilar to initial distribution

EuroSDR-3-9 Shape Line Contourlines

Relief form Contour lines thatcompose a relief form(e.g., riff, valley)

Spatial distribution ofcontour lines

Yes Target distribution of contourlines should preserve therelief form

EuroSDR-3-10 Shape Polygon Object inter-distance <xmap mm

Shape Yes The shape of derived group ofobjects should be similar tothe shape of the initial group

EuroSDR-3-11 Shape Point/polygon

Alignments Alignment Yes Alignment should be kept

EuroSDR-3-12 Distribution/statistics

Polygon Urbanblocks

Buildings surroundedby minimal cycle ofroads (in urban areas)

Distribution ofcharacteristics ofbuildings (shape, size,function. . .)

Yes Target distribution should besimilar to initial distribution

EuroSDR-3-13 Distribution/statistics

Polygon Urbanblocks

Buildings surroundedby minimal cycle ofroads (in urban areas)

Density of buildings(black/white ratio)

Yes Target density should beequal to initial density ±x%

Appendix C. Appendix Harmonized constraints for group of objects

J.Stoteret

al./Computers,Environm

entand

Urban

Systems

33(2009)

311–324

323

324 J. Stoter et al. / Computers, Environment and Urban Systems 33 (2009) 311–324

References

Barrault, M., Regnauld, N., Duchêne, C., Haire, K., Baeijs, C., Demazeau, Y., et al.(2001). Integrating multi-agent, object-oriented, and algorithmic techniques forimproved automated map generalisation. In Proceedings of the 20thinternational cartographic conference (ICC 2001) (pp. 2110–2116), 6–10August 2001, Beijing, China. CD-ROM.

Bard, S. (2004). Quality assessment of cartographic generalisation. Transaction inGIS, 8(1), 63–81.

Beard, M. K. (1991). Constraints on rule formation. In B. P. Buttenfield & R. B.McMaster (Eds.), Map generalisation: Making rules for knowledge representation,Longman group (pp. 121–135). London: Longman. ISBN: 0-582-08062-2.

Brewer, C. A., & Buttenfield, B. P. (2007). Framing guidelines for multi-scale mapdesign using databases at multiple resolutions. Cartography and GeographicInformation Science, 34(1), 3–15.

Burghardt, D., Neun, M. (2006). Automated sequencing of generalisation servicesbased on collaborative filtering. In: M. Raubal, H. J. Miller, A. U. Frank, M.Goodchild (Eds.), Geographic information science. 4th international conference(pp. 41–46), GIScience 2006, IfGIprints 28. ISBN 9-783936-616255.

Burghardt, D., Schmidt, S., Stoter, J. E. (2007). Investigations on cartographicconstraint formalisation. In 10th ICA Workshop of ICA commission ongeneralisation and multiple representation, August 2–3, Moscow, Russia.

Burghardt, D., Schmid, S., Duchêne, C., Stoter, J., Baella, B., Regnauld, N., et al. (2008).Methodologies for the evaluation of generalised data derived with commercialavailable generalisation systems. In 11th ICA workshop of ICA commission ongeneralisation and multiple representation, 20–21 June 2008, Montpellier.<http://aci.ign.fr/BDpubli/moscow2007/Burghardt-ICAWorkshop.pdf>(accessed 3.11.08).

Buttenfield, B. P. (1991). A rule for describing line feature geometry. In B. P.Buttenfield & R. B. McMaster (Eds.), Map generalization: Making rules forknowledge representation (pp. 150–171). Longman.

Foerster, T., Stoter, J. E., Kraak, M. -J. (2009). Challenges for automatedgeneralisation at European mapping agencies. The Cartographic Journal,submitted for publication.

Harrie, L. (2001). An optimisation approach to cartographic generalisation. Ph.D.Thesis, Department of Technology and Society, Lund University.

Harrie, L., & Weibel, R. (2007). Modelling the overall process of generalisation. In W.A. Mackaness, A. Ruas, & L. T. Sarjakoski (Eds.), Chapter 4 generalisation ofgeographic information: Cartographic modelling and applications (pp. 67–88).Elsevier. ISBN 978-0-08-045374-3.

Hubert, F., Ruas, A. (2003). A method based on samples to capture user needs forgeneralisation. In 5th ICA workshop on progress in automated mapgeneralisation, Paris, 2003. <http://www.aci.ign.fr/BDpubli/paris2003/papers/hubert_et_al_v0.pdf> (accessed 3.11.08).

Kilpela, T. (2000). Knowledge acquisition for generalisation rules. Cartography andGeographic Information Science, 27(1), 41–50.

Leitner, M., & Buttenfield, B. (1995). Acquisition of procedural cartographicknowledge by reverse engineering. Cartography and Geographic InformationSystems, 22(3), 232–241.

McMaster, R. B., Shea, K. S. (1988). Cartographic generalisation in a digitalenvironment: A framework for implementation in a geographic informationsystem. In GIS/LIS proceedings (pp. 240–249), San Antonio, TX.

McMaster, R. B. (1995). Knowledge acquisition for cartographic generalization. In J.C. Mueller, J. P. Lagrange, & R. Weibel (Eds.), GIS and generalization: Methodologyand practice (pp. 161–179). London, UK: Taylor & Francis.

Mackaness, W. A., Ruas, A., Sarjakoski, L. T. (2007). Generalisation of geographicinformation: Cartographic modelling and applications. Series of internationalcartographic association, Elsevier. ISBN 978-0-08-045374-3.

Mackaness, W. A., & Ruas, A. (2007). Evaluation in map generalisation process. In W.A. Mackaness, A. Ruas, & L. T. Sarjakoski (Eds.), Chapter 5 in generalisation ofgeographic information: Cartographic modelling and applications (pp. 89–112).Elsevier. ISBN 978-0-08-045374-3.

Müller, J. C., Mouwes, P. J. (1990). Knowledge acquisition and representation for rulebased map generalisation: An example from the Netherlands. In GIS/LISproceedings 90 (Vol. 1, pp. 58–67), Anaheim, California.

Mustière, S. (2005). Cartographic generalization of roads in a local and adaptiveapproach: A knowledge acquisition problem. International Journal ofGeographical Information Science, 19(8–9), 937–955.

Mustière, S. (2001). Apprentissage supervisé pour la généralisation cartographique.Thèse de doctorat, Université Paris VI, France 2001.

Nickerson, B. G. (1991). Knowledge engineering for generalization. In B. Buttenfield& R. B. McMaster (Eds.), Map generalization: Making rules for knowledgerepresentation (pp. 40–55). London: Longman.

OpenJump. (2008). OpenJUMP – The free, Java based and open source geographicinformation system for the world. <http://www.openjump.org/wiki/show/HomePage> (accessed 28.10.08).

Plazanet, C., Bigolin, N., & Ruas, A. (1998). Experiments with learning techniques forspatial model enrichment and line generalization. Geoinformatica, 2(4),315–333.

Reichenbacher, T. (1995) Knowledge acquisition in map generalization usinginteractive systems and machine learning. In Proceedings of the 17thinternational cartographic conference (pp. 2221–2230). Barcelona, Spain.

Rieger, M. K., & Coulson, M. R. C. (1993). Consensus or confusion: Cartographers’knowledge of generalization. Cartographica, 30(2–3), 69–80.

Ruas, A. (1998). OO-constraint modelling to automate urban generalization process.In Proceedings of the eight international symposium on spatial data handling(pp. 225–35), Vancouver, Canada, July 12–15.

Ruas, A. (1999). Modèle de généralisation de données géographiques à base decontraintes et d’autonomie. Doctoral Thesis, Université de Marne-la-Vallée.

Ruas, A. (2001). Automatic generalisation research: Learning process frominteractive generalisation, OEEPE, Report no. 39.

Sarjakoski, L. T. (2007). Conceptual models of generalisation and multiplerepresentation. In W. A. Mackaness, A. Ruas, & L. T. Sarjakoski (Eds.). Chapter2 of generalisation of geographic information: Cartographic modelling andapplications, series of international cartographic association (pp. 11–35). Elsevier.

Sester, M. (2000). Generalisation based on least-squares adjustment. In the XIXthinternational congress, commission IV, international archives of photo-grammetry and remote sensing (pp. 931–938), Amsterdam, The Netherlands.

Stoter, J. E (2005). Generalisation: The gap between research and practice. InProceedings of the 8th ICA workshop on generalisation and multiplerepresentation, 7–8 July, 2005, A Coruña, Spain, 10 pages. <http://www.aci.ign.fr/Acoruna/Papers/Stoter.pdf> (accessed 3.11.08).

Ware, J. M., Jones, C. B., & Thomas, N. (2003). Automated map generalisation withmultiple operators: A simulated annealing approach. International Journal ofGeographical Information Science, 17(8), 743–769.

Weibel, R. (1991). Amplified intelligence and rule-based systems. In B. Buttenfield &R. B. McMaster (Eds.), Map generalization: Making rules for knowledgerepresentation (pp. 172–186). Longman.

Weibel, R. (1995). Three essential building blocks for automated generalization. In J.Mueller, J. P. Lagrange, & R. Weibel (Eds.), GIS and generalization: Methodologyand practice (pp. 56–70). London: Taylor & Francis.

Weibel, R., Keller, S., & Reichenbacher, T. (1995). Overcoming the knowledgeacquisition bottleneck in map generalization: The role of interactive systemsand computational intelligence. In A. U. Frank & W. Kuhn (Eds.), Spatialinformation theory – A theoretical basis for GIS (COSIT’95) (pp. 139–156). Berlin,Heidelberg: Springer.