Page 1 of 12

An Urban Growth Strategy using Multi-Criteria Analysis – Fact or Fiction?

Philip J Hawkes 2014

Abstract

Summerland is a small town in British Columbia, facing growing pains. In 2013 the municipal

government decided to engage the public in a process that included workshops and surveys, to

determine a growth strategy for the community. The Project was called ‘Future Summerland’ which

concluded with the publication of the Summerland Urban Growth Strategy Final Report, in November,

2013. The results identified the top five areas to develop, which can be seen in the tables below.

Purely quantitative analysis indicated that the top five areas for potential development are:

1 Barkwill-Cook Area (currently ALR)

2 Aeneus Creek to Blair St Bluff Area (currently ALR)

3 Quinpool to Aeneus Crk Area (currently ALR)

4 Downtown Infill

5 Mayne St Area

As indicated by in-person engagement and survey responses, the majority of public preference for future growth areas are:

1 Infill in the Town Core, Lower Town and Trout Crk

2 Quinpool to Aeneus Creek area (currently ALR)

3 Deer Ridge

4 Aeneus Creek to Blair Street area (currently ALR)

5 Victoria Road South

The municipal government has seen this as a mandate to amend the Official Community Plan to include

farmland which is currently in the ALR (top 3 choices in the quantitative table highlighted in yellow).

Since the report was published there have been claims of misrepresentation by workshop participants,

and an ongoing petition, signed by those opposed to using ALR farmland, is approaching 3000. This

paper provides a critical analysis of the Summerland Urban Growth Strategy method and final report.

The paper raises concerns over the use of multi-criteria analysis and the subjective selection of survey

results. The paper findings indicate that the analysis and conclusions are fundamentally flawed leading

to evidence of misrepresentation.

Introduction

Summerland is a small town in British Columbia, with a population of 11,2801 and like many towns of

this size it faces the challenges of growing to support infrastructure investment. The town has a strong

connection to agriculture with many orchards and several vineyards. It is also home to the Pacific Agri-

Food Research Centre. Summerland’s municipal government is proposing to amend its Official

Community Plan (OCP) to develop a more compact community citing sustainability and infrastructure

costs as the main motivators.

During 2013 the municipal government embarked on the Future Summerland Project, which was

designed to engage the community through workshops and surveys to create a new vision.

1. Statistics Canada (2011) http://www12.statcan.gc.ca/census-recensement/2011/as-sa/fogs-spg/Facts-csd-

eng.cfm?LANG=Eng&GK=CSD&GC=5907035 (accessed 10 April 2014)

Page 2 of 12

The resultant Summerland Urban Growth Strategy (SUGS) Final Report dated Nov 20132 was published.

It has proven controversial as the multi-criteria analysis identified some areas for development that are

farm land and presently protected by the Agricultural Land Reserve (ALR) Act. Based on the SUGS report

the OCP amendment (Bylaw 2014-002) recommends the removal of the top three areas identified by

quantitative multi-criteria analysis. This paper will attempt to critique the way multi-criteria analysis was

used in the SUGS report.

Multi-Criteria Analysis

The SUGS final report uses Multi-Criteria Analysis (MCA) both quantitative and qualitative to determine

which areas in Summerland are most appropriate for future development.

The United Nations describes MCA as a structured approach used to determine overall preferences

among alternative options, where the options accomplish several objectives. In MCA, desirable

objectives are specified and corresponding attributes or indicators are identified (UN, 2014).

However, caution is required as the tool is subjective (EU, 2014).

Multi-Criteria Analysis is highly subjective and potentially open to abuse through the influence of special interests. Moreover, it is seriously flawed conceptually and mathematically. At best, its continued use by public sector policy advisers and decision-makers is due to ignorance. At worst, it represents gross negligence in the provision of advice……. (Dobes and Bennett, 2010).

MCA Quantitative analysis

First this paper will consider the quantitative analysis as it also forms the basis for the qualitative

analysis.

Thirteen areas were analyzed (Appendix A) using a ten criteria model shown in Figure 1. Purely

quantitative analysis indicated that the top five areas for potential development are:

1. Barkwill-Cook Area (currently ALR)

2. Aeneus Creek to Blair St. Bluff area (currently ALR)

3. Quinpool to Aeneus Crk Area (currently ALR)

4. Downtown Infill

5. Mayne St. Area

2. The Summerland Urban Growth Strategy Final Report was produced by Sustainable Solutions Group and can be viewed here

http://futuresummerland.com/ (accessed 9 April, 2014)

Page 3 of 12

The SUGS final report (page 12) states that - Ten criteria were chosen to quantitatively and objectively

analyze the 13 areas in a development impact assessment model constructed by the consulting team.

Assumptions and numbers behind each criteria can be found in Appendix 4.

According to the SUGS final report the multi-coloured bars shown in Figure 2 indicate the sum of the

relative criteria scores, with the longer the bars presenting themselves as the more suitable areas for

development.

The first criteria Preserving ALR Lands is given the value of ‘yes’ or ‘no’ in Appendix 4. There is no

quantitative data to support this criteria. It has been attributed an initial value of 10, which did not

change throughout the quantitative analysis. There has been no attempt to quantify this value with for

example greenhouse gasses (GHGs) so it can be compared with the other criteria. The analysis is

therefore not quantitative as the SUGS report claims.

Figure 1: The 10 Criteria - Summerland Urban Growth Strategy Final Report (Page 15)

Quantitative research attempts precise measurement of something;" it determines facts and figures (Cooper & Schindler, 2006, Ch. 8, p. 3)

Page 4 of 12

Figure 2: MCA Quantitative - 13 Potential Growth Areas - Summerland Urban Growth Strategy Final Report

(Page 15)

Another obvious problem with the analysis is that the ALR land has been unfairly weighted with high

density housing (criteria 2 & 3 - red and orange bars derived from density and dwelling types assumed

appendix 4 page 77). This is at odds with the SUGS report advice as twice in the report it states….

Although certain areas may accommodate denser housing types (e.g.: multi-unit residential buildings), comparing areas with single family housing to others with denser housing types is an inaccurate method of comparison. In order to fairly compare areas, like must be compared with like. Page 14 (and on Appendix 3 page 11).

The high density housing issue also appears to affect the decreasing GHG emissions criteria weighting

(Appendix 4 page 78). In the same vein, and for comparison, it is curious as to why the Downtown infill

has been assigned single family dwellings (Appendix 4 page 77), while the proposed Wharton St. multi-

residential building (500 units)3 is seemingly ignored.

3. Wharton Street Project; http://www.summerland.ca/Local%20Projects/Wharton_street/New_Future_development_%20proposal.htm

(accessed 10 April, 2014)

Page 5 of 12

MCA Qualitative analysis

The SUGS final report states (page 16) - ‘Recognizing that planning decisions are rarely made on factual

evidence alone – there are almost always qualitative factors that affect the outcomes – a MCA tool was

incorporated into the model so that user-defined weighting could be assigned to each criteria, thus

making some criteria more important than others.’

In the previous section we question the validity of the quantitative model and the ‘factual evidence’ and

so it is difficult to move forward with a qualitative critique. Questions regarding the quantitative model

have been submitted to staff at the District of Summerland and to Sustainable Solutions Group, but to

date no sensible answers have been offered.

A Freedom of Information request has also been sent to the District of Summerland for all the survey

and workshop questions and answers. This is due to the information presented in the report being

unclear and insufficient for the author to carry out a critical analysis using the qualitative data. However,

Survey #2 results have been made available to the author from another FOI request by a Summerland

resident who took part in the workshops.

One example why some of those taking part felt misrepresented can be seen in Survey # 2 (Appendix B)

where the following two seemingly straight forward questions were asked:

1. Which areas should be designated for future growth?

2. Which areas should not be designated for future growth?

The results are shown in Table 1: Survey 2 (question 5 of 5), which suggests that the respondents did not

want to designate ALR land for development.

Table 1: Survey #2 Question 5 of 5

Order of preference

Should be designated for growth Should NOT be designated for growth

1 Downtown, lower Town & Trout Creek infill Switchback Rd. Area

2 Mayne St. Area Hunters Hill

3 Hespeler Rd. Area Trout Creek Shore

4 Deer Ridge Thornber St. Area

5 Victoria Rd. South Barkwill-Cook Area

5 Quinpool to Aeneus Creek (ALR) Deer Ridge

7 Mountford-Cedar Area (ALR) Aeneus Creek to Blair St.

7 Aeneus Creek to Blair St. (ALR) Mountford-Cedar Area

9 Thornber St. Area (ALR) Victoria Rd. South

10 Barkwill-Cook Area (ALR) Quinpool to Aeneus Creek

11 Trout Creek Shore Hespeler Rd. Area

12 Hunters Hill Mayne St. Area

13 Switchback Rd. Area Downtown, lower Town & Trout Creek infill

Page 6 of 12

Also, in Survey # 2, Question 1 of 5 asks: Considering future growth in Summerland, how important is

each individual modelling criteria? The result can be seen in Table 2 below. The results show that

preserving ALR land is important (3rd), and yet criteria relating to high density housing is least important

(9th and 10th). This is contrary to the outcome, as the OCP amendment is - based on the earlier

quantitative analysis - proposing to build high density housing on existing ALR land.

Table 2: Survey #2 Question 1 of 5: Considering Future growth in Summerland, how important is each

individual modelling criteria?

Order of importance

Criteria Rating Average Response Count

1 Respecting Eco System 8.37 153

2 Increase opportunities for walking 7.61 153

3 Preserving ALR land 7.59 153

4 Keep water and sewer costs low 7.32 153

5 Keep house hold transport and energy costs low 7.15 153

6 Keep costs to society low 7.09 153

7 Decreasing GHG emissions 7.07 153

8 Decreasing Travel by Car 6.6 153

9 Increasing housing density on lots 5.69 153

10 Developing as many homes as possible on lots 4.84 153

‘I took part in the workshops and I believe the residents of Summerland have been misrepresented by this process and final report.’ – Erin Carlson (Summerland resident)

The preferred areas for growth identified in the conclusion based on ‘survey respondents’ is derived

from Survey #2 Question 4, which asks: What are the top 3 areas to locate future development in

Summerland? The answers can be seen in Appendix C and Table 3. Note that the 2nd and 3rd choices are

given the same weighting in the chart (appendix C) as the 1st choices. Is this fair?

Table 3: Qualitative Top 5 results based on Survey # 2 Question 4 (only)

Order of Preference Areas to Develop

1 Infill in the Town Core, Lower town and Trout Creek

2 Quinpool to Aeneus Creek Area

3 Deer Ridge

4 Aeneus Creek to Blair St. Area

5 Victoria R. South

If the authors of the SUGS report had selected Survey #2 Question 5 then the results would be different.

Which begs the question have the results been cherry picked?

Page 7 of 12

An analysis and report that leaves so many questions and lacks transparency is something local

governments need to be wary of.

BC Local Government Act - OCP Process and Consultation [As the OCP could] become the subject of a court challenge, local governments may want to take particular care to ensure that their decision-making process with respect to this consultation is transparent. (BC Gov 2000)

While Summerland local government officials and staff celebrate the success of this extensive public

consultation process (Appendix D), in reality the OCP amendment suggests that the public were

completely ignored. Even the questionable qualitative conclusion was disregarded in favour of the MCA

quantitative results; which have now formed the basis of the OCP amendment, and proposed ALR

exclusion application.

“This is the most comprehensive public engagement exercise ever undertaken within our community and on a very important issue” says Mayor Janice Perrino. (Civic Info BC, 2013)

Conclusion

During 2013 the community embarked on the Future Summerland Project, which was designed to

engage the community through workshops and surveys to create a new vision for growth. The resultant

Summerland Urban Growth Strategy (SUGS) Final Report dated Nov 20132 was produced. It has proven

to be very controversial with claims of misrepresentation. The SUGS Final Report uses MCA to identify

the top five areas for potential residential development. The quantitative model is built on a subjective

foundation and therefore fundamentally flawed. Multi criteria analysis has been the subject of criticism

in academic papers as it can be highly subjective and open to abuse.

Misrepresentation of public opinion constructs, rather than reflects true public opinion…! – Professor Justin Lewis (MediaEd, 2014)

The SUGS report offers two top five areas for potential development (shown in the Abstract above): one

is a table based on a cherry picked survey question, and the other is based on flawed quantitative

modelling analysis. The latter was selected to support the OCP amendment, and public opinion was

completely ignored.

Page 8 of 12

It is difficult to comprehend how a local government can legitimately and morally introduce a bylaw

amending the OCP and Summerland’s future, based on such a report.

This paper recommends that Summerland set up an independent committee to investigate this further.

References

BC Gov, 2000, OCP Process and Consultation Bulletin No.: G.2.1.0

http://www.cscd.gov.bc.ca/lgd/intergov_relations/planning_bulletins/bulletinG210.htm (accessed 10

April, 2014)

Civic Info BC, 2013 Summerland Completes a Year-Long Urban Growth Strategy Review,

http://www.civicinfo.bc.ca/302n.asp?newsid=5503&r=5503&r=5502&r=5501 (accessed 10 April, 2014)

Cooper & Schindler, 2006 Cooper, D. R., & Schindler, P. S. Business Research Methods (th ed.). New

York, NY: McGraw-Hill.

Dobes and Bennet, 2010, L. Dobes, J. Bennet Multi-criteria analysis: Ignorance or negligence?

Proceedings of Australian Transport Research Forum 29 September-1 October, Canberra, Australia,

Australian Transport Research Forum (2010)

EU 2014,Multi-Criteria Analysis

http://ec.europa.eu/europeaid/evaluation/methodology/examples/too_cri_res_en.pdf (accessed 9

April, 2014)

MediaEd 2014, Justin Lewis, How Politicians and the Media Misrepresent the Public

http://www.mediaed.org/assets/products/106/studyguide_106.pdf (accessed 10 April, 2014)

Page 9 of 12

Appendix A

Page 10 of 12

Appendix B

Page 11 of 12

Appendix C

Page 12 of 12

Appendix D

COMMITTEE OF THE WHOLE

MEETING AGENDA

Monday, December 9th, 2013 - 8:30 a.m.

Council Chambers