1 a core course on modeling contents the need for interpretation approach criteria for...

1

A Core Course on Modeling

Contents • The Need for Interpretation

• Approach

• Criteria for Modeling

•Genericity•Scalability•Specialization•Population•Convincingness•Distinctiveness •Surprise •Impact

• Criteria for modeling and purposes

• Summary

• References to lecture notes + book

• References to quiz-questions and homework assignments (lecture notes)

Week 7-A working model – and then?

2


The Need for Interpretation


Model makers in 19th century London predicted 3 meter rising of horse manure levels by 1950

Model makers in 21st century predict 0.5 meter rising of sea levels by 2050

A model outcome in itself cannot solve a problem.

Interpretation is necessary final step.

How to achieve a good-quality interpretation?

3


Approach


We want to say that one model (including the interpretation of the outcomes) is better than another.

‘Better’ means: with respect to fulfilling the model’s purpose.

Therefore we need criteria to compare.

The first step to quality assessment is the ability to compare things that are utterly different. Comparison invariably needs criteria.

4


Approach


So: we will try to formulate criteria for comparing problem solutions.

Problem solution =

model outcome + interpretation

can be assessed (compared) with the tools of Chapter 6 (sensitivity analysis, uncertainty analysis)

As yet, we have no means to compare ‘quality’ of interpretations

5


Criteria for Modeling


When formulating criteria for comparing problem solutions, the main problem is the immense variety of problems asking for modeling.

Our criteria need to apply to all of them.

The large variety of possible modeling processes is a complication for finding a single generic set of criteria.

6




The standard approach to finding a complete set of ‘things’, is to construct a complete taxonomy.

A complete taxonomy requires enumerable properties: properties where all possible values can be enumerated.

The quarto-game is based on 4 enumerable properties: color(black,white), size(tall,short), shape(round,square), interior(hollow,solid). Rows of 4 must be formed of pieces which all have a common value for one property

7




Property 1:

Does the criterion regard the begin (definition stage of the modeling process) or the end (conclusion stage)?

The input-side of a problem relates to stage 1 (problem definition); the output-side relates to stage 5 (conclusion: presentation and interpretation). This photo seems to suggest that input and output are relative notions …

•Example of criterion related to definition: what is the scale of the problem? (scalability)

•Example of criterion related to conclusion: how much depends on it? (impact)

8




Property 2:

Does the criterion regard the modeled system or the (stakeholders + context) ?

In this famous Velasquez painting, the stakeholders and the modeled system are deliberately confused. In modeling practice, the distinction between subject (the modeled system) and object (the stakeholder(s)) are clearly distinct. Hopefully.

stakeholder2: the modeler stakeholder1:

the customer

the model: the present painting

the modeled system: part of the world,

including the model and some stakeholders

•Example of criterion related to modeled system: what is the scale of the problem? (scalability)

•Example of criterion related to stakeholders: how much knowledge do the stakeholders need? (specialization)

9




Property 3:

Does the criterion regard qualitative or quantitative aspects ?

The distinction quantitative – qualitative is sometimes associated to the closed – open distinction. ‘Closed’ means that all possible outcomes can be foreseen or enumerated; ‘open’ means that this is not possible.

•Example of criterion related to quantitative aspect: what is the scale of the problem? (scalability)

•Example of criterion related to qualitative aspect: how different can modeled systems be? (genericity)

10




Combinations

The 3 distinctions together form a imaginary cube, each corner corresponds to one of the criteria.

impactquantitativestakeholdersconclude

surprisequalitativestakeholdersconclude

distinctive-ness

quantitativemodeled system

conclude

convin-cingness

qualitativemodeled system

conclude

populationquantitativestakeholdersdefine

speciali-zation

qualitativestakeholdersdefine

scalabilityquantitativemodeled system

define

genericityqualitativemodeled system

define

criterionqualitative / quantitative

modeled system / stakeholders

define/ conclude

11


Genericity


Genericity: to which extent is the approach capable to handle various types of modeled systems / purposes?

Examples:

Physics: involve more effects, principles and interactions (flow, continuum mechanics)

Mathematics: abstraction

Computer Science: generic tools (graphs, databases, compilers, XML, ...)

12


Genericity


Genericity: example

Given an object of some shape and density ; we want to know its mass.

1.If it is a cube with side p: m=a3

2.if it is a rectangular block with height h: m=h*Areatop

3.if it is a truncated pyramid: m= h*(Areatop + (Areatop*Areabottom) + Areabottom)/3 (works for all these shapes!)

Above models are increasingly generic; each can deal with the simpler cases as

well.

How heavy is the Borobodur? (E.g., in the context of risks for landslides etc.: the Borobodur is one of the heaviest temples in the world, restauration in the 1980-ies required partial reinforcement of the hill on which it stands to prevent landslides).

Only formula (‘model’) 3 is sufficiently generic

13


Scalability


Scalability: to which extent can some characteristic dimensions of the problem increase, where the model still functions?

Scale: n (=number of elements, states, records, time steps, …)

Performance:

O(1) – solving effort (SE) does not depend on n

O(log n) – SE hardly depends on n

O(n) – SE proportional to n

O(n log n) – SE slightly worse than proportional

O(np) – SE polynomial

O(2n), O(n!), … – SE worse than polynomial

Many problems contain some characteristic dimension (e.g., n=number of boxes). If n grows, there will come a point where the model fails when trying to solving the problem. The larger n can be, the better the scalability of the solution.

14


Scalability




Performance:







The size of Google’s data repository, according to http://www. worldwide websize.com/ , varied between 15 and 55 billion webpages in less than 4 months. Still, the response time stays near constant due to sophisticated indexing strategies

n

O(1)

http://www.worldwidewebsize.com/





15


Scalability




Performance:







To tell if there is a needle in the haystack is easy, provided the hay is sorted: check the halfway element. If this is larger than the needle, discard the upper half, otherwise discard the lower half – and checking the halfway element in the remaining part until either found or not found.

n

O(1)

16


Scalability




Performance:







For many tasks, the effort of doing n things is proportional to n. If one chicken lays three eggs per week, 5 chicken lay 15 eggs per week. This does not hold for everything, though … climbing n flights of stairs may take more effort than n the effort of one. But computers don’t get tired, and many computer tasks scale similarly to chicken laying eggs.

n

O(1)

17


Scalability




Performance:






O(2n), O(n!), … – SE worse than polynomialSorting an array of unsorted things can be done in O(n log n). Military go faster, but they work in parallel.

n

O(1)

18


Scalability




Performance:







Straightfowardly sweeping a matrix takes O(n3), but faster methods exist that bring this down to something like O(n2.5). The difference is noticable only for n sufficiently large.

n

O(1)

19


Scalability




Performance:







The travelling salesman problem asks for the shortest route connecting n places. Exact solution is believed impossible in O(np) but good approximations sometimes take only O(np)

n

O(1)

20


Specialization


Specialization: to which extent does the model / model outcomes requires specialized knowledge on behalf of the problem owner?

How much control should the modeler give the problem owner over

•validity of assumptions,

•regime of application,

•use of results?

A model, intended for specialists has to face the challenge of withstanding expert scrutiny. On the other hand, a model intended for laymen has to be robust against unintended misuse.

21


Specialization



1.Invest in presentation

•graph style (line plot, scatterplot, polar plot, barchart, pie chart, 3D surface, …)

•labels, captions, scales (zero, lin/log, units, major / minor scale lines …)

•how to represent uncertainty?

•which plots to combine in a single graph?

The appropriate visualization style is the answer to the question:

what message should this graph convey?

22


Specialization



2.When in doubt, be conservative

A modeler resembles a blindfolded torreador: without full knowledge, consequences of the modeled system being mischievous may be quite harmful. For good reasons, the jackass adagium reads: ‘don’t try this at home!’

23


Specialization



3.Be warned for biased users

• give a full and detailed account of as much as possible assumptions and estimates

• insist on the report being indivisible

• refrain from easy-to-misapprehend results

• Get a second opinion Users having personal interests in model outcomes may not always take a balanced stand.

24


Specialization



4.Take care for self-fulfilling and self-denying prophecies

• wicked problems: the problem solution post-hoc modifies the problem statement

Both legendary prophet Michael Nostradamus (glass bowl?) and present-day opinion investigators (black ballot box?) relate to the working principle of self-fulfilling and self-denying prophecies.

25


Population


Population: what size of intended audience does the model address?

•Large size: low level of specialization

•Large size: no option for bi-directional communication

•Large size: consider interactive model (example: ‘stemwijzer’)

In the 1999, Wachowski-brothers’ movie ‘the Matrix’, a simulation’s intended audience size is stretched so as to encompass the entire world’s population – with the extra that nearly everybody is ignorant of the fact.

26


Convincingness


Convincingness: a model is more convincing if it contains fewer and/or less implausible assumptions.

•Are assumptions logically deducible from other, less problematic assumptions?

•If not, are there any first-principle ‘laws’ to back up assumptions (example: if this component behaves as a lever, we can use

the physical laws of torque in a lever)?

•If not, can we construct a simplified model system (example: tidal motions as a lumped system) with outcomes that can be empirically verified?

• If not, can we get support from comparison to empirical observations on an existing system (example:

behavior of visitors in shopping malls to predict buying behavior)?

•If not, is there at least any argument for consistency of the assumptions? (example: price elasticity, ‘fun’ in the taxi model)

In the early days of photography, a photo (as prototypical empirical evidence) counted as proof. As this no longer holds, we need to re-think what constitutes ‘plausible evidence’.

27


Convincingness


Convincingness: a model is more convincing if it contains fewer implausible assumptions.

•Convincingness in the absence of ‘laws’:

•conceive a plausible formal model system to which laws do apply (e.g., replace the world + oceans by a homogenous ball covered by a sheet of water). The measure of correspondence between these two is left out of the discussion;

•convincingness relies on the agreement with empirical data.

Tides are caused by subtle interplay between gravity, water (viscosity vs. inertia), and the complex geometry of the earth’s sea floors. A (glass box) model to predict tidal motions contains none of these, but a hypothetical ‘sheet of water on a perfect sphere’ instead.

28


Convincingness


Convincingness: a model is more convincing if it contains fewer implausible assumptions.

•Convincingness in the absence of a formal model system and/or no empirical validation:

•look for an empirical model system

• physical: water tank, wind tunnel

• economical, social: comparative populations, historical survey data

• biological: the guinea pig!

Scale models are physical replica of the modeled system, usually at a smaller scale. Applications include windtunnel (flow, drag) and hydromechanical investigations, but also scale models as used in architecture and urban studies. These are examples of empirical model systems. BTW: J. Swift’s story seems to relate to a scale model; it is actually a social model.

29


Distinctiveness


Distinctiveness: a model has higher distinctiveness if it allows distinction between alternatives that are more similar.

•Most purposes ask for some form of distinctiveness. Examples:

•prediction 1: something is correctly predicted to happen at time T1 rather than T2; how close can T1

and T2 be?

•explanation: Q1 causes P1 and not P2; Q2 causes P2

and not P1. How close can Q1 vs. Q2, or P1 vs. P2 be

so that the explanation holds?

•optimization: input P1 yields output Q1, and input P2

yields output Q2, where Q1 correctly is found to

dominate Q2. How close can P1 and P2 be?

Many modeling purposes boil down to the ability of distinguishing between two alternatives that are as similar as possible. This relates to accuracy and precision, but also to non-numerical purposes.

30


Distinctiveness



This hinges on the notion of ‘distance’, explained in week 6.

The quantitative measure for distinctiveness is usually a similarity measure: the more similar A and A’ whereas the model can successfully distinguish them, the better its distinctiveness.

31


Distinctiveness



Distinctiveness relates to two common types of errors:

•false positive (conclude X where there is no X)

•false negative (don’t conclude X where there is X)

Successful spreading of false money is an example of false negative: the counterfeit goes unnoticed. Coin operated vending machines have to safeguard against both false positives and false negatives; the consequences of the two types of errors being very different.

This is true for most models where distinctiveness plays a role.

32


Surprise


The potential for surprise of a model outcome is the extent to which it may bring unforeseen new ideas.

There are Open and closed spaces of outcomes.

Examples of closed outcomes:

•A model computing the probability of X can only produce a number between 0 and 1.

•A model verifying Y can only produce ´true´ or ´false´.

A widespread opinion is, that computations cannot ‘output more than you put in’. This underestimates computers’ scope of computation, or overestimates humans’ scope of anticipation.

33


Surprise


The potential for surprise of a model outcome is the extent to which it may bring unforeseen new ideas.

Examples of open outcomes:

•result of using ontologies

•outcome of evolutionary algorithms

•outcome of PCA, abstraction, …

Common feature: output space has size O(2n) whereas input space has size O(n)

Evolutionary programming, applied to shape design. Given a comprehensible genotype representation, the space of phenotypes is virtually infinity.

Xai Xu et. al., Special Issue of SIGGRAPH, Vol. 31, No. 4, pp. 57:1-57:10, 2012

34


Impact


The potential for impact of a model is the extent to which the model outcome can affect the stakeholders (either beneficially or adversely).

Two perspectives:

•prestige and profit: the more impact the better

•risk and responsability: the less impact the better

‘Impact’ is a double-edged sword: some modelers attempt to increase the impact of their models whereas other modellers prefer to play safe and seek modeling with less impact.

35


Impact


The potential for impact of a model is the extent to which the model outcome can affect the stakeholders.

Intended impact can be capitalized:

r1=revenues in absence of model outcome;

r2=revenues with model outcome present;

c1=cost of ownership in absence of model

outcome;

c2=cost of ownership with model outcome

present;

=((r2-r1)-(c2-c1))/(|r2-r1|+|c2-c1|)

To assign a quantitative value to a model outcome we must assume some market mechanism where we identify revenues and costs of ownership, both in the absence of the model outcomes and in their presence.

• 0 ||1: magnitude of impact

• If 0 1: positive impact (e.g., models for optimization);

• if -1 0 negative impact (e.g., risk prediction or analysis)

36


Impact



Adverted impact can be quantified or capitalized:

C=the estimated chance (per time interval) of an incident (=an erroneous model outcome, e.g. a type-I or type-II error)

V=the estimated value loss per incident

Both C and V can serve to quantify the negative impact of a model. Their product CV has dimension of money / time and can be added to c2 in the formula for .

37


Impact



The impact of a model should balance with its reliability.

38


Criteria for modeling and purposes


Not each of the eight criteria is equally relevant for all purposes.

Given the purpose of the model, the modeler should identify the most relevant criteria to seek improvement when necessary.

For instance: … (see next sheets)

39





prediction (1, 2): convincingness, distinctiveness, impact

An almanac is a model to serve the purpose of predictions (both 1 and 2). Its impact is rather large (given the long historical record of almanac-use), but the distinctiveness is small, and its convincingness decreases over time.

40





compression: scalability, population, distinctiveness

Printing is a form of compression (transferring written text into printed text) with huge population and distinctiveness; by present day’s standards, the scalability is not impressive

41





inspiration: surprise

All the world’s religious traditions acknowledge holy scriptures of some sort as primary source of inspiration. Their potential for surprise is sometimes rather limited.

42





unification: genericity, convincingness, surprise

In alchemy, the pre-scientific forerunner of both chemistry and psychology, unification of conflicting properties was one of the leading purposes. In modern times, scores on all three criteria are low – when seen through modern eyes.

43





abstraction: distinctiveness, convincingness, surprise

The term ‘abstraction’ in modern art no longer defines a relation between a modeled system and a model; rather, the absence of any modeled system is the defining feature for true abstract art. The potential for surprise, however, sometimes still applies.

44





verification: scalability, convincingness, impact

Biting a golden coin is the proverbial way to verify its authenticity. Scalability and convincingness are probably low, but if the coin contains chocolate, or if the biter has metal inlays, the impact may be quite considerable.

45





exploration: genericity, surprise

Few models for exploration will have a larger potential of surprise than the Mars exploration robot.

46





decision: convincingness, distinctiveness, impact

The Delphi oracle was a decision-making model that derived its convincingness from its deliberately low distinctiveness – thereby being known for its major impact.

47





optimization: genericity, scalability, convincingness, impact

Of all generic optimization models for continuous systems, hill climbing is perhaps one of the most generic ones. It often suffers from scalability, though – even when implemented by modern, automated devices.

48





specification: genericity, distinctiveness

Sometimes the distinction between a specification model (blueprint) and a realization is somewhat blurred.

49





realization: genericity, distinctiveness, impact

Realization …

50





steering & control: distinctiveness

The prototypical stick-and-carrot steering, with the stick tied to the donkey’s back is an example of open-loop control without a set-point.

51


Summary


•Leading question: to what extent has the initial problem been solved?

•Approach: criteria to assess the quality of the modeling process

•Taxonomy: Input or output side? Modeled system or stakeholders? Qualitative or quantitative?

•Resulting criteria:

• Genericity: how many di®erent modeled systems can we handle?

• Scalability: how large can the size of the problem be?

• Specialization: how much should the intended audience know?

• Population: how large can the intended audience be?

• Convincingness: how plausible are the assumptions?

• Distinctiveness: e.g., how accurate, how certain, how decisive can the model outcome be?

• Surprise: to what extent can the model outcome give new insight?

• Impact: how big can the consequences of the model outcome be?

•Criteria for modeling quality are related to purposes.

1 a core course on modeling contents the need for interpretation approach criteria for...

Documents

modeling criteria

modeling week

working model

modeling approach week

modeling contents

core course

model makers

interpretation week