![Page 1: Sensitivity Analysis of an Agent-Based ... - dma.unive.it · Sensitivity analysis for ABM •Agent-based models may be very sensitive to parameter changes in particular parts of parameter](https://reader034.vdocuments.us/reader034/viewer/2022052017/602fc48fad356f7d40522edb/html5/thumbnails/1.jpg)
Sensitivity Analysis of anAgent-Based Model of
Culture’s Consequences for Trade
Saskia Burgers, Gert Jan Hofstede,Catholijn Jonker, Tim Verwaart
September 9-10, 2010 - Treviso (Italy)
![Page 2: Sensitivity Analysis of an Agent-Based ... - dma.unive.it · Sensitivity analysis for ABM •Agent-based models may be very sensitive to parameter changes in particular parts of parameter](https://reader034.vdocuments.us/reader034/viewer/2022052017/602fc48fad356f7d40522edb/html5/thumbnails/2.jpg)
Sensitivity analysis
• Generally considered “good modelingpractice”
• Of the essence i actual parameter values areuncertain
• A powerful tool in the process of modelverification and validation
• Specific problems arise when performingsensitivity analysis for agent-based models
![Page 3: Sensitivity Analysis of an Agent-Based ... - dma.unive.it · Sensitivity analysis for ABM •Agent-based models may be very sensitive to parameter changes in particular parts of parameter](https://reader034.vdocuments.us/reader034/viewer/2022052017/602fc48fad356f7d40522edb/html5/thumbnails/3.jpg)
Sensitivity analysis for ABM• Agent-based models may be very
sensitive to parameter changes inparticular parts of parameter space:– Nothing may happen in large areas in the joint
parameter space– Areas may exist where the system responds
dramatically to slight changes• Parameters may significantly interact• Sensitivity may be studied for aggregated
individual level outputs
![Page 4: Sensitivity Analysis of an Agent-Based ... - dma.unive.it · Sensitivity analysis for ABM •Agent-based models may be very sensitive to parameter changes in particular parts of parameter](https://reader034.vdocuments.us/reader034/viewer/2022052017/602fc48fad356f7d40522edb/html5/thumbnails/4.jpg)
The model
• Agent-based model of a gamingsimulation
• Models effects of deceit and trust in trade• Concepts of transaction cost economics(contract incompleteness and opportunism)
![Page 5: Sensitivity Analysis of an Agent-Based ... - dma.unive.it · Sensitivity analysis for ABM •Agent-based models may be very sensitive to parameter changes in particular parts of parameter](https://reader034.vdocuments.us/reader034/viewer/2022052017/602fc48fad356f7d40522edb/html5/thumbnails/5.jpg)
Simulation gaming
![Page 6: Sensitivity Analysis of an Agent-Based ... - dma.unive.it · Sensitivity analysis for ABM •Agent-based models may be very sensitive to parameter changes in particular parts of parameter](https://reader034.vdocuments.us/reader034/viewer/2022052017/602fc48fad356f7d40522edb/html5/thumbnails/6.jpg)
Agent roles in the game
producermiddleman
retailer
consumer
tracing agency
![Page 7: Sensitivity Analysis of an Agent-Based ... - dma.unive.it · Sensitivity analysis for ABM •Agent-based models may be very sensitive to parameter changes in particular parts of parameter](https://reader034.vdocuments.us/reader034/viewer/2022052017/602fc48fad356f7d40522edb/html5/thumbnails/7.jpg)
Strategies a buyer can chose
• trust• trace random samples• in addition to random tracing, negotiate
some refund in case quality of non-compliance
• require certification• buy low quality (no risk)
![Page 8: Sensitivity Analysis of an Agent-Based ... - dma.unive.it · Sensitivity analysis for ABM •Agent-based models may be very sensitive to parameter changes in particular parts of parameter](https://reader034.vdocuments.us/reader034/viewer/2022052017/602fc48fad356f7d40522edb/html5/thumbnails/8.jpg)
Process Model (TCE)
Partnerselection
DeliverNegotiation
Trust or Monitorand enforce
F e e d b a c k : b e l i e f u p d a t e
Information asymmetry:opportunity to defect
![Page 9: Sensitivity Analysis of an Agent-Based ... - dma.unive.it · Sensitivity analysis for ABM •Agent-based models may be very sensitive to parameter changes in particular parts of parameter](https://reader034.vdocuments.us/reader034/viewer/2022052017/602fc48fad356f7d40522edb/html5/thumbnails/9.jpg)
Purpose of the Sensitivity Analysis
• To explore the behaviour of the model allover the parameter space
![Page 10: Sensitivity Analysis of an Agent-Based ... - dma.unive.it · Sensitivity analysis for ABM •Agent-based models may be very sensitive to parameter changes in particular parts of parameter](https://reader034.vdocuments.us/reader034/viewer/2022052017/602fc48fad356f7d40522edb/html5/thumbnails/10.jpg)
Decision models
Endowment factorNegative update factorBelief updateTrust (initial value)Honesty decay factorMinimal honestyDeceit and trustRisk aversionQuality preferenceImpatienceNegotiation speedConcession factorNegotiationPreference (initial value)LearningLoyaltyPartner selectionParameter or variableActivity
![Page 11: Sensitivity Analysis of an Agent-Based ... - dma.unive.it · Sensitivity analysis for ABM •Agent-based models may be very sensitive to parameter changes in particular parts of parameter](https://reader034.vdocuments.us/reader034/viewer/2022052017/602fc48fad356f7d40522edb/html5/thumbnails/11.jpg)
Influence of culture
• Culture modifies parameter values in thedecision functions
• Describe culture based on Hofstede’s fivedimensions of national cultures
• Relational attributes have differentsignificance in different cultures:– Group distance– Status difference– Interpersonal trust
![Page 12: Sensitivity Analysis of an Agent-Based ... - dma.unive.it · Sensitivity analysis for ABM •Agent-based models may be very sensitive to parameter changes in particular parts of parameter](https://reader034.vdocuments.us/reader034/viewer/2022052017/602fc48fad356f7d40522edb/html5/thumbnails/12.jpg)
Total of 21 parameters
• Which areas in parameter space result inrealistic behavior?
• In which areas of parameter space cantipping points occur?
• Which parameters have significant effectsfor which outputs?
• Which interactions between culture andother parameters are important?
• Are the answers different betweenaggregate and individual level?
![Page 13: Sensitivity Analysis of an Agent-Based ... - dma.unive.it · Sensitivity analysis for ABM •Agent-based models may be very sensitive to parameter changes in particular parts of parameter](https://reader034.vdocuments.us/reader034/viewer/2022052017/602fc48fad356f7d40522edb/html5/thumbnails/13.jpg)
Approach is based on Jansen et al.
• meta modeling of results of parameter setsdrawn at random from the joint distribution(Monte Carlo analysis), by GLM;
• analysis of contributions of Top MarginalVariance (TMV) and Bottom MarginalVariance (BMV) of individual parametersor groups of parameters to the varianceexplained by the meta model
![Page 14: Sensitivity Analysis of an Agent-Based ... - dma.unive.it · Sensitivity analysis for ABM •Agent-based models may be very sensitive to parameter changes in particular parts of parameter](https://reader034.vdocuments.us/reader034/viewer/2022052017/602fc48fad356f7d40522edb/html5/thumbnails/14.jpg)
Step 1
• Generate random parameter sets, drawnfrom the joint distribution:
• Values of parameters were drawnindepentently, from uniform distributions,over the maximal realistic range
• Note: uniform distribution, because thepurpose is to explore model behavior, notto assess uncertainty
• Check for correlations
![Page 15: Sensitivity Analysis of an Agent-Based ... - dma.unive.it · Sensitivity analysis for ABM •Agent-based models may be very sensitive to parameter changes in particular parts of parameter](https://reader034.vdocuments.us/reader034/viewer/2022052017/602fc48fad356f7d40522edb/html5/thumbnails/15.jpg)
Step 2
• Run simulations for each parameter set• Observe relevant outputs• Replications with different values of the
random seed, to estimate the influence ofinternal stochastic effects on outputvariance
![Page 16: Sensitivity Analysis of an Agent-Based ... - dma.unive.it · Sensitivity analysis for ABM •Agent-based models may be very sensitive to parameter changes in particular parts of parameter](https://reader034.vdocuments.us/reader034/viewer/2022052017/602fc48fad356f7d40522edb/html5/thumbnails/16.jpg)
Observed outputs
• number of transactions;• number of failed negotiations;• average duration of negotiations;• number of high quality transactions;• number of deceitful transactions;• number of traces requested;• number of fines issued by the tracing agency;• average loyalty
![Page 17: Sensitivity Analysis of an Agent-Based ... - dma.unive.it · Sensitivity analysis for ABM •Agent-based models may be very sensitive to parameter changes in particular parts of parameter](https://reader034.vdocuments.us/reader034/viewer/2022052017/602fc48fad356f7d40522edb/html5/thumbnails/17.jpg)
Step 3
• The standard approach is to fit a linearmodel for each output yk
yk = β0 + Σ βi xi
• And analyse the contributions of theparameters xi to output variance
• However: system behavior is far from linear
![Page 18: Sensitivity Analysis of an Agent-Based ... - dma.unive.it · Sensitivity analysis for ABM •Agent-based models may be very sensitive to parameter changes in particular parts of parameter](https://reader034.vdocuments.us/reader034/viewer/2022052017/602fc48fad356f7d40522edb/html5/thumbnails/18.jpg)
Standard solution
• If linear model does not fit, try higher orderterms, e.g.
yk = β0 + Σ βi xi + Σ βiʹ′xi2
• Or interaction terms, e.g.
yk = β0 + Σ βi xi + Σ γij xi xj
• But this didn’t result in good fit (R2 > 80%)
![Page 19: Sensitivity Analysis of an Agent-Based ... - dma.unive.it · Sensitivity analysis for ABM •Agent-based models may be very sensitive to parameter changes in particular parts of parameter](https://reader034.vdocuments.us/reader034/viewer/2022052017/602fc48fad356f7d40522edb/html5/thumbnails/19.jpg)
First finding of further exploration
• For many of the parameter sets drawn atrandom, no transactions occur
• No obvious regions in parameter spacecould be discovered where transactionsoccur / no transations occur, nor by visualinspection, no by PCA
• Logistic regression could be applied todiscover the parts of parameter spacewhere transactions occur
![Page 20: Sensitivity Analysis of an Agent-Based ... - dma.unive.it · Sensitivity analysis for ABM •Agent-based models may be very sensitive to parameter changes in particular parts of parameter](https://reader034.vdocuments.us/reader034/viewer/2022052017/602fc48fad356f7d40522edb/html5/thumbnails/20.jpg)
transactions are unlikely to occur if
• group distance and LTO are both high;• status difference and concession factor are both low;• MAS, LTO, and negotiation speed are all high or all low;• status difference and intitial trust are both low;• IDV, LTO, and quality preference are all high;• status difference is low and initial partner preference is high;• initial partner preference is low and quality preference is high;• IDV and minimal honesty are both high;• honesty decay factor and endowment factor are both high
![Page 21: Sensitivity Analysis of an Agent-Based ... - dma.unive.it · Sensitivity analysis for ABM •Agent-based models may be very sensitive to parameter changes in particular parts of parameter](https://reader034.vdocuments.us/reader034/viewer/2022052017/602fc48fad356f7d40522edb/html5/thumbnails/21.jpg)
Conclusion of this first exercise
• For a model with complex interactions, it isvirtually impossible to understand why itbehaves like it behaves
• For further sensitivity analysis, focus onthe regions in parameter space wheretransactions occur, by resampling from theoriginal sample
![Page 22: Sensitivity Analysis of an Agent-Based ... - dma.unive.it · Sensitivity analysis for ABM •Agent-based models may be very sensitive to parameter changes in particular parts of parameter](https://reader034.vdocuments.us/reader034/viewer/2022052017/602fc48fad356f7d40522edb/html5/thumbnails/22.jpg)
Metamodel with quadratic terms,and 30 forwardly selected two-factor
interactions (R2 = 80 %)
0.00.0Honesty decay factor3.01.2Initial partner preference
0.00.0Minimal honesty6.32.7Initial trust
0.00.1Endowment factor1.90.0Status difference
0.30.9Negative update factor5.10.3Mean status
3.00.0Risk avoidance6.82.7Group distance
0.71.5Quality preference7.72.0-- MAS
2.50.6Impatience6.80.7-- LTO
39.331.8Negotiation speed3.40.8-- UAI
25.09.1Concession factor5.30.2-- IDV
0.10.0Loyalty decay factor0.60.0-- PDI
0.00.0Loyalty parameterIndex of culture
BMV(%)
TMV (%)ParameterBMV(%)
TMV(%)
Parameter
![Page 23: Sensitivity Analysis of an Agent-Based ... - dma.unive.it · Sensitivity analysis for ABM •Agent-based models may be very sensitive to parameter changes in particular parts of parameter](https://reader034.vdocuments.us/reader034/viewer/2022052017/602fc48fad356f7d40522edb/html5/thumbnails/23.jpg)
Sensitivity of other outputs• negotiation failure:
– negotiation speed, concession factor, impatience• negotiation duration:
– negotiation speed, UAI, MAS• quality:
– intitial trust, LTO, quality preference• deceit:
– initial trust, MAS• tracing:
– MAS• fines:
– MAS• loyalty:
– initial partner preference, negotiation speed
![Page 24: Sensitivity Analysis of an Agent-Based ... - dma.unive.it · Sensitivity analysis for ABM •Agent-based models may be very sensitive to parameter changes in particular parts of parameter](https://reader034.vdocuments.us/reader034/viewer/2022052017/602fc48fad356f7d40522edb/html5/thumbnails/24.jpg)
Conclusions
• Number of transactions is most sensitiveto negotiation speed and concessionfactor
• Cultural and relational factors have theireffects only in interaction with other factors
• Analyse sensitivity culture by culture,using a sample of 62 actual nationalcultures
![Page 25: Sensitivity Analysis of an Agent-Based ... - dma.unive.it · Sensitivity analysis for ABM •Agent-based models may be very sensitive to parameter changes in particular parts of parameter](https://reader034.vdocuments.us/reader034/viewer/2022052017/602fc48fad356f7d40522edb/html5/thumbnails/25.jpg)
Sensitivity differences (TMV)
15.80.10.930.81.68.30.00.5Japan
6.94.811.527.12.63.80.00.0Austria
0.00.656.210.30.40.93.11.3Iran
2.10.028.846.20.31.60.00.7Netherlands
0.00.024.123.45.58.51.74.9Uruguay
11.52.41.437.61.911.50.00.0Hungary
0.00.037.917.54.73.38.70.7Morocco
0.00.040.911.60.00.20.116.9Indonesia
1.60.530.224.81.83.41.04.1mean
riskavoid.
qualitypref.
negot.speed
conces.factor
partnerpref.
initialtrust
meanstatus
groupdistance
nationalculture
![Page 26: Sensitivity Analysis of an Agent-Based ... - dma.unive.it · Sensitivity analysis for ABM •Agent-based models may be very sensitive to parameter changes in particular parts of parameter](https://reader034.vdocuments.us/reader034/viewer/2022052017/602fc48fad356f7d40522edb/html5/thumbnails/26.jpg)
Conclusion
• Sensitivity analysis and the interpretationof its results are much easier if parametershaving many interactions are fixed
• For specific situations the results of thispartial analysis may be more useful thanglobal results
![Page 27: Sensitivity Analysis of an Agent-Based ... - dma.unive.it · Sensitivity analysis for ABM •Agent-based models may be very sensitive to parameter changes in particular parts of parameter](https://reader034.vdocuments.us/reader034/viewer/2022052017/602fc48fad356f7d40522edb/html5/thumbnails/27.jpg)
Overall conclusion• Logistic regression can be used to predict
regions where realistic behavior occurs.• Parameters that have significant effects can be
identified through metamodeling, even forcomplex systems. However, the analysis is notstraightforward.
• When keeping culture constant, straightforwardmethods for sensitivity analysis can be applied.Results differ considerably across cultures.
• Sensitivity of individual agents can differconsiderably from aggregate level sensitivity.