Risk Driven Web Site Testing

Adding Value to Business Critical Applications

Overview

• Risk management principles

• Mechanics of risk management

• A live example – online stock exchanges

• Conclusions

Risk Management Principles

• Identify key web site stakeholders

• Define and prioritise key stakeholder objectives

• Identify web site “value drivers” - factors that create or destroy shareholder value

• Proactively identify, assess and manage associated risks

Stakeholder Objectives

Shareholders and other investors

Return on investment, increased shareholder value

Employees Challenging work, growth, monetary reward

Strategic partners and suppliers

Business opportunities, competitive advantage

Customers Product and service quality

Government Taxation and regulatory compliance

Community Social responsibility and contribution

Environment Environmental sustainability

Online Stock Exchanges

Listed companies

Real-time performance feedback, liquidity, low price volatility, spreads, SEC compliance, accurate share records, listing fees, status and reputation, access to capital

Retail brokerages

Real-time access to professional trading services, low fees, fast and accurate trades, liquidity, depth of market, narrow spreads

Market makersHigh order volumes, exclusive access to order flow, liquidity, continuous quotations

Day tradersReliable access, fast and accurate trades, accurate and timely information, liquidity, low fees

Institutional investors

Confidentiality, low market impact of large trades, liquidity, market depth, reasonable fees

Web Site Value Drivers

FunctionalityAccuracy, interoperability, compliance, security, auditability, suitability

Reliability Maturity, fault tolerance, recoverability

Usability Operability, learnability, understandability

Efficiency Time and resource behaviour

Maintainability Analysability, changeability, stability, testability

Portability Replaceability, adaptability, installability, conformance

Online Stock Exchange Value Drivers

LiquidityVolume of shares that can be bought or sold without major price fluctuation, volume of trading activity in the marketplace, number of participants

SpeedFast order execution (“near instantaneous”), real-time performance feedback, continuous quotations, timely information

AccuracyAccurate price discovery, accurate execution of trade orders, accurate representation of stocks

EfficiencyEfficient supply/demand resolution, keeping pace with competitors, low cost automated transactions/communication

Trust

Enforcement of and compliance with relevant regulations, system security and integrity, perceived fairness and legitimacy of transactions, auditable systems, reliable clearance and settlement of transactions

Risk Identification

Risk Analysis

Risk Mitigation Planning

On-going risk mitigation and monitoring

Who is responsible

All Players

Risk Owner

Risk Owner

Risk Owner & Risk Management Team

Risk Management

Mechanics of Risk Management

• Identify opportunity, uncertainty and hazard risks

• Rank risks by impact and likelihood to establish a web site risk profile

• Plan specific risk management and control processes to respond to risks

• Align risk management and control processes with web site testing activities

Types of Risk

• Pursue opportunities for competitive advantage and enhanced shareholder value

• Manage uncertainty in achieving operational performance

• Control the impact of bad events (hazards) through compliance and prevention

• Strike a balance between risk, growth and return

Online Stock Exchange Risks

Commercial and legal System functionality breaches SEC regulations

Economic Transaction fees are higher than those of competitors

Human behavior Chicken and egg nature of critical mass and market liquidity

Natural events Flooding of the exchange’s underground server room due to cracks in nearby water mains

Political circumstances

Government initiates restraints on continuous listing, closing “day trader” opportunities

TechnologyThe new release of the order matching system is not backwards compatible with existing Nasdaq transaction gateways

Management controls Insufficient audit mechanisms exist to identify and track down trading anomalies

Individual activities Exchange participants act in “collusion” to set high stock prices

High ConsequenceLow Likelihood

Low ConsequenceLow Likelihood

High ConsequenceHigh Likelihood

Low ConsequenceHigh Likelihood

Likelihood of occurrence

Pot

enti

al C

onse

qu

ence

Denotes an identified risk

Risk Likelihood and Consequence

Web Site Risk Profile

Consequences

Insignificant Minor Moderate Major Catastrophic

Likelihood 1 2 3 4 5

A – (almost certain)

H H E E E

B – (likely) M H H E E

C – (moderate)

L M H E E

D – (unlikely) L L M H E

E – (rare) L L M H H

Web Site Testing Activities

• Test usability, performance, security and availability

• Rigorously test high or extreme risk characteristics

• Reduce testing on lower risk characteristics

• Complete high or extreme risk testing at the expense of lower risk testing, if necessary

Conclusions

• Impact of risk assessment principles and processes on web site testing effectiveness and efficiency

• Outcomes of the wider application of risk driven web site testing

• Sources of further information