metrics for mmp development and operations lessons learned: the sims online
DESCRIPTION
Metrics for MMP Development and Operations Lessons Learned: The Sims Online. Larry Mellon GDC, Spring 2004. Metrics: Catch-22. Useful. Careful…. GI / GO. Hard Data. Optimization Tool. Expensive. Importance of Metrics is Relative. Mark Twain. Lord Kelvin. Measure Everything. Measure - PowerPoint PPT PresentationTRANSCRIPT
Metrics forMMP Development and Operations
Lessons Learned: The Sims Online
Larry Mellon
GDC, Spring 2004
Metrics: Catch-22
Useful
Hard Data
Optimization Tool
Careful…
GI / GO
Expensive
Importance of Metrics is Relative
Lord Kelvin Mark Twain
MeasureEverything
Measure “Just Enough”
Invest No More Than You Need
Kelvin Twain
Complexity
Scale
Precision
$$$ $MMP
MMP: Measurement Focal Points
Infrastructure
Operational Costs
Player Actions
Economy
Similar Use Case: Casinos(Harrah’s “Total Reward”)
Unified Player Action DB
CasinoCasino
Table / MachineTable / MachineTable / Machine
Player Player Player
Track every Player Action
Casino
Highly Profitable, Highly Popular
Unified Player Action DBProfit (per Casino, per Player)
Patterns of Play
Analyze:
Casino Operations
Player Awards Program
Modify:
“This is one of the best investments that we have ever made as a corporation and will prove to forge key new business strategies and opportunities in the future.“ John Boushy (Harrah's CIO, 2000)
TSO: Live Monitors, Summary Views
Player Actions & Persistent Data
Embedded Profiler (Server Side)Automated Report Generators
Designers CommunityManagers
Metrics System
Server Reactions
Engineers Operations
Outline
Background: Metrics & MMP
Implementation Overview
Metrics in TSO
Applications & Sample Charts
WrapupLessons Learned
Conclusions
Questions
Implementation:Driving Requirements
Common Infrastructure
Ease of use
Low overhead
Esper Architecture
esperProbeesperProbeesperProbe
Live Server CPUs
ProcessProcessProcess
UserUserUser
esperLogesperLogesperLog
Event-level Sampling, Aggregated Reporting
esperStore
DBImporter
esperLogesperLogesperLog
esperViewesperViewesperView
esperFetch
t1 t2 t3 t4 t5
esperProbes
Min, Max, Av, Count
Esper Probes• Self-organizing: “class” hierarchy
• Data driven: new probes and/or new game content immediately visible on web
• Example: ESPER_PROBE – (“Object.interaction.%s”, chair->picked)– (“Object.interaction.puppet.%s”, self->picked)
• Human-readable intermediate files
EsperView: Web-Driven Presentation
Daily Reports
Graph Caching & Archiving
Filtering & Meta Data
Report Generator
EsperView: Hierarchical Presentation
Process-Level Collection
Server Cluster
Process Class
Process Instance
Outline
Background: Metrics & MMP
Implementation Overview
Metrics in TSO
Applications & Sample Charts
WrapupLessons Learned
Conclusions
Questions
Applications of Metrics
Load Testing(Realistic Inputs)
Beta Testing & Live Operations (Game Tuning, Community Management)
Load Testing & Live Operations(Server Performance)
Test Servers
Load Testing:“Monkey See / Monkey Do”
AlphaVille Servers
Live Beta Testers
Sim Actions (Player Controlled) Sim Actions (Script Controlled)
Applications of Metrics
Load Testing:Realistic Inputs
Beta Testing & Live Operations: Tuning/Mngmnt
Load Testing & Live Operations:Server Performance
Make Friends, Shake Hands
beats outGive Money / Get Money
Least Used: Disco Dancing
History
Meta Data
Top: TD Dance, WoohooBottom: Dance
Players per Lot 0 to 70 players: < 2 / lot 70 to 400 players: > 3 /
lot
Top: Metrics Bug (sorta)Next: Garden Gnome, Toilet
Bottom: Buffet Table
Beta: numPlayers by numRMates
1. Most have one Roommate
2. Hard-core players: 8
Highest
1. Using Skill Objects2. Using Pizza Maker3. Selling Objects
Lowest
1. Visitor Bonus2. New Avatar3. Object “Rentals”
Economy: Detailed View
1. Using Objects = $$$2. Visitor Bonus == $
Visitor Bonus: Who Makes Money?
1. Most getting no V. Bonus2. Hard-core players: $$
4 of top 5: windows?
?
House Categories (Beta Test)
1. Not a well-used feature2. Shopping least of all
Community Management
Community Actions & Trends
Influencing Player Activity
Free Content
Tracking Problem Players
Marketing
Press Release Teasers
In-Game Brand Exposure
Special Events
NYEve: Kiss Count
Esper Cities All Cities (extrapolated)===========================================New Year's Kiss 32,560 271,333Be Kissed Hotly 7,674 63,950Be Kissed 5,658 47,150Be Kissed Sweetly 2,967 24,725Blow a Kiss 1,639 13,658Be Kissed Hello 1,161 9,675Have Hand Kissed 415 3,458===========================================
Total 52,074 433,949
Applications of Metrics
Load Testing:Realistic Inputs
Beta Testing & Live Operations: Tuning/Mngmnt
Load Testing & Live Operations:Server Performance
DB byte count oscillates out of
control
A single DB Request is clearly
at fault
Most-Used DB Queries (unfiltered)
11,000,000 level Queries need attention, and
drown out others
DB Queries (Filtered)
Filters on 11,000,000 level Queries show patterns of 7,000
level Queries
Incoming & Outgoing Packets
Outline
Background: Metrics & MMP
Implementation Overview
Metrics in TSO
Applications & Sample Charts
WrapupLessons Learned
Conclusions
Questions
Lessons Learned
• Implement early
• Ownership, senior engineers
• Aggregated probes vs event-level tracking
• Automation: collect / summarize / alarms
• “There can be only one”…
Conclusion: Very Useful!
Data MiningOn Players:
Untapped Gold…
Game Design
Realistic Load Testing
Engine Fixes, Optimization
Server Cost, Launch Timing
Critical Feature: Accessibility
Questions
Slides available @
www.maggotranch.com/MMP