killzone's servers: flexible architecture and component-based design (mbl305) | aws re:invent...

© 2013 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified, or distributed in whole or in part without the express consent of Amazon.com, Inc.

(re)Building Killzone’s Servers How We Used AWS in Killzone: Shadow Fall and

Killzone: Mercenary

Tim Darby, Sony Computer Entertainment Europe

November 14, 2013

Bio

Aims

What is Killzone?

A brief history of Killzone’s servers

Killzone • Serving core multiplayer

experience

• Centrally developed backend tech

• No title-specific backend code

• C/C++

• Regional real-time gaming servers

• Centralized core servers

Killzone

(2004)

Killzone : Liberation

(2006)

Killzone 2

(2009)

Killzone 3

(2011)

Killzone : Mercenary

Killzone : ShadowFall

(2013)

KZ : Liberation • Original real time backend plus …

• Java 3-tier backend for community features

• Shared tech again

• Customization at presentation layer, plus configuration

• Plugins for score processing and tournament progression in ‘singleton’

• Leaderboards sat on the DB

Killzone

(2004)

Killzone : Liberation

(2006)

Killzone 2

(2009)

Killzone 3

(2011)

Killzone : Mercenary

Killzone : ShadowFall

(2013)

Killzone 2 • Applying lessons learned from Killzone :

Liberation

• Materialized views added to leaderboard backend

• Battle Replay using shared file system (NFS)

• Plugins in ‘singleton’ servers for Tournament progression

• Additional webapp delivering XML for website

Killzone

(2004)

Killzone : Liberation

(2006)

Killzone 2

(2009)

Killzone 3

(2011)

Killzone : Mercenary

Killzone : ShadowFall

(2013)

Killzone 3 • Iteration on previous backend

(java and C++)

• Redone leaderboards from Materialized Views to MV + TSDB

• Plugins to middle tier

• C++ side matchmaking server with title-specific MM rules plugin

• And it wasn’t just us feeling the constraints…

Killzone

(2004)

Killzone : Liberation

(2006)

Killzone 2

(2009)

Killzone 3

(2011)

Killzone : Mercenary

Killzone : ShadowFall

(2013)

A Busy Year

September 2013 November 2013

Killzone

(2004)

Killzone : Liberation

(2006)

Killzone 2

(2009)

Killzone 3

(2011)

Killzone : Mercenary

Killzone : ShadowFall

(2013)

Re-arch Killzone

(2004)

Killzone : Liberation

(2006)

Killzone 2

(2009)

Killzone 3

(2011)

Killzone : Mercenary

Killzone : ShadowFall

(2013)

• Separately versioned

shared components

• Independent game

projects

• Maven + Jenkins

• Dynamic Scaling

• Use of AWS services

Mercenary • Relatively straightforward REST API

• Frontend, Scheduler, Admin Elastic Beanstalk applications

• Multiple Beanstalk Environments per app (staging, or dev envs)

• Both “SingleInstance” and “ELB” Beanstalk Env types used

• PSN functionality used for real-time elements

– Matching, NAT Traversal, P2P

Killzone

(2004)

Killzone : Liberation

(2006)

Killzone 2

(2009)

Killzone 3

(2011)

Killzone : Mercenary

Killzone : ShadowFall

(2013)

ShadowFall • Straight REST, with addition of

Comet style push messaging

• Similar topology for Elastic

Beanstalk elements to KZM

• Addition of real-time servers

• Hybrid deployment – part Sony

hosting, part AWS

Killzone

(2004)

Killzone : Liberation

(2006)

Killzone 2

(2009)

Killzone 3

(2011)

Killzone : Mercenary

Killzone : ShadowFall

(2013)

Use of AWS Services

Scheduled Jobs • Scheduler webapp simple spring-quartz

• Generates jobs, posts to Amazon SQS

• Main webapp runs SQS receivers

• Scheduled job receivers spawn worker threads per job

• Wrappers around these jobs provide – Amazon CloudWatch duration/error tracking

– Distributed job management

• Admin app provides ability to manually schedule/fire jobs, and view distributed job state info

Score Posting #1 • Score posts pushed to SQS for async

processing

• Listeners on component for game

history store triggered on successful

write to Amazon DynamoDB

• Individual listeners perform many

secondary tasks

• Failure modes in secondary tasks not

easily recovered

Score Posting #2 • Changed game history listeners to

post to secondary game jobs SQS

queue

• Added separate receiver threads for

secondary game jobs SQS queue

messages

• Receivers then run secondary jobs

• Failure modes in secondary jobs now

result in non-confirmation of SQS

message, and re-presentation by SQS

Leaderboards • SimpleDB, transitioned to DynamoDB

when available

• DynamoDB required secondary indices in some form for multiple sort columns on a single board

• Single Table vs. Multiple Table implementations

• Fast ranking a problem for both

• Fast ranking provided by reverse index rank calculator or statistical predictor

Issues and Lessons Learned

Issues • Old-style monitoring of dynamically

scaled environments

• Manual scaling management on DynamoDB

• DynamoDB, leaderboards, and hotspots

• Surprise! Changes to console

• Limits of AWS CloudFormation

Lessons Learned • Appreciate the change in mindset going

from SQL to NoSQL

• Swappable implementations generate

flexibility

• Flexibility around architecture adds

options

• Dynamic scaling/management doesn’t

necessarily stop at server boxes

• Still no silver bullet for leaderboards

Summary

Questions?

Please give us your feedback on this

presentation

As a thank you, we will select prize

winners daily for completed surveys!

MBL305