QUESTIONS

Expanding the virtual world

Dave PooleCCP Games, Web Technical Director

• Intro to EVE Online• EVE Gate

• What is it?• Why it is important• How it came to be• Lessons learned

• EVE Gate – the Technology Behind it• The software architecture• The hardware powering it• Performance

• CCP and IBM• Schooner Caching Solutions

• Q & A

AGENDA

A vast persistent virtual universe

Over 330,000 players inhabit a single game world competing for military, economic & political power

Players in over 230 countries

Steady growth since 2003

Hardcore, dark and ruthless

Playground, sandbox and emergence

Introduction to EVE Online

Key design tenet:

“MMOGs are social networks

• The key factor that makes MMOs succeed or fail is the strength of their social network.

• People will quit games, but are less likely to quit friends.

• Maximize human interaction!“

Reynir Hardarson FOUNDER and CREATIVE DIRECTOR of CCP

EVE Online: Design

EVE Gate: Overview

What is it?

EVE Gate: Overview

INITIAL RELEASE

Website with:

• Real time access to EVE mail system• A commenting system similar to

Facebook• A calendar system in game and on

website fully synched• A reworked contact system combined

with a personal settings system to give player control of privacy and access to their info

• INSERT SCREENSHOTS

FUTURE FORM

We have a full time development team continually adding to it

Proposed features:

• Expanded tagging system• Web access to the skill queue• New forums possibly including Corporation level forums (Groups)• Real time access to the market systems, read only to start• Fitting windows, saving / trading popular ship loadouts• Killboards• Public profiles and info to push to external apps like Facebook (player controlled)• Sovereignty results, controls, etc• Corporation Management• Basically anything that does not involved flying, shooting, etc in game

EVE Gate: Overview

– EVE Gate will be utilized for the whole EVE universe– It will be used to connect EVE players and DUST players– Facilitates corporation to corporation communication and

coordination– DUST specific features will be added, more applicable to its product

type (game results, statistics, heatmaps, screenshots,etc)– Important bridge between 2 different games, in 2 different genres

within the same persistent universe

EVE Gate: Overview

BEYOND THE WEB

• The importance of APIs• Mobile possibilities• SOA: Flexibility for the future

EVE Gate: Overview

Important for CCP

Purpose and goals: Breaking down barriers and exposing the EVE universe for all to experience and interact with

• A gateway for interaction from anywhere, at anytime with anyone• A tool to operate and organize player entities• A pathway to syndicate information, propaganda and events• A mechanic for current customer retention• A vehicle for new customer acquisition• A connection between the EVE and Dust communities• A platform to respond swiftly and capitalize on opportunities and disruptions

This is a critical service provided to the players, for a business driven by a service model

EVE Gate: Why is it important

Important for the industry

• More and more developers are expanding beyond the game client• Turbine with My LOTRO• Blizzard with Battle.NET• Bungie with Bungie.Net

• This is a social network designed specifically for the specific customer base, not forced upon them from an external solution like Facebook

• Nature of EVE demands extreme privacy and information controls• Nature of alts (spies, etc) drives the need for this to be character focused, not

player focused – thus linking to external individual focused solutions is detrimental• Not intended as an advertising / revenue source but as a service

EVE Gate: Why is it important

Observations and studies of player behavior and trends

• The importance of social interaction• Product stickiness and what drives it• The importance of player driven events and the

syndication of information• Full time Research & Statistics analyzing needs, plus

CSM input

Project requirements fleshed out• Developed over 2+ years by the EVE producers• Need to find the right time to implement

• Player needs, market conditions, talent available, plus other company initiatives

• The need for a project ‘champion’

EVE Gate: How it came to be

Step 1 (done): what do we want to build?

Step 2: how do we build it?

• Architectual designo build on proven high scalability designs from other industrieso work with company strengths (python, .Net, SQL Server)

• Staffing and resource allocation

• Hardware Infrastructure Designo the importance of working closely with Operationso use of phased stress testing and product evals to nail down requirementso Scheduling development and testing to coincide with and support infrastructure needs

EVE Gate: How it came to be

Step 3: development

2 sprint teams working for a full yearTeams consisted of:

• Web developers• EVE Software developers• Graphic designers• DBAs• Dedicated QA

Included talent from Atlanta and Reykjavik so we faced challenges with doing SCRUM development across the pond

EVE Gate: How it came to be

Game design• Don’t force a solution upon your player base, look for what services fit their need and give them control

o Example: privacy settings and access controls, external integration?• The importance of information security

Project Management• The importance of a champion to drive the project• The importance of operations involvement from the start

Development Process• Learned how to do SCRUM over distance

o Less effective but sometimes necessaryo Mitigate through extra communication effort (vid conferencing) and as much face-to-face time as possible

through travelo Team building HUGELY important

• The importance of scripted and public stress testing (HUGE)• The importance of close liaison with operations and your vendor partners

EVE Gate: Lessons learned

EVE Gate: The Technical Solution

Umbraco – Content management system

View

Controllers

Data Model

EVEGate.web

EVEGate.dataobjects

Web DBCachingCluster

Memcached

ForumApplication

BloggingApplication

Other .NetApplications

sessions

Web Database Cluster(SQL Server 2008)

Nexus

EVE Gate Web Application(s)

SOL Processes SOL Cluster

EVE Database Cluster(SQL Server 2008)

TQ DB

Tranquility

EVE Gate Web Cluster (Windows Server 2008)

EVE Gate Service Cluster (Windows Server 2008)

EVE Gate WCF Application

FUTURE RELEASES

EVE Gate Launch Architecture

Denotes new components

EVE Gate MVC Application

EVE Gate: The Technical Solution

The software architecture

Key points Design needs to minimize impact on game systems Cannot have in game lag caused by out of game systems Design needs to have scalability and reliability built into each layer Needs to be easy to maintain and upgrade Needs to consider security without impacting performance

Driven by a Service Oriented Architecture design approach Proven approach used by high volume web applications and web services View layer built with modern web techniques (.Net MVC C#) Connect to data providers including existing systems with a communication layer (WCF) Added a service agent to the game cluster (SOL nodes) to coordinate data flow Backend databases and game services retained (Python, SQL Server) with some enhancements for new

services Each layer can be independently scaled, refactored or replaced Each layer separately secured and monitored

EVE Gate: The Technical Solution

Front End• Latest .Net (C#) framework MVC

• Strong MS support, high industry uptake• Designed after well proven web MVC frameworks like STRUTS• Lots of tools, techniques, components available• Works well with AJAX based techniques, we use heavily

• Use our built-up expertise in web development where it makes most sense

Communications Tier• .Net Windows Communication Foundation (WCF) application• Coordinates all data access requests• Encapsulates all cache management, hides implementation • Re-usable service layer can be used for other external apps (APIs, Mobile, etc)• Can later be refactored into a common data layer for ALL CCP applications

EVE Gate: The Technical Solution

Caching Tier- Caching is critical to minimize load on backend services- Greatly improves web application performance- Replaces stock .Net caching implementation - Use Schooner Caching Appliances which implements memcached

- Able to use standard Memcached APIs minimizing code effort- Transparent implementation, in vendor trials we were able to swap in

hardware without changing a SINGLE line of code- Memcached – open source, high use, developed for LiveJournal,

heavily used by Facebook and other high scale sites

The Hardware Infrastructure

Introducing James Wyld, CCP Virtual Worlds Administrator

EVE Gate: The Technical Solution

• Breaking down the software

EVE Gate: Under the hood

Umbraco – Content management system

View

Controllers

Data Model

EVEGate.web

EVEGate.dataobjects

Web DBCachingCluster

Memcached

ForumApplication

BloggingApplication

Other .NetApplications

sessions

Web Database Cluster(SQL Server 2008)

Nexus

EVE Gate Web Application(s)

SOL Processes SOL Cluster

EVE Database Cluster(SQL Server 2008)

TQ DB

Tranquility

EVE Gate Web Cluster (Windows Server 2008)

EVE Gate Service Cluster (Windows Server 2008)

EVE Gate WCF Application

FUTURE RELEASES

EVE Gate Launch Architecture

Denotes new components

EVE Gate MVC ApplicationEVEGate Web Servers

EVEGate DB

Service Layer (WCF)Memory Cache Cluster

Presentation Layer (Frontend)

EVE Game Server

EVE DB

EVE Proxy

EVE Sol

• Alpha Test– Getting the Alpha test up and running

was paramount.– The focus was on the architecture

elements and not features.– Vertical slice.

• Needed solid data before building the final system.

EVE Gate: Performance testing

• Key performance metrics– CPU per user -> users per core– Network bandwidth per user

• Bottlenecks– Service Layer CPU– Caching Network Bandwidth– Firewall placement

EVE Gate: Under the hood

EVEGate Web Servers

EVEGate DB

Service Layer (WCF)Memory Cache Cluster

Presentation Layer (Frontend)

• Infrastructure plan was changed accordingly

– Additional CPU power required across the board.

– Physically separated network for cache layer.

– Prompted a look at the core network switching capabilities.

EVE Gate: Under the hood

EVEGate Web Servers

EVEGate DB

Service Layer (WCF)Memory Cache Cluster

Presentation Layer (Frontend)

• Frontend / Backend server farm– Built on (in?) IBM Bladecenter, the same platform as

EVE Online.– Blades allow for a large amount of CPU power in a

small amount of rack space.– For CCP, IBM is simply a solid, reliable partner.

• Caching layer– Original plan was to build out the cluster manually– “Black Box” approach means we do not have to spend

time managing a linux/memcached farm.– More focus on running EVEGate and EVE Online!

• Database– MS SQL– Most data resides in the existing game DB

EVE Gate: The final system

EVE Gate: Putting it all together

• Fine-tuning the system– We created a scripted, scalable, repetitive load test.– The focus was on overloading each area of the infrastructure separately.– We found many new and interesting ways to break the application.– From an operational standpoint, we learned as much from this phase as the rest of the dev

cycle.– It took quite some time to stabilize it.