the server technology of eve online: how to cope with 300,000 players in one world kristján valur...
TRANSCRIPT
![Page 1: The Server Technology Of Eve Online: How To Cope With 300,000 Players In One World Kristján Valur Jónsson Senior Software Architecht (CCP Games) Most](https://reader036.vdocuments.us/reader036/viewer/2022062417/551afa3155034607418b462a/html5/thumbnails/1.jpg)
![Page 2: The Server Technology Of Eve Online: How To Cope With 300,000 Players In One World Kristján Valur Jónsson Senior Software Architecht (CCP Games) Most](https://reader036.vdocuments.us/reader036/viewer/2022062417/551afa3155034607418b462a/html5/thumbnails/2.jpg)
The Server Technology Of Eve Online: How To Cope With 300,000 Players In One World
Kristján Valur JónssonSenior Software Architecht (CCP Games)
Most MMOGs separate their players onto multiple servers in order to scale; these servers are often called shards. The shards are essentially instances of the game world and the player base is thus broken into segments, based on which shard (server) they are playing on. EVE Online takes a different approach: the technology and the game is developed around the goal of one unified world. All the players log onto the same server cluster and play in the same world. This session will explore the technology developed by CCP to achieve this goal and discuss the technology choices made by CCP, focusing on the software side. Topics include Cluster Technology, Stackless Python, Networking and Database technology.
![Page 3: The Server Technology Of Eve Online: How To Cope With 300,000 Players In One World Kristján Valur Jónsson Senior Software Architecht (CCP Games) Most](https://reader036.vdocuments.us/reader036/viewer/2022062417/551afa3155034607418b462a/html5/thumbnails/3.jpg)
The server technology of EVE
How to have 360000 players on one server
![Page 4: The Server Technology Of Eve Online: How To Cope With 300,000 Players In One World Kristján Valur Jónsson Senior Software Architecht (CCP Games) Most](https://reader036.vdocuments.us/reader036/viewer/2022062417/551afa3155034607418b462a/html5/thumbnails/4.jpg)
About EVE
• One world, one server cluster• Launched in May 2003, has been steadily growing
ever since• Currently at over 360,000 players (~300,000 paying
subscribers, trials account for rest)• Peak Concurrent Users (PCU) record at 60,453• That’s a lot of people!
![Page 5: The Server Technology Of Eve Online: How To Cope With 300,000 Players In One World Kristján Valur Jónsson Senior Software Architecht (CCP Games) Most](https://reader036.vdocuments.us/reader036/viewer/2022062417/551afa3155034607418b462a/html5/thumbnails/5.jpg)
60,000 people!
![Page 6: The Server Technology Of Eve Online: How To Cope With 300,000 Players In One World Kristján Valur Jónsson Senior Software Architecht (CCP Games) Most](https://reader036.vdocuments.us/reader036/viewer/2022062417/551afa3155034607418b462a/html5/thumbnails/6.jpg)
The Network Effect
The value of a service to a potential customer is dependent on the number of customers already using that service
Value of a network equals approximately the square of the number of users in the system
EVE Online is inherently a better game with 300.000 subscribers than 50.000 subscribersSize Matters!
Network effect
Metcalfe’s law
Ergo
![Page 7: The Server Technology Of Eve Online: How To Cope With 300,000 Players In One World Kristján Valur Jónsson Senior Software Architecht (CCP Games) Most](https://reader036.vdocuments.us/reader036/viewer/2022062417/551afa3155034607418b462a/html5/thumbnails/7.jpg)
EVE China
• Launched in 2006• The Serenity cluster is Tranquility’s little brother• This talk is about EVE International, running on the
Tranquility cluster in London.
![Page 9: The Server Technology Of Eve Online: How To Cope With 300,000 Players In One World Kristján Valur Jónsson Senior Software Architecht (CCP Games) Most](https://reader036.vdocuments.us/reader036/viewer/2022062417/551afa3155034607418b462a/html5/thumbnails/9.jpg)
I could talk all day!
• Game design• Operating systems• Multicore• Multiprocessing• IPC• Winsock• OLEDB• Switches/Routers
• SQL• Introspection• Revision control• Profiling• Racist bike shops• Live debugging• Compression• Session propagation
![Page 10: The Server Technology Of Eve Online: How To Cope With 300,000 Players In One World Kristján Valur Jónsson Senior Software Architecht (CCP Games) Most](https://reader036.vdocuments.us/reader036/viewer/2022062417/551afa3155034607418b462a/html5/thumbnails/10.jpg)
But we have to be practical!
• I will focus on a few key aspects• These are deep and complex topics.
I will skip some of the dangerously complicated details.
• There will still be technical stuff• Demos and videos• Additional reference material
provided at the end.
![Page 11: The Server Technology Of Eve Online: How To Cope With 300,000 Players In One World Kristján Valur Jónsson Senior Software Architecht (CCP Games) Most](https://reader036.vdocuments.us/reader036/viewer/2022062417/551afa3155034607418b462a/html5/thumbnails/11.jpg)
The main reasons for EVE’s scalability
• Database system• Stackless Python• Destiny and space partitioning• Hardware configuration
![Page 12: The Server Technology Of Eve Online: How To Cope With 300,000 Players In One World Kristján Valur Jónsson Senior Software Architecht (CCP Games) Most](https://reader036.vdocuments.us/reader036/viewer/2022062417/551afa3155034607418b462a/html5/thumbnails/12.jpg)
Cluster architecture
![Page 13: The Server Technology Of Eve Online: How To Cope With 300,000 Players In One World Kristján Valur Jónsson Senior Software Architecht (CCP Games) Most](https://reader036.vdocuments.us/reader036/viewer/2022062417/551afa3155034607418b462a/html5/thumbnails/13.jpg)
EVE‘s foundation is the database
• Efficient management of data in an un-sharded MMOG is crucial
• All game-related data is stored in a single database• Provides final synchronization of game logic• EVE database is currently at 1.3 terabytes• Database transactions are usually around 1500 per
second, reaching 2000 per second at peak hours• ‘Items’ table receives over 9 million insertions in a
day (not counting update and select statements)
![Page 14: The Server Technology Of Eve Online: How To Cope With 300,000 Players In One World Kristján Valur Jónsson Senior Software Architecht (CCP Games) Most](https://reader036.vdocuments.us/reader036/viewer/2022062417/551afa3155034607418b462a/html5/thumbnails/14.jpg)
EVE demands an enterprise level DB
• Microsoft SQL 2008 x64 (enterprise edition)• IBM X-series 3850 M2 brick server• 128 GB RAM• 2 x CPU, each six-core at ?? GHz• 2x 128 GB RamSan 400 (solid-state drive)• 1x 2TB RamSan 500
![Page 15: The Server Technology Of Eve Online: How To Cope With 300,000 Players In One World Kristján Valur Jónsson Senior Software Architecht (CCP Games) Most](https://reader036.vdocuments.us/reader036/viewer/2022062417/551afa3155034607418b462a/html5/thumbnails/15.jpg)
…and enterprise level DB engineers
• Tables and procedures are carefully designed• Constant monitoring (through DB traces, health
checks on DB server) to watch for bottlenecks and optimization opportunities
• Optimizations sometimes trickle all the way up to game design
• Recently created robust versioning system for DB content, you will eventually need this.
![Page 16: The Server Technology Of Eve Online: How To Cope With 300,000 Players In One World Kristján Valur Jónsson Senior Software Architecht (CCP Games) Most](https://reader036.vdocuments.us/reader036/viewer/2022062417/551afa3155034607418b462a/html5/thumbnails/16.jpg)
Application DB model
• Stored procedures for everything– Hand crafted by DB Engineers– Lowest possible run-time cost– All DB logic in one place
• Simple application model– Integration with Python– Looks like regular synchronous function call– Return highly optimized Rowset objects
![Page 17: The Server Technology Of Eve Online: How To Cope With 300,000 Players In One World Kristján Valur Jónsson Senior Software Architecht (CCP Games) Most](https://reader036.vdocuments.us/reader036/viewer/2022062417/551afa3155034607418b462a/html5/thumbnails/17.jpg)
Example code:
![Page 18: The Server Technology Of Eve Online: How To Cope With 300,000 Players In One World Kristján Valur Jónsson Senior Software Architecht (CCP Games) Most](https://reader036.vdocuments.us/reader036/viewer/2022062417/551afa3155034607418b462a/html5/thumbnails/18.jpg)
A more effective way to program
What we wanted • A dynamic scripting language for rapid turnaround• A procedurally synchronous model
We tried many things
We almost got it working with an implementation of JavaScript and NT fibers
Python 1.5.2 As soon as we found Stackless Python early summer 2000 we knew we had a winner.
![Page 19: The Server Technology Of Eve Online: How To Cope With 300,000 Players In One World Kristján Valur Jónsson Senior Software Architecht (CCP Games) Most](https://reader036.vdocuments.us/reader036/viewer/2022062417/551afa3155034607418b462a/html5/thumbnails/19.jpg)
What is Python
• A dynamic programming language• Simple intuitive syntax• Easily embeddable and extendable• Rapid dynamic programming environment with
dynamic reloading of code (in-house tech)• Open source
For more info, go to http://python.org
![Page 20: The Server Technology Of Eve Online: How To Cope With 300,000 Players In One World Kristján Valur Jónsson Senior Software Architecht (CCP Games) Most](https://reader036.vdocuments.us/reader036/viewer/2022062417/551afa3155034607418b462a/html5/thumbnails/20.jpg)
But what is Stackless?
• An alternative version of Python• Forked from the Python code• Completely backwards compatible• Contains additional features:
– Supports a “tasklet” based application programming model
– Enables many logical tasks– Allows the user to use an imperative, procedural
programming model instead of an event driven model to achive parallel IO
![Page 21: The Server Technology Of Eve Online: How To Cope With 300,000 Players In One World Kristján Valur Jónsson Senior Software Architecht (CCP Games) Most](https://reader036.vdocuments.us/reader036/viewer/2022062417/551afa3155034607418b462a/html5/thumbnails/21.jpg)
Why stackless?
• Lightweight cooperative “threading” using tasklets• Allows you to write robust procedural code, without
the boilerplate and machinery of an event-driven model
• Allows switching of running of both python code and also C code, using stack slicing.– Extends to third party libraries, C/C++, etc.
![Page 22: The Server Technology Of Eve Online: How To Cope With 300,000 Players In One World Kristján Valur Jónsson Senior Software Architecht (CCP Games) Most](https://reader036.vdocuments.us/reader036/viewer/2022062417/551afa3155034607418b462a/html5/thumbnails/22.jpg)
Tasklet essentials
• Execution context, not threads!– Only one is running at a time– An application can have 10000 active tasklets or more.– An application can only have relatively few threads
• Co-operative switching– Context switches occur at known point, almost no need for
locks!
• Can even switch C Extensions!– Windows overlapped, asynchronous IO looks like
synchronous calls.
![Page 23: The Server Technology Of Eve Online: How To Cope With 300,000 Players In One World Kristján Valur Jónsson Senior Software Architecht (CCP Games) Most](https://reader036.vdocuments.us/reader036/viewer/2022062417/551afa3155034607418b462a/html5/thumbnails/23.jpg)
Example code:
![Page 24: The Server Technology Of Eve Online: How To Cope With 300,000 Players In One World Kristján Valur Jónsson Senior Software Architecht (CCP Games) Most](https://reader036.vdocuments.us/reader036/viewer/2022062417/551afa3155034607418b462a/html5/thumbnails/24.jpg)
Co-operative multitasking (demo)
import stacklessdef func(string): for i in xrange( 10000 ): print string stackless.schedule()
t = stackless.tasklet( func )s = stackless.tasklet( func )t(“foo”)s(“bar”)
![Page 25: The Server Technology Of Eve Online: How To Cope With 300,000 Players In One World Kristján Valur Jónsson Senior Software Architecht (CCP Games) Most](https://reader036.vdocuments.us/reader036/viewer/2022062417/551afa3155034607418b462a/html5/thumbnails/25.jpg)
Stackless python is cool
No more this:
![Page 26: The Server Technology Of Eve Online: How To Cope With 300,000 Players In One World Kristján Valur Jónsson Senior Software Architecht (CCP Games) Most](https://reader036.vdocuments.us/reader036/viewer/2022062417/551afa3155034607418b462a/html5/thumbnails/26.jpg)
Stackless python is cool
But this:
www.stackless.com
![Page 27: The Server Technology Of Eve Online: How To Cope With 300,000 Players In One World Kristján Valur Jónsson Senior Software Architecht (CCP Games) Most](https://reader036.vdocuments.us/reader036/viewer/2022062417/551afa3155034607418b462a/html5/thumbnails/27.jpg)
Destiny
Divide and conquer Space Partitioning dynamically calculates the causality bubble for each ship in a fast way
Deterministic Allows state to be evaluated by the EVE client, knowing only initial state and time
Differential Equations Ships are essentially balls rolling around in vector fields
![Page 28: The Server Technology Of Eve Online: How To Cope With 300,000 Players In One World Kristján Valur Jónsson Senior Software Architecht (CCP Games) Most](https://reader036.vdocuments.us/reader036/viewer/2022062417/551afa3155034607418b462a/html5/thumbnails/28.jpg)
Hierarchical space partitioning
Binary space partitioning in three dimensions. Each sphere belongs to only one box – the smallest box that is able to contain the spere.
ttt STfS 1
NttMttt SfSfSSS 01
011 ,,
becomes:
![Page 29: The Server Technology Of Eve Online: How To Cope With 300,000 Players In One World Kristján Valur Jónsson Senior Software Architecht (CCP Games) Most](https://reader036.vdocuments.us/reader036/viewer/2022062417/551afa3155034607418b462a/html5/thumbnails/29.jpg)
But it is more complex than that
• The time-extrusion of a sphere determines what space partitions it can interact with (causality bubble)
• Collision is defined as the intersection of two time-extruded spheres (cylinders with dome shaped end-caps)
• Collision response is highly elastic (conservation of momentum)
![Page 30: The Server Technology Of Eve Online: How To Cope With 300,000 Players In One World Kristján Valur Jónsson Senior Software Architecht (CCP Games) Most](https://reader036.vdocuments.us/reader036/viewer/2022062417/551afa3155034607418b462a/html5/thumbnails/30.jpg)
And here the details:
QED
![Page 31: The Server Technology Of Eve Online: How To Cope With 300,000 Players In One World Kristján Valur Jónsson Senior Software Architecht (CCP Games) Most](https://reader036.vdocuments.us/reader036/viewer/2022062417/551afa3155034607418b462a/html5/thumbnails/31.jpg)
Evolving time
• The server solves the system of equations for an entire “system”.• The client only solves the equations for the causality bubble that it
belongs to.
Server regularly sends updates to the initial conditions for a time step. The client can work its way backwards to adjust for changes in initial conditions and derive how they affect current state.
![Page 32: The Server Technology Of Eve Online: How To Cope With 300,000 Players In One World Kristján Valur Jónsson Senior Software Architecht (CCP Games) Most](https://reader036.vdocuments.us/reader036/viewer/2022062417/551afa3155034607418b462a/html5/thumbnails/32.jpg)
This gives defines scaling:
• A client scales based on the population in a causality bubble
• A server scales based on the population in the “system” that we need to solve the differential equations for
• How are these concepts defined in EVE?• Answers after this break!
![Page 33: The Server Technology Of Eve Online: How To Cope With 300,000 Players In One World Kristján Valur Jónsson Senior Software Architecht (CCP Games) Most](https://reader036.vdocuments.us/reader036/viewer/2022062417/551afa3155034607418b462a/html5/thumbnails/33.jpg)
Movie
![Page 34: The Server Technology Of Eve Online: How To Cope With 300,000 Players In One World Kristján Valur Jónsson Senior Software Architecht (CCP Games) Most](https://reader036.vdocuments.us/reader036/viewer/2022062417/551afa3155034607418b462a/html5/thumbnails/34.jpg)
The EVE Universe
Solar systems typically consists of a dozen of planets, surrounded by moons and asteroid belts
Routes connect them into constellations that themselves form conglomerates called regions
EVE consists of over 5000 solar systems
![Page 35: The Server Technology Of Eve Online: How To Cope With 300,000 Players In One World Kristján Valur Jónsson Senior Software Architecht (CCP Games) Most](https://reader036.vdocuments.us/reader036/viewer/2022062417/551afa3155034607418b462a/html5/thumbnails/35.jpg)
Creating solar systems
EVE solar systems are created using the disc accretion model
Gives convincing enough results and is possible to calculate them all in less than 24 hours
![Page 36: The Server Technology Of Eve Online: How To Cope With 300,000 Players In One World Kristján Valur Jónsson Senior Software Architecht (CCP Games) Most](https://reader036.vdocuments.us/reader036/viewer/2022062417/551afa3155034607418b462a/html5/thumbnails/36.jpg)
Large scale design
• The large scale shape of the eve universe is defined first and foremost by Routes.
• Routes define how pilots can travel between systems.
• Position of the planets is derived from this.
![Page 37: The Server Technology Of Eve Online: How To Cope With 300,000 Players In One World Kristján Valur Jónsson Senior Software Architecht (CCP Games) Most](https://reader036.vdocuments.us/reader036/viewer/2022062417/551afa3155034607418b462a/html5/thumbnails/37.jpg)
Routes make it interesting!
Designing the layout, we wanted interesting properties:• Settlements• Stratetic value• Hierarcical properties
![Page 38: The Server Technology Of Eve Online: How To Cope With 300,000 Players In One World Kristján Valur Jónsson Senior Software Architecht (CCP Games) Most](https://reader036.vdocuments.us/reader036/viewer/2022062417/551afa3155034607418b462a/html5/thumbnails/38.jpg)
Diffusion Limited Aggregation
• Successfully used to model various growth processes
• Extended to work in 3 dimensions and have multiple seeds.
![Page 39: The Server Technology Of Eve Online: How To Cope With 300,000 Players In One World Kristján Valur Jónsson Senior Software Architecht (CCP Games) Most](https://reader036.vdocuments.us/reader036/viewer/2022062417/551afa3155034607418b462a/html5/thumbnails/39.jpg)
![Page 40: The Server Technology Of Eve Online: How To Cope With 300,000 Players In One World Kristján Valur Jónsson Senior Software Architecht (CCP Games) Most](https://reader036.vdocuments.us/reader036/viewer/2022062417/551afa3155034607418b462a/html5/thumbnails/40.jpg)
Implications for scalability
• The smallest unit of scaling is a solar system• Some system are busier than others
– Pass-through systems– Strategic access to trade
• Occasional tuning of routes necessary
![Page 41: The Server Technology Of Eve Online: How To Cope With 300,000 Players In One World Kristján Valur Jónsson Senior Software Architecht (CCP Games) Most](https://reader036.vdocuments.us/reader036/viewer/2022062417/551afa3155034607418b462a/html5/thumbnails/41.jpg)
Scaling with increased hardware
• A unique combination of software and hardware layers
• Tiered architecture allows software to scale well with increased hardware
• Make maximum use of worker threads and IO Completion ports underneath micro-threaded Stackless Python logic
![Page 42: The Server Technology Of Eve Online: How To Cope With 300,000 Players In One World Kristján Valur Jónsson Senior Software Architecht (CCP Games) Most](https://reader036.vdocuments.us/reader036/viewer/2022062417/551afa3155034607418b462a/html5/thumbnails/42.jpg)
![Page 43: The Server Technology Of Eve Online: How To Cope With 300,000 Players In One World Kristján Valur Jónsson Senior Software Architecht (CCP Games) Most](https://reader036.vdocuments.us/reader036/viewer/2022062417/551afa3155034607418b462a/html5/thumbnails/43.jpg)
![Page 44: The Server Technology Of Eve Online: How To Cope With 300,000 Players In One World Kristján Valur Jónsson Senior Software Architecht (CCP Games) Most](https://reader036.vdocuments.us/reader036/viewer/2022062417/551afa3155034607418b462a/html5/thumbnails/44.jpg)
Adding more hardware
• Architecture can be split up and distributed to more hardware:– Solar systems– Space stations– Market regions– Corporation services– Proxy services…
![Page 45: The Server Technology Of Eve Online: How To Cope With 300,000 Players In One World Kristján Valur Jónsson Senior Software Architecht (CCP Games) Most](https://reader036.vdocuments.us/reader036/viewer/2022062417/551afa3155034607418b462a/html5/thumbnails/45.jpg)
EVE runs on a supercomputer!
Active servers ~195
Weight 2.5T
#CPUs >420 cores, >1 THz
RAM >7.6Tb
FLOPS >7.5TFlops
Bandwidth ~ 400Mbps
![Page 46: The Server Technology Of Eve Online: How To Cope With 300,000 Players In One World Kristján Valur Jónsson Senior Software Architecht (CCP Games) Most](https://reader036.vdocuments.us/reader036/viewer/2022062417/551afa3155034607418b462a/html5/thumbnails/46.jpg)
Summary
• Structure your application in ways that promote scalability– Make it as granular as possible, so that load balancing is
more efficient, and you can add more hardware.
• Chose a powerful, expressive programming language– Allows you to restructure key areas as needs arise with the
least possible engineering effort.
• Design your database system wisely– Be one step ahead, so that it doesn’t become a bottleneck
for scaling– You are using a DB, aren’t you?
![Page 47: The Server Technology Of Eve Online: How To Cope With 300,000 Players In One World Kristján Valur Jónsson Senior Software Architecht (CCP Games) Most](https://reader036.vdocuments.us/reader036/viewer/2022062417/551afa3155034607418b462a/html5/thumbnails/47.jpg)
Burt Reynolds says:
Always use a Database!
![Page 48: The Server Technology Of Eve Online: How To Cope With 300,000 Players In One World Kristján Valur Jónsson Senior Software Architecht (CCP Games) Most](https://reader036.vdocuments.us/reader036/viewer/2022062417/551afa3155034607418b462a/html5/thumbnails/48.jpg)
That’s it!
• Additional information:– http://www.eve-online.com– http://stackless.com– http://www.slideshare.net/Arbow/stackless-python-in-eve– http://us.pycon.org/2009/conference/schedule/event/91/– (google for “stackless python in eve”)