1  

Versioning Everything with Perforce

Halldor Fannar CTO CCP

Logo area

2  

Facts and figures

Released in 2003 for Windows PC Grown year-on-year to 500,000+ subscribers Record PCU of 63,170 players

Company founded in 1997 Privately held Over 500 employees worldwide

Released 2012 on Sony PS3 (in beta) Free-to-play with microtransactions

3  

Connecting Dust and Eve

Play short movie showing DUST & EVE

4  

Perforce server is our Definitive Software Library

• Art assets (audio, textures, geometry, shaders, etc) • Built binaries (executables, dynamic libs, static data) • External libraries and tools • Configuration files (settings, ini files, etc) • Text (UI messages, translations, etc)

Branching is used where it makes sense!

5  

Versioning art assets

//depot/content/eve //depot/games/branches

model.mb image1.psd image2.psd image3.psd

EVE-­‐TRILAMBDA  

[file sizes ~100 MB]

model.gr2 image1.dds image2_and_3.dds

[file sizes ~1 MB] settings.red

6  

•  Ar*sts  want  technology  that  is  user  friendly  and  provides    seamless  workflow  

•  Visual  tools  p4win  &  p4v  are  good  but  …  •  Crea*ng/managing  client  specs  is  a  burden  •  Selec*ng  the  right  folder/branch  to  sync  is  a  burden  

•  Without  any  support  this  is  how  they  make  it  “seamless”:  •  Create  a  “god  spec”  •  Sync  the  root  folder  

Artists and versioning

7  

•  The  problem  isn’t  restricted  to  ar*sts  –  we  have  3rd  party  tools  and  libraries  that  we  don’t  branch  (e.g.  //depot/games/sdk)  

•  Temp*ng  to  go  for  a  ‘god  spec’  in  such  an  environment  

•  Because  syncing  from  two  loca*ons  is  annoying  •  Because  crea*ng  new  client  specs  is  annoying  

→  We  remove  those  annoying  bits.  

Shared and branched files

8  

Streamlining Perforce user experience

9  

Stream-like branching model

MAIN  

EVE-­‐DX11  

EVE-­‐TRILAMBDA  

EVE-­‐JAPAN  

EVE-­‐TRANQUILITY  EVE-­‐2012-­‐R2  

EVE-­‐2012-­‐R2-­‐ALBERTO  

EVE-­‐2012-­‐R2-­‐BERYL  

sandbox   staging   release  

10  

Perforce streams are popular on smaller projects

//eveLauncher

11  

Perforce streams are popular on smaller projects

//virtualGoodsService

12  

No one size fits all

13  

•  Treat  all  your  configura*on  values  as  code  and  version  them  in  source  control  

•  Otherwise  you  don’t  really  have  a  Defini*ve  So[ware  Library  

Versioning your configuration

14  

Nobody is born a versioning hero – you become one after a freak software accident

15  

•  Myth:  “Built  binaries  have  no  place  in  source  control”  •  Usual  reasoning:  

•  Because  it  takes  up  so  much  space  •  Because  your  build  process  should  be  perfectly  repeatable  and  you  

are  then  storing  redundant  informa*on  

•  Perforce  is  our  Defini*ve  So[ware  Library  (DSL)  for  development  –  don’t  spread  your  development  so[ware  across  Perforce,  file  share  folders,  [p  sites,  and  a  bunch  of  DVDs  

Versioning built binaries

16  

Example: Python packages

•  Checked into //pythonPackages •  Automagically appear on http://pypi •  Bi-winning:

•  Makes the desired behavior convenient •  Offers two deployment models

17  

Example: External libraries

•  Make  Perforce  your  one  stop  shop  for  SDKs  •  Applying  that  policy  to  the  reference  PNG  library,  libpng:  

Transform  with  a  branch  spec:  sdk/libpng<-­‐sdk_installs/libpng  

18  

Example: External library requiring more tricks

•  Applying  our  policy  to  Microso[’s  DirectX  is  more  challenging  •  Crea*ng  the  “easy  to  consume”  compile  *me  environment  is  easy  

19  

Example: External library requiring more tricks

•  Crea*ng  the  right  run*me  environment  is  the  tricky  part  (system  lib)  •  O[en  developers  will  ask  you  to  manually  do  this:  

20  

Example: External library requiring more tricks

•  Create  a  bootstrapping  sequence  for  automa*c  silent  installa*on:  

Import  failure  

Import  library  that  depends  on  run*me  

•  Store  the  silent  installer  in  Perforce  for  each  version  of  the  SDK  •  Developers  can  just  sync  and  run!  (same  flow  for  our  customers)  

no  

Execute  silent  installer  for  run*me  

yes  

21  

Yes, it’s extra work but every developer will thank you Actually, they won’t because they won’t notice a thing

22  

•  “Sta*c  data”  covers  all  game  related  data  that  doesn’t  change  during  play  and  isn’t  captured  in  industry  standard  formats  

•  Typically  this  data  describes  the  construc*on  of  an  object  or  a  game  level  by  referring  to  external  binary  files  and  providing  specific  sekngs  for  how  they  are  combined  

Versioning static data

23  

•  2002:  Sta*c  data  is  stored  unversioned  in  an  MS  SQL  DB  •  2003-­‐2011:  Explosive  growth  and  explosive  issues  with  this  approach  •  2012:  Versioning  of  sta*c  data  moves  over  to  Perforce  

Abridged history of static data at CCP

24  

•  Syntax  for  code  well  understood  and  helps  us  with  resolving  conflicts  and  merging  

•  Compiler  can  most  o[en  tell  us  when  we  botched  the  resolve  or  merge  •  File  granularity  for  code  projects  is  usually  reasonable  and  known  

upfront,  thus  reducing  chances  of  conflic*ng  changes  •  Organiza*on  within  a  code  file  follows  rules  and  conven*ons  that  also  

reduce  chances  of  conflicts  

How  does  this  map  to  sta9c  data?    

Code versus Static data

25  

•  It  doesn’t  map!  •  Using  a  typical  text  merging  tool  on  XML-­‐like  data  will  most  o[en  end  in  

tears  

Code versus Static data

26  

products: - sku : BL394D quantity : 4 description : Basketball price : 450.00 - sku : BL4438H quantity : 1 description : Super Hoop price : 2392.00

•  We  use  YAML  (less  cru[)    

Stable  ordering  of  items  (sets  rather  than  lists)  

Uniquely  iden*fiable  objects  

Making static data merge-able

27  

Custom merge and diff tool

28  

•  Mul*ple  smaller  files  reduce  chances  of  resolving  –  but  inefficient  for  the  game  to  load  and  tough  on  your  file  system  

•  Use  automated  build  steps  to  transform  the  data  from  the  authoring/merge-­‐able  format  to  an  op*mal  run*me  format  (also  versioned  in  Perforce)  

•  Build  process  also  verifies  the  referen*al  integrity  of  the  data  across  files  and  validates  that  each  file  is  not  viola*ng  its  schema  

Solving other static data issues

29  

What about dynamic data?

•  Of course stored in a DB •  No branching support so changes must be backwards

compatible •  Schema and stored procedures are versioned in

Perforce •  CI servers monitor all changes and can verify

backwards compatibility

30  

Catch the problem at the source - not downstream

31  

Versioning everything requires an investment It’s worth it.