OOnn--DDeemmaanndd DDaattaa DDoowwnnllooaadd

1.0

Darkhand Studio T3D Extension November 2009

1

CONTENTS

INTRODUCTION ......................................................................................................................................... 3

PRINCIPLES ............................................................................................................................................... 4

APPLICATION PACKAGE ......................................................................................................................... 4

ODDE MANAGER .............................................................................................................................. 4

DEFAULT PACKAGE GROUPS ............................................................................................................. 5

APPLICATION ATTRIBUTES................................................................................................................ 5

DATA STREAMING ................................................................................................................................. 5

DELAYED DATA LOADING ...................................................................................................................... 6

ON DEMAND DATA MECHANISM ....................................................................................................... 7

ODDE GHOST BLOCKER OBJECT ...................................................................................................... 7

REMARKS FOR THE SERVER-SIDE APPLICATION .................................................................................. 8

NET API .............................................................................................................................................. 8

EDITORS ................................................................................................................................................. 10

OBJECT BINDER EDITOR ...................................................................................................................... 10

BINDER CREATION .......................................................................................................................... 11

BINDER EDITOR .............................................................................................................................. 11

APPLICATION PACKAGE EDITOR .......................................................................................................... 13

MAIN SERVER SETTINGS ................................................................................................................. 13

PACKAGE SERVER SETTINGS ........................................................................................................... 14

CREATION OF PACKAGE .................................................................................................................. 14

EDITION OF PACKAGE ..................................................................................................................... 14

UPDATE OF PACKAGE ..................................................................................................................... 16

ODDE GHOST BLOCKER EDITOR....................................................................................................... 17

PACKAGING EDITOR............................................................................................................................ 18

UPLOAD FILES TO SERVER ............................................................................................................... 18

C++ API ................................................................................................................................................. 20

HOW TO COMPILE................................................................................................................................ 20

C++ CLASSES AND API....................................................................................................................... 20

2

T3D SCRIPT API ..................................................................................................................................... 21

MAIN PACKAGE .................................................................................................................................. 21

WORKFLOW ........................................................................................................................................ 21

API .................................................................................................................................................... 21

ARTIST ................................................................................................................................................... 22

HOW TO EDIT MISSION ........................................................................................................................ 22

HOW TO PACKAGE MISSION ................................................................................................................ 22

3

INTRODUCTION

On Demand Data Download (referred as ODDE in this document) is an extension to T3D which allows T3D application to request for files of the running application when needed. It embeds 3 major mechanisms:

• File package management with either on demand file request, or immediately • Mission object delayed loading mechanism for all their art assets depending on

player / camera position • File upload management

This mechanism will allow any T3D application 1. to start with a small set of script / art files, then 2. to download art assets for mission when player / camera is closed to objects and 3. to interact with web servers (upload / download of information from application to servers).

4

PRINCIPLES

This chapter will present the different principles put in place inside ODDE.

APPLICATION PACKAGE

Application package is a list of all application files which will require to be downloaded from a server either immediately as soon as the application starts or later when required. It is important to note that the server download will be triggered only if a file is missing and its checksum is not consistent with the information inside the package file. This allows management of version and update of files quite automatically.

Application package (.pk file) is stored inside ~/cache/<application name>.pk and can be created inside the editor. A special editor is available in order to setup all important attributes of files listed inside the package.

A file package description consists then of:

• File group which will listed set of files; those group can be either downloaded automatically as soon as application is starting; or on demand when required during the application flow.

• Files inside group where important attributes are their name and checksum in order to check that files are correctly downloaded on the client and also to manage different release. Moreover it is possible to distinguish mandatory files and not mandatory ones. This is important in case of download error as errors on mandatory files will put the application in an error mode.

It is important to note that compression is handled (toggle available in the package definition editor) in order to limit number of bytes to transfer from server to client. It is also important to note that the server can be completely different than your application server as supported protocols for data transfer are either HTTP or FTP (FTP is the recommended one thanks to its speed).

ODDE MANAGER

One object ‘ODDEManager’ is available for all application. This object is the one which is managing all files request linked to the package definition. It is used internally by the object for the ‘Delayed Data mechanism’ and also inside the scripts for package / file request. It manages automatically the connection threads and also all the error management (retries of download, mandatory files…). This object is created when application starts and only one object is available for one application instance.

ODDE Manager is handling errors on mandatory files, retries on those files, balance of connection to server in case of errors, files to be downloaded and files available, and so on. ODDE Manager is also taking care of executing all scripts that are linked to a particular group package when this package is finished to download, this feature is important in case of materials, datablocks and so on.

5

DEFAULT PACKAGE GROUPS

Already four different package groups are required when you are defining a new package description. Those five package groups are the following:

• logics: this group should listed all the important scripts / arts that are mandatory for your application to start and at least go to the main menu. This package should be tagged immediate.

• levels: this group should list your mission file description (.mis file) and important files you will like to attach with the mission, like a small screenshot of the mission for example

• assets: this group should list all the arts assets / scripts that are mandatory for your mission to be available before the mission is loaded. This can regroup skyboxes, player model…

• data: this group all the art / scripts which will be loaded on demand when required by specific objects inside the mission

• binder: this groups all the binding xml file (see paragraph on delayed data loading)

If using the scripts files inside the ODDE template, those five package groups are required as they will be used by the scripts inside the template.

APPLICATION ATTRIBUTES

With the ODDE extension, an application has four new attributes:

• Application Name: this name is used for cached files and package • Application Server: this is the server from which it is possible to download the

package definition • Application CRC: this is the package checksum which will be used in order to know

if package need to be reloaded • Application Key: this is a key used for the package encryption mechanism

Those attributes are important as they will define which package file to load, from where and how to check that the package file is correct. Those attributes can be set-up either in scripts or through the web plugin. A correct use of those attributes inside a web plugin will allow management of package release and automatic update of application files as application will always download a new package as soon as its checksum is different than file on disk.

When using the web plugin, it is possible through these four attributes to have different games with the same plugin and then upload the required files for each different game as long as the main application dll can covers all game features. A specific extension to offer better management of web plugin will be available.

DATA STREAMING

6

Data streaming is managed through a direct interface with the Net API (see below for a description of the Net API). This interface allows multiple simultaneous downloads / uploads without any interruption of application execution.

Data streaming can be done either through the package manager (see below) or directly through the ICurl interface. The interface can manage a unlimited open connections and is completely thread-safe with regards to script execution, memory allocation and so on.

Connections can be reused in order to speed up the data transfer and to not reinitialize the connection with targeted server. This is an important feature in order to maintain a steady download / upload rate.

Connections can also be compressed meaning all the data transfer are either compressed before uploaded or uncompressed after download, thus optimizing bandwidth.

Data streaming can be achieved through callback attached to the ICurl interface and current implementation supports either file transfer or memory transfer. One class can benefit from this interface just by deriving this interface (this is used inside ODDE implementation for both package download manager and upload / download file console interface).

Upload / download status can be interactively checked through the ODDE Data Streaming Statistics window, which is activated by CTRL + F5 keys combination.

Figure 1: Data Streaming Statistics popup

DELAYED DATA LOADING

T3D ICurl API

Net

7

For all mission objects listed in the following table, it is possible to delay the use of their art assets, and to download those art assets on demand. This feature will then allow a mission definition file to load correctly even if the materials / models / textures are not present on the disk; and afterwards to download those materials / models / textures when required in the background without stopping the player experience inside the game.

Current mission objects that support this delayed data loading mechanism are listed in the following table.

TSStatic FxReplicator StaticShape Water

GroundCover AIPlayer

Table 1: Objects with delayed data loading support

ON DEMAND DATA MECHANISM

In order to activate this feature to any of listed objects above, three actions are required:

• Need to create the binder file description for that object. This binder file (xml format) is a file listing all the important data that are required for that particular object. In order to create it, an option is available in the editor.

• Need to toggle the checkbox ‘On Demand Data’ stating that this object will now be managed differently than other objects and its assets will be created after mission loading.

• Need to select how to create its assets: either immediately after the mission is loading, application is looking for the data and if not available will download them from a server, or depending on the distance with the player / camera, in that case object will be created, and data downloading accordingly, when player / camera is close enough to its bounding-box (close to a selected radius).

An editor is available to list all objects which support this mechanism, and it offers also means to edit the different parameters linked to this feature like threshold radius and so on.

Important note: in order to have this feature completely available, all binder xml file linked to mission objects must be included inside the application package description (see previous paragraph). The binder xml files are save inside a folder named ~/cache/<application name>/###.xml.

ODDE GHOST BLOCKER OBJECT

T3D is managing what is called ‘Ghost’ on the client-side which are the instance of one object on the server. Some object will be ‘ghosted’ when player is closed to them, so are ‘ghosted’ at the beginning of the mission. Those particular objects are called ‘ghost always’ objects and are for example the static shapes. This can bring two issues when dealing with large mission file:

It is important to note that for the moment there are no priority lists based on distance with player to order the object creation of those objects that are inside the player radius. It means that if this radius is quite large some objects far from the player may be created first compared to closer objects.

8

• Long delay during the connection with the server as the server will have to upload all the ghost object information before the mission can actually start. This is done inside the main thread and thus player must wait to have this process finished before starting to enjoy the game.

• Memory usage on the client can be important in case of large mission / world description with numerous objects.

In order to tackle these two drawbacks, but without removing the capabilities of having those kind of ‘always present’ objects, the odde ghost blocker object has been created. This object will manage differently all objects that are its children:

• All objects that are children of this object will have their ‘GhostAlways’ status removed and will be ghosted as usual with following rules

• As soon as camera / player is inside the bounding box of the odde ghost blocker, all objects children to it will be ghosted as usual; otherwise they will not be ghosted.

This object is available inside the mission editor.

This means that this object will allow to design large areas in the mission files which will be loaded on the client when the client enters the surroundings and un-loaded otherwise. A right combination of this feature and the delayed loading of art assets allows level designer to fully control how their level will be loaded on the client.

REMARKS FOR THE SERVER-SIDE APPLICATION

For the moment, on the server-side all objects are forced to download immediately the art assets if data are not present in the application folder. It means that all objects will be immediately created on the server. This is done mainly because of 1. there is no current management of player position on the server; thus impossible to know where a player is linked to the objects, 2. if an object is not created on the server collision are not handled and then player with object under loading may experience strange things (falling from ground…).

This particular limitation will be worked out soon, as soon as design for the multi-server support will be available as most of the collision-code will be managed on the client and very simple collision remaining on the server.

NET API

Regarding the NET API which is used internally for ODDE extension, it relies mainly on libCurl (http://curl.haxx.se/libcurl/) implementation inside T3D with some few enhancements. This implementation is done inside on interface named ICurl.

• Compressed stream transfer (for download / upload) using zlib (http://www.zlib.net) integration,

• Encryption of stream transfer (for download / upload) using Crypto++ (http://www.cryptopp.com/) integration,

• Support of unlimited parallel transfers (thread implementation) with priority list,

9

• Management of open connections (in order to quickly download numerous files from save server),

• Thread-safe execution of script callbacks from ICurl interface, • Thread-safe robustness added inside T3D for file management, script callback, script

events, script compilation and so on, • Support of libCurl features like proxy, http form and so on, • Thread naming (useful for debugging purpose).

Implementation is quite different than the one available on T3D resources (almost 70% rewritten from scratch) and as it is and interface it can be used in any other class.

Support for encryption is not yet finalized in current release and will be available only in next release.

Thread-support involved a lot of changes inside T3D engine not only linked to ICurl interface. Current supported T3D release is 1.0. Merges will be done on T3D 1.1 as soon as alpha release is over.

10

EDITORS

Few editors are available inside the main mission editor in order to help with the design of packages and object binder, linked with ODDE features. For the moment all those editors are embedded inside the T3D stock mission editor which can be opened by pressing F11 key.

Figure 2: Additional menu inside World Editor for ODDE

OBJECT BINDER EDITOR

Object Binder editor is available through the new On-Demand Data menu drop-down as illustrated by following figure.

Figure 3: ODDE Binder Editor Options

A separated editor with the mission editor will be available in next release.

11

BINDER CREATION

By selection the ‘Create Binding’ menu, the application will automatically creates all binding XML files for all mission objects which support delayed data download. A small popup will be displayed and by clicking the ‘Create’ button, all binding files will be created.

Figure 4: XML Binder popup

It is important to note that if some specific settings have been done through the Binder editor, those settings will be lost. All files will be saved as the ~/cache/<application name>/###.xml where ### is a unique hexadecimal number linked to each object.

BINDER EDITOR

The binder editor is opened by selecting the ‘Edit binding’ menu. It will open a popup as displayed here. This popup has three distinctive areas:

• One named ‘Objects’ listing all objects which can be managed for delayed data download,

• One named ‘Bind files’ which shows files that are bound to one selected object,

• One named ‘Properties’ which shows the ODDE Binding properties of that object.

OBJECTS GROUP

Object group lists all the mission objects which are supported by the ODDE Delayed Data Download features. It lists object id, object class and object name.

By clicking on the highlighted button, it will refresh the list of all objects inside this list. This is important in case new objects have been added inside the editor and are not listed inside this list.

BIND FILES GROUP

12

This group lists all files which are attached to selected object. It means in order to create the selected object; all those files will be required and downloaded from the server (or available on disk).

By clicking on the highlighted option, this will save the binding document displayed in this list. It mean that all changes in the file dependencies will be save accordingly inside the binding XML file linked to the selected object.

By clicking on the highlighted option, this will reinitialize the list of binding files from the object and then reinitialize the binding XML accordingly.

By clicking on the highlighted option, this will open a file browser in order to bind a new file (add a new file) with this object. This can be useful in case script files need to be downloaded at same time as all other assets for this particular object.

By clicking on the highlighted option, this will remove the selected file from the list of resources required for that particular object.

After changing the list of resources linked to an object, it is important to not forget saving the binding XML file otherwise all changes will be lost.

PROPERTIES GROUP

The properties group allows to set-up the five attributes which are linked on the ODDE Delayed Data Download feature. It is important to note that those attributes are saved inside the mission file as they are new attribute for all objects managed by ODDE:

• OnDemand: if selected, this object will be managed through the ODDE system and not through the T3D system for its creation,

• ImmediateAccess: if selected this object will be created as soon as possible after start of the mission,

• OnDemandRadius is the threshold pertaining to when an object will be created as soon as camera / player is close to its bounding box,

• OnDemand2DDist: is selected the threshold distance is computed only on the XY plane and not in 3D,

• OnDemandXML is the filename of the binding XML listing all resources required for this object.

By clicking on the highlighted option, this will set-up the same value for all ODDE attributes (except for the OnDemandXML)

13

to all objects listed in the editor. This is a helper feature in order to quickly set values for the ODDE objects. It is important afterwards to save the mission file in order to keep those settings.

APPLICATION PACKAGE EDITOR

Application package editor and functions are available through the On Demand Data menu dropdown as illustrated.

Figure 5: ODDE Package Editor options

MAIN SERVER SETTINGS

By selecting the ‘Set main server’ option, the following popup will appear. This popup is used to define the application information regarding the main server which is hosting the application package and its checksum as illustrated below.

Figure 6: Application Server information

Important to note that all those values are for the moment not stored in any files but should be either inside the webplugin or inside the main.cs file. A preference mechanism will be put in place in next release in order to store those parameters along with application name.

14

Passkey is not yet implemented and has no impact on the package settings. By clicking on the compute, CRC of the package will be displayed which is important as this value should be setup inside the web plugin.

PACKAGE SERVER SETTINGS

By selecting the ‘Set package server’ option, it is possible to setup information related to the package server, login and password to access this server and also compression / encryption scheme.

Figure 7: Package Server information

This server is important for the application as it is the one who will host all the required files. This information is stored inside the package file (.pk file). If compression checkbox is selected then all communication with the server will be done through compressed stream.

CREATION OF PACKAGE

The ‘Create package’ option will open the next popup which will automatically go through all files inside the <application name> folder and will add all those files inside a package group named “ungrouped”. This is a quick and dirty way of creating a package file.

Figure 8: Initialization of package information

EDITION OF PACKAGE

Encryption is not yet supported in current ODDE release.

It is important to note that in case of using template scripts, package requires at least five different groups.

15

‘Edit Package’ option will open the Package Manager window which will display all information linked to the current application package. This interface has five different areas:

• Application which shows the application name • Packages which shows all different package groups • Pck options which display the name and option of the package • Files will list all files belonging to one package group • File options which display attributes of the file linked to this package

APPLICATION INFORMATION

The application information allows to change the name of the application and to save also the application package file. Name of the application is important has the package filename is composed of this application name with .pk prefix attached to it. By clicking the highlighted button, the application package file is saved inside the ~/cache folder.

PACKAGES INFORMATION

This area lists all the different package groups which are present inside the package file definition. A package group can be seen as one repository and can be requested with its name through the ODDE Manager object to be downloaded.

By clicking the highlighted button, it is possible to create a new package. A window asking for the new package name will appear and then after selecting done, the new package group will be available.

On the opposite the following button will allow to remove one package group (and corresponding file from the package definition.

PACKAGE OPTIONS

The package option area allow to setup one major attribute which is to set either the package as an immediate download or not. If a package group is set as an immediate download, then it will be downloaded immediately before the application starts the game. This is important for the initial logics files (scripts / arts) that are mandatory before running the game, like those defining the main gui and so on.

16

By clicking the highlighted button, it is possible to setup the same option to all the package groups (immediate attribute only).

FILES INFORMATION

This area lists all files which are inside a selected group and thus will be downloaded when this package group will be requested.

By selecting the highlighted button, it is possible to move the selected file to another package group. A new popup will be displayed listing all different package groups and after selection the file will be move to the chosen package group.

After selecting this highlighted option, a folder dialog is opened. This allows adding all files belonging to selected folder to the current package group.

After selecting this highlighted option, a file dialog is opened. This allows adding selected file to the current package group.

This latest highlighted button can be used in order to remove a file (that should be selected in the file list) from the current package group.

FILE OPTION INFORMATION

The file option information lists name and mandatory checkbox linked to selected files in the file list. It is then possible to select if this file is mandatory or not. Not mandatory file will not put application in error in case of issues during the download.

By selecting the highlighted button, it will setup the ‘mandatory’ attribute to all files of this package group.

UPDATE OF PACKAGE

By selecting the ‘Refresh package’ option in the main menu, application will refresh all information inside the package. It means it will refresh:

• File size information, • File checksum information.

Sometimes in case of long list of files, it might happen that GUI seems frozen (no screen update). It is important to just wait as the screen will be refreshed few minutes later. This issue will be tackle in next release of ODDE.

17

It is important after an update of a package to save it through the package editor window or all changes will be lost.

ODDE GHOST BLOCKER EDITOR

OddeGhostBlocker is an utility object which helps to create area which will be loaded when player is inside the area. In order to create this object inside the mission, just select the Library / Level / System, it will display the SimGroup and odde Ghost Blocker objects.

Figure 9: System object list

After selecting the ‘odde Ghost Blocker’ object, it will open the following window where it will be possible to choose a name for that object, and also the bounding area which is used to trigger the object download.

Figure 10: oddeGhostBlocker popup editor

After creating the oddeGhostBlocker object, just add to it all objects required for this area definition, as illustrated below.

Figure 11: oddeGhostBlocker object children

18

PACKAGING EDITOR

Packaging editor is available through the option illustrated below from the main menu.

Figure 12: Packaging editor options

UPLOAD FILES TO SERVER

In order to have ODDE working correctly it is important to use the next GUI to upload all files defined in the package on the package server. Indeed, only current ODDE application can managed transfer of compressed data. So just fill the next interface with information regarding the package server (access, login and password) and after clicking ‘upload’, it will upload automatically all data to selected server.

Figure 13: Upload package GUI

Current release of ODDE only supports file upload. New release will support upload of package file and of file patches. For the moment it is important to copy manually the package file (.pk) on the defined server.

19

It is important to note that write / create folder permissions should be available for selected login / password account; otherwise the system will be unable to upload requested files.

20

C++ API

HOW TO COMPILE

ODDE Extension is directly available inside T3D thanks to T3D project generation scripts. After creating a new project using ODDE template, just open source code with VS2008.

ODDE Extension are either available inside the dk folder of source engine, or inside #ifdef DK_ODDE / #endif code segments.

Figure 14: Source code listing

C++ CLASSES AND API

Please look at the attached html files linked to this document.

21

T3D SCRIPT API

MAIN PACKAGE

All sources for the main package are inside the odde folder. This folder contains script source code and gui for the ODDE extension. Other important source code are inside the tools / oddeEditor folder and regroups all ODDE editor panels.

WORKFLOW

Next figure lists the general workflow of an application managed through ODDE. Important functions in order to setup application parameters are:

oddeSetAppName(string) To set up the name of the application

oddeSetAppKey(string) To set up the application key (for encryption purpose) – not yet supported

oddeSetAppServer(string) To set up the application package server information (from where it is possible to download the package)

oddeSetAppCRC(string) To set up the application checksum

API

Please look at the attached html files linked to this document.

22

ARTIST

HOW TO EDIT MISSION

Editing mission with ODDE support requires the following important points:

• To use as much as possible the oddeGhostBlocker object in order to define areas which will be loaded / unloaded depending on camera / player proximity,

• To add a script file for each objects which have specific materials (transparent materials) and to add this script file inside the binding XML file of appropriate objects,

• To select carefully mandatory files by using mandatory checkbox only for important script / gui files,

HOW TO PACKAGE MISSION

When packaging the mission, it is important to separate two kinds of mission objects:

• Required objects at mission start for which all materials should be available before the mission is starting.

• Objects which are managed by ODDE and for which materials can be downloaded after mission is loaded.

For the first category of objects, it is important to put all those object materials (models, arts, sounds, scripts…) inside the assets package group. For the other objects, they should be put inside the data package group.

Two other important package groups are:

• level where all level information (mission file, script and screenshot) should be placed,

• binder where all binding document should be placed.

Last important package group is the logics group which should regroups all the scripts / gui / arts which are mandatory to be loaded when application is starting.

As soon as the package is set-up, do not forget to save it; and then to copy the file ~/cache/<application name>.pk on the server setup in the script.