Towards an Application Programming Interface forAutomated Testing of Artificial Intelligence Agents inMassively Multi-Player On-Line Role-Playing Games

Markus Schatten, Bogdan Okreša Đurić, Igor TomičićArtificial Intelligence Laboratory

Faculty áf árganization ánd ánformatics, University of Zagreb Pavlinska á, á2000 áaraždin, ároatia

{markus.schatten, dokresa, igor.tomicic}@foi.hr

Abstract. An initial implementation of an applicationprogramming interface (API) for automated testingof artificial intelligence agents in massively multi-player on-line role-playing games (MMORPGs) ispresented and analyzed. The API, which is based onresults from the Large-Scale Multi-Agent Modellingof Massively Multi-Player On-Line Role-PlayingGames (ModelMMORPG) project, is implementedin Smart Python Agent Development Environ-ment (SPADE) and allows for the implementation ofartificial agents which are able to play The ManaWorld (TMW), an open source MMORPG. The APIprovides access to basic player behaviours includingbut not limited to movement, fight, pick-up of itemsand interaction with non-player characters (NPCs)as well as more advanced features like interactionwith other players through chat, creation of playerparties and trade. The basic proposition of the APIis to allow concurrent testing of various types ofartificial agents in playing the game which allows foragent behaviour analysis as well as gameplay analysis.

Keywords. application programming interface, auto-mated testing, artificial intelligence, massively multi-player on-line role-playing game

1 IntroductionThe gaming industry is recognized as a rapidly grow-ing domain spanning various intended uses, from en-tertainment and education, to programming and designindustries, to serious games and research. The im-portance and success of the gaming world is empha-sized by the Electronic Sports (eSports) movement aswell. As one of the identified application domains oflarge-scale multiagent systems (LSMASs), along withthe Internet of Things (IoT), smart cities, smart grid,complex systems and smart transportation (Schatten,Tomicic, and Ðuric, 2017), to name a few, the gam-ing industry gained a lot of attention in recent researchtrends, especially when MMORPGs are considered.

Other than being a good test-bed for technologies suchas virtual reality and augmented reality, the gaming do-main succeeded in becoming an effective environmentfor social studies as well.

Computer games have been a source of training datafor various approaches to development of a form of anartificial intelligence (AI) using different heuristics aswell as machine learning methods. These forms of spe-cialized AI were trained and developed for use withcomputer games that are of a simple point-and-click,first person shooter, or simple arcade game genre. Onlyrecently has the gaming industry turned towards de-veloping a form of a general AI, since games can bemodified quickly, with many re-runs of experiments.Such a short response and customization time sup-port the processes of learning when artificial agentsare considered. In this context, Peng et al. have pre-sented multiagent bidirectionally-coordinated network"with a vectorised extension of actor-critic formula-tion", which could "handle different types of combatsunder diverse terrains with arbitrary numbers of AIagents for both sides" (Peng et al., 2017). Authorsused StarCraft game as a test-bed scenario and demon-strated that their network facilitates learning of vari-ous types of agent coordination strategies. A litera-ture overview on AI techniques used in real-time strat-egy games was presented in (Robertson and Watson,2014), with a focus on a StarCraft game. Authors haveidentified the main areas of current research: tacticaland strategic decision making, plan recognition, andlearning. Another application on StarCraft game waspresented in (Synnaeve and Bessiere, 2016), where au-thors showed the use of Bayesian models in three dis-tinct core components: micro-management/units con-trol, tactics, and strategy, arguing the possibilities ofprobabilistic models on sources of uncertainty and in-completeness, which are inherent to the AIs in real-time strategy games.

This paper therefore presents initial steps towardsimplementation of an API for automated testing of AIagents in MMORPGs. Since testing an MMORPG isa tedious task that requires countless hours of game-

Proceedings of the Central European Conference on Information and Intelligent Systems______________________________________________________________________________________________________11

______________________________________________________________________________________________________29th CECIIS, September 19-21, 2018, Varaždin, Croatia

play, having it performed by an artificial agent greatlyincreases relevance of the received testing results, andthe sole amount of such testing results.

The rest of this paper is organized as follows: firstlyin section 2 we provide an overview of related work.Then in section 3 we show the architecture of the im-plemented API and in section 4 we discuss what ad-ditional steps have to be taken for the API to be com-plete. In the end in section 5 we draw our conclusionsand provide guidelines for future work.

2 Related WorkThe majority of related research is performed on sim-ple games that usually use only two dimensions, andare linear in gaming style - either it’s simply moving,collecting items, and avoiding obstacles, or almost onlyavoiding obstacles and moving in a single direction, orsomething similar.

The major distinctive difference between the pub-lished research and the research proposed in this paper,is the game genre that is put in the focus. While relatedresearch is focused primarily on games that are linearin the nature of their objectives, the API of this researchis being built for an MMORPG which is a part of therole-playing games (RPGs) genre that usually utilizesnon-linear story that is a combination of various questsand side-quests thus introducing a level of creativity ingameplay.

The topic of APIs for RPGs has received an increasein popularity when DeepMind and Blizzard (Vinyals etal., 2017) opened Blizzard’s StarCraft 2 game as an AIresearch environment. The published set of tools de-signed for AI research combines the API developed byBlizzard, and a modified open source version of Deep-Mind’s toolset that can be used to integrate Python-based agents with Blizzard’s API. The combination ofthe two, along with the additional elements of the setof tools, can freely be used by researchers and develop-ers in their AI-related pursuits. StarCraft 2 is a sequeland a modernized version of an old but highly popu-lar and valued real-time strategy (RTS) computer gamethat has secured its popularity as a long-time championin the domain of eSports. The cited research is an ex-ample and proof of the growing importance of com-puter games in research and their importance in AI de-velopment.

Another research on the topic of how RPGs can beused to further and enhance AI research is presentedin (Tian et al., 2017), where a research-oriented plat-form is presented that conforms to a defined set of fea-tures of an ideal game environment for fundamental re-inforced learning – one that is diverse in the context ofproperties of offered games, efficient in running sim-ulations, and provides highly customized environmentsettings. RTSs are used as a way of providing richer en-vironments for reinforced learning, ensuring that train-ing using e.g. raw pixel data is supported, since not

many RPGs exist that can be used for research directly(Tian et al., 2017). The research presented by Tian etal. (2017) therefore provides a platform for enhanceduse of reinforced learning, in the context of computergames of RTS genre, which is significantly differentthan RPG games that are the focus of this paper’s re-search.

Another take on the concept of machine learning,using visual input from a computer game for train-ing the AI, is associated with a game of first-personshooter (FPS) genre (Kempka et al., 2017). ViZDoomis purposed to be used for developing bots that playDoom, the classical FPS computer game released in1997. The game is played based on visual reinforcedlearning principles, with the software emulating ac-tions corresponding to key presses and mouse move-ments used to control the gameflow, and providing thedeveloper with some directly available in-game data.Doom, the computer game used in this context, is farmore simple than an average modern RPG since theprovided and requested gameplay is much simpler anddoes not require devised planning.

Classic machine learning approaches have been usedin development of artificial agents that can play so-called platformer games, such as Super Mario World,Super Mario Kart, or Pacman. While all of these exam-ples are interesting to observe from the point of view ofusing computer games in training an AI agent, they donot represent APIs by themselves.

Figure 1: MarI/O agent training, a screenshot fromYouTube

MarI/O1, shown on figure 1, is a piece of softwaredeveloped by an Internet user known as SethBling thatuses neural networks and generic algorithms to build anagent that is capable of successfully playing a simpleSuper Mario World game. The software interacts withthe game using emulated keystrokes, just like seen inViZDoom above. The use of visual input coupled with

1For further information, visit: https://www.youtube.com/watch?v=qv6UVOQ0F44, and https://pastebin.com/ZZmSNaHX

12______________________________________________________________________________________________________Proceedings of the Central European Conference on Information and Intelligent Systems

______________________________________________________________________________________________________29th CECIIS, September 19-21, 2018, Varaždin, Croatia

the use of recurrent neural networks is shown by thesame author on another example, working with a driv-ing computer game named Super Mario Kart – Mari-Flow2. As opposed to the earlier described example,this time the AI does not learn by genetic modifica-tions, but is fed input data in form of sample games.The last such example presented here, for the sake ofemphasizing the use of computer games in informationcommunication technology (ICT) research, is anothershowcase of neural network methods being used totrain and play a game, this time of Pacman3.

3 ImplementationThe importance and added value of the research laidout in this paper comes from the chosen applicationdomain – MMORPGs. The difference of such a genre,when compared to some of the examples presented inSection 2, is obvious for those accustomed to computergames.

FPS is the most simple genre, when strategizing isconsidered, as it is based on destroying as many oppo-nents as possible or demanded, and may be the simplestto implement, since the agent only needs to identify itstarget, and shoot at its most vulnerable point.

The genre of RTS is complex in terms of strategicthinking, but is mostly based on a single goal, pack-aged into successfully outrunning the opponent in thecontext of, usually, economic or militaristic prowessand power. While the possible ways of achieving vic-tory are diverse and mostly depend on the nature of thegiven player and the way they want to interact with thegame (e.g. emphasizing technological advancement, orbuilding their armies in numbers), the goals are seldommore complex than those derived from the aforemen-tioned context.

Finally, an RPG boasts a much more complex worldin general, since many more decisive elements arepresent, more so than in either FPS or RTS genre. Fur-thermore, role-playing games, being based on develop-ing a player’s character (often referred to as an avatar),are built around a story that helps develop this avatar,most often defined by their various traits, features, abil-ities, inventory, etc. The in-game storyline is not aslinear as that of a typical FPS or a typical RTS ei-ther. Although these two genre may provide a story-line to be followed during the game in extent, the gam-ing experience is heavily divided by levels that act asa whole. An RPG is more often modeled as an open-world game, allowing the players to roam the in-gameworld and choose objectives for themselves, while thegame in general is directed by a set of quests that de-mand interaction with concepts of the in-game world.Thus the possible set of actions and their utilization is

2For further information, visit https://www.youtube.com/watch?v=Ipi40cb_RsI

3For further information, visit: https://www.youtube.com/watch?v=t5--kLRI4UE

much larger than that of either of the aforementionedcomputer game genre.

In addition to computations in terms of devis-ing plans on how to solve a quest, the context ofMMORPGs demands solving social-level problemsand challenges as well, since socialization is an impor-tant and integral part of the massively multi-player partof this genre. More so, socialization is very importantin utilizing all the features of a MMORPG. Therefore,building an API for an MMORPG is not only about en-abling an agent to play a game and interact with the in-game world, but to interact with other players as well,which is an element that is not present in research pre-sented in Section 2.

During the ModelMMORPG project, which aimedon developing artificial agent organizations that areable to play MMORPGs, a byproduct was the de-velopment of an interface to a very particular opensource MMORPG called TMW. The interface wasimplemented in Python and comprises two parts: (1)a lower-level interface – dealing with particular low-level communication with the game server; as well asa (2) higher-level interface – dealing with the actualimplementation of a playing agent (Schatten, Tomicic,Ðuric, and Ivkovic, 2017; Schatten, Ðuric, Tomicic,et al., 2017; Schatten, Ðuric, Tomicic, et al., 2017).4

The architecture of the ModelMMORPG agent imple-mentation system is shown on figure 2, on which theMMORPG plug-in is the interface to TMW. Detailedmodel of both lower- and higher level interfaces isshown in figure 3.

The lower-level interface, shown on figure 4 imple-ments the network protocol of the TMW servers andemulates an actual client connecting to the server andplaying the game. It also implements a number of func-tions that allow actual gameplay including navigation(moving the avatar including automated navigation likemoving to some character or dropped item, followinga player or mob etc.), attack, sitting down, standingup, whispering communication to other players, pick-ing and dropping up items, equipping items, creatingparties of players (including inviting, responding to in-vitations, leaving parties and communicating on thegroup chat), environment sensing (what is my location,what is the location of nearby players/items/NPCs),and trading of items with other players, interacting withNPCs.

The higher-level interface builds upon the lower-level by introducing an agent template implementedin SPADE (Gregori et al., 2006) and adding addi-tional parts like a STRIPS-based planning system im-plemented in Prolog, as well as an agent knowledgebase implemented using the SPADE knowledge-basesystem for SWI Prolog. The higher-level interface actsas a layer around the low-level interface by providinghigher-level agent behaviours based on low-level func-

4The current version of the interface is available at https://github.com/tomicic/ModelMMORPG

Proceedings of the Central European Conference on Information and Intelligent Systems______________________________________________________________________________________________________13

______________________________________________________________________________________________________29th CECIIS, September 19-21, 2018, Varaždin, Croatia

Figure 2: Architecture of the ModelMMORPG agent implementation system (Schatten, Ðuric, Tomicic, et al.,2017)

Lower-Level interface

Gameplay functions

Movement

Attack

Equip item

Sensing

Trade

. . .

Network protocol

Clie

ntem

ulat

ion

Higher-Level Interface

SPADEagent

implementationtemplate

planning system

knowledge baseagent behaviours

Figure 3: Detailed model of lower- and higher-levelinterfaces and their featuers

tion including but not limited to: navigation (includ-ing random walks), NPC conversation handling, fighthandling, and party management. It is a typical belief-desire-intention (BDI) agent that firstly senses the envi-ronment, updates it’s knowledge base, chooses an ob-jective to accomplish next, generates a plan for this par-ticular objective and then starts executing it, while theobjective and the plan still make sense.

4 DiscussionIn its current state, the interface cannot be really calledan API, since a number of core functionality is miss-ing, most prominently adequate documentation on howto implement an agent able to play TMW including afew simple examples as well as the abstraction of corefunctions from the actual implementation of an agentto be reusable. Currently, the implementation featuresa finished agent prototype that is able to play some ofthe initial quests of the game, as well as create, joinor leave parties based on its assigned role. This pro-totype has to be dissected into parts that can be reusedfor new types of agents, possibly using different AI,

reasoning and reaction technology like machine learn-ing for example, since the prototype is based on BDIand STRIPS.

In order to do so, some basic rewriting of the coderelated to the high-level interface has to be done, forexample: all prototype related code that was used forlarge-scale experiments has to be moved into an exam-ple usage, while core functionality like random walks,creation of parties, conversation with NPCs and sim-ilar have to be retained for future use. On the otherhand, the low-level interface can stay more or less in-tact, since it is already used as a module for the high-level interface and acts as a low-level API.

Also, the planning system as well as the knowledgebase of the agent can be easily reused for future agents’implementation, but the interface towards them has tobe made more accessible for new types of agents.

Even though the API is to be used for testingMMORPGs in the context of their logical, quest, andstory soundness, and balanced gameplay, the basic usecase of the work presented and proposed in this pa-per is to develop testing scenarios whose details dependon the developer and testing needs. Therefore the suc-cess of testing largely depends on the game developersand testers, and the implementation details beyond thescope of this research which aims to provide functionsand agent behaviours necessary for successful interac-tion between agents and an MMORPG.

5 Conclusion

Herein we have presented an initial step towards theimplementation of an API for the automated testingof AI agents for MMORPGs based on some byprod-ucts of the ModelMMORPG project. We have imple-mented a lower-level interface, a higher-level interface,a STRIPS based automated planning system as well asa knowledge base for the future API that will allow forthe implementation of AI agents for the open sourceMMORPG TMW.

The most important contribution of the API is its fo-

14______________________________________________________________________________________________________Proceedings of the Central European Conference on Information and Intelligent Systems

______________________________________________________________________________________________________29th CECIIS, September 19-21, 2018, Varaždin, Croatia

Figure 4: Lower-level interface to TMW (Schatten, Ðuric, Tomicic, et al., 2017)

cus on a new genre, as opposed to current approaches.The RPG genre, and more recently the MMORPGgenre hasn’t been part of recent research for automatedtesting of AI agents, most probably due to the com-plexity of the domain, since such games include lots ofcomplex behaviour of players to be exhibited in orderto advance in the game.

Our future research will therefore be aimed towardscompleting the API by dissecting the prototype codeinto well established parts that can be reused in fu-ture AI agent implementation projects, by developinga number of simple AI agents for developers to startwith as well as more profound documentation.

AcknowledgmentsThis work has been fully supported by the CroatianScience Foundation under the project number 8537.

ReferencesGregori, M. E., Cámara, J. P., & Bada, G. A. (2006).

A jabber-based multi-agent system platform. InProceedings of the fifth international joint con-ference on autonomous agents and multiagentsystems (pp. 1282–1284). ACM.

Kempka, M., Wydmuch, M., Runc, G., Toczek, J.,& Jaskowski, W. (2017). ViZDoom: A Doom-based AI research platform for visual reinforce-ment learning. IEEE Conference on Computa-tonal Intelligence and Games, CIG. doi:10.1109/CIG.2016.7860433. arXiv: 1605.02097

Peng, P., Yuan, Q., Wen, Y., Yang, Y., Tang, Z.,Long, H., & Wang, J. (2017). Multiagentbidirectionally-coordinated nets for learning toplay starcraft combat games. arXiv preprintarXiv:1703.10069.

Robertson, G. & Watson, I. (2014). A review of real-time strategy game ai. AI Magazine, 35(4), 75–104.

Schatten, M., Ðuric, B. O., Tomicic, I., & Ivkovic,N. (2017). Agents as bots–an initial attempt to-wards model-driven mmorpg gameplay. In Inter-national conference on practical applications ofagents and multi-agent systems (pp. 246–258).Springer.

Schatten, M., Ðuric, B. O., Tomicic, I., & Ivkovic,N. (2017). Automated mmorpg testing–an agent-based approach. In International conference onpractical applications of agents and multi-agentsystems (pp. 359–363). Springer.

Schatten, M., Tomicic, I., & Ðuric, B. O. (2017). A re-view on application domains of large-scale mul-tiagent systems. In T. Hunjak, V. Kirinic, & M.Konecki (Eds.), Central european conference oninformation and intelligent systems (pp. 201–206).

Schatten, M., Tomicic, I., Ðuric, B. O., & Ivkovic, N.(2017). Towards an agent-based automated test-ing environment for massively multi-player roleplaying games. In Information and communica-tion technology, electronics and microelectron-ics (mipro), 2017 40th international conventionon (pp. 1149–1154). IEEE.

Synnaeve, G. & Bessiere, P. (2016, December). Mul-tiscale Bayesian Modeling for RTS Games: AnApplication to StarCraft AI. IEEE Transactionson Computational Intelligence and AI in Games,8(4), 338–350. doi:10 . 1109 / TCIAIG . 2015 .2487743

Tian, Y., Gong, Q., Shang, W., Wu, Y., & Zitnick, C. L.(2017). ELF: An Extensive, Lightweight andFlexible Research Platform for Real-time Strat-egy Games. (Nips), 1–14. arXiv: 1707.01067

Vinyals, O., Ewalds, T., Bartunov, S., Georgiev, P.,Vezhnevets, A. S., Yeo, M., . . . Tsing, R. (2017).StarCraft II: A New Challenge for ReinforcementLearning.

Proceedings of the Central European Conference on Information and Intelligent Systems______________________________________________________________________________________________________15

______________________________________________________________________________________________________29th CECIIS, September 19-21, 2018, Varaždin, Croatia