diseño juegos 15 level 15_ blindtesting _ game design concepts

1/6/2015 Level 15: Blindtesting | Game Design Concepts

https://gamedesignconcepts.wordpress.com/2009/08/17/level-15-blindtesting/ 1/6

Game Design Concepts

An experiment in game design and teaching

Level 15: Blindtesting

When your game is playtested without you (or another of the games designers) beingpersonally present to observe, it is called blindtesting. This is the subject we cover today.

Readings

No readings for today. As before, if you know of any relevant readings you have encountered,post it as comments to this blog post, or on Twitter with the #GDCU tag.

The Challenges with Blindtesting

As you might imagine, blindtesting has a lot of problems compared with a playtest that yourun in person:

Without you there, any issue the players run into is a show-stopper that prevents themfrom finishing.

Perhaps worse, the players may interpret the rules incorrectly and play anyway. If they areunaware they are playing wrong your test results may be skewed, and the testers wonteven know it.

Feedback is highly variable; as you have no doubt seen by now, some playtest groups arebetter than others. This problem is made worse when you are not present and cannot asktargeted questions.

You cannot observe in real-time. The only information you get is what the group can reportback to you in retrospect. The fidelity of information is much lower than when you arepresent.

Setting up a blindtest takes a bit more time. You cannot simply bring your prototype withyou to a friends house and play right then. You have to find a way to get a copy of yourgame into the hands of your testers, and then you have to leave. Then you have to wait whileyour testers play your game, on their schedule, and then finally get back to you with someresults. Even in the best case, blindtest results may take a day or two to get back to you,which is far more than an in-person playtest which you can conduct in a matter of minutesor hours.

Blindtesting is therefore limited in the type of information it can provide, and the schedule onwhich it can provide it.

Why Blindtesting?



If blindtesting is so limited, why bother with it at all? What use is it?

This is a technique we borrow from science. Awhile ago, some researchers noticed that theresults of an experiment would be different if the researchers are in the room. Sometimes, testsubjects would say what they thought the researchers wanted to hear rather than what theyactually thought. Sometimes, the researchers would give subtle non-verbal cues that neitherthey nor the subjects would consciously notice, but that would bias the results of the test. Forexample, if the experiment is a taste test between two leading soft drinks, if the researchersknow which drink is which and they know which one they want the test subjects to pick, theymight suddenly find that all of the test subjects are choosing the right drink but onlybecause the researchers are (accidentally) telling them what to choose!

The solution to this is the so-called double-blind experiment, where neither the researchersnor the test subjects know what the right answer is. This eliminates many sources ofaccidental bias, making the results of an experiment more valid.

Playtests share a lot in common with science experiments. In both cases there is a hypothesis(I think this game is fun), an experiment is designed (building a prototype), the experimentis run (the playtest), results are analyzed. The purpose of both is to find out more informationabout the inner workings of the system that you are studying.

Blindtests, then, are the only way to get a true idea of what your game is like in the wild that is, how players will react to it if they have just purchased it from a store and are playing itfor the first time, without a designer present at their table to answer questions.

When to Blindtest?

Because you can get so little information from blindtesting, it is not suitable for an early testwhen your game is in a rough state (the testers would likely run into problems, be unable tocontinue, and you would have spent a lot of time waiting for something that you could havefigured out much faster with an in-person playtest. Blindtesting is most suitable near the endof development, when you already have a high degree of confidence in your game.

The purpose of blindtesting is to catch the non-obvious problems that you may not be catchingin your in-person playtests, because you accidentally bias the results of the test by helpfullybeing available to answer questions. Even if your playtesters never have to ask you anything,the very knowledge that you could can sometimes make people relaxed enough to get throughthe rules, where they might otherwise get flustered and give up. Or, you might say some keypiece of information when introducing the players to the game (this is an auction/biddinggame) that is not written anywhere and clarifies a lot by creating some preconceived notionsin the minds of your testers. Without you present, youll see just how accurate your in-personplaytests are, and you may catch some surprising errors that you did not notice before.

Who to Blindtest With?

Most board games require multiple players. For practical reasons, most blindtests done withprofessional games are done with regular game groups. Some companies keep a list ofvolunteer groups and put out calls to their private list for playtesters; they will then send out



advance copies of their game in exchange for feedback. For our purposes, this is unhelpful,because most of you do not work at such a company and do not have a database of volunteersto call at a moments notice.

Ideally, your blindtest should be with people who are in the target audience for your game. Ifyoure making a childrens game, your blindtest should involve kids in your target age range(and perhaps their parents). If your game is made for people who are already avid Eurogameplayers, youd do well to find a group that plays these kinds of games regularly. And so on.

There are a few ways to find blindtesters for your Design Project in this course:

Your social network of friends, family, and colleagues. These people may be local, or theymay live in another city or even another country from you. Since you dont have to bepresent, at least this time, geography is not a barrier.

Other people who are taking this course. If you already have a local playtest group thatyouve used before, you can put out a call for help on Twitter or the forums(http://gamedesignconcepts.aceboard.com/). Offer a blindtest exchange: youll blindtesttheir game if they will return the favor with yours.

As a last resort, you can look outside of this course for other discussion forums or otheronline hangouts for board game designers and/or playtesters, and see if there is a methodfor recruiting blindtesters.

How to Blindtest?

With in-person tests you can afford to be a little bit sloppy. Maybe you accidentally left someof your game components at home, or some of your rules are a little unclear and needclarification. Its not ideal, but a playtest session that runs into problems can be salvaged whenyou are there. With blindtesting, you do not have this luxury, so you must be extra careful tomake sure that your testers have everything they need to give your game a proper playtest.This includes:

A complete set of game components. Double and triple check to make sure that you haveeverything the players need, together, in one package.

A list of everything that your package should contain, so that the testers can verify forthemselves that nothing was left out (and if it was, it will at least give them a clue of whatthey need to supply as replacements).

A complete set of rules describing how to play. (The components list can simply be part ofthe rules.)

A separate set of instructions on how to conduct the blindtest. Is there anything inparticular you are looking for the players to do? Do you just want them to see if they canplay through the game? Do you want to know if they find the game enjoyable? Do youwant them to try to find rules exploits and imbalances?

A final set of instructions on what should be done on conclusion of the playtest. Howshould the testers contact you (phone, email, etc.)? What should they tell you when theycontact you if you dont give a list of questions for them to answer, you will just getwhatever they happen to feel like telling you, making your results a lot less focused thanthey could be. Give some thought to what information would be the most useful to have,what kinds of feedback are most important to you and ask for it!



How do you get your package into the hands of the playtesters? If they are local to you, it is assimple as taking your playable prototype and handing it off. If your blindtesters are not local,this presents additional challenges. You have two options here.

First, you can mail your prototype to them through the postal service. Depending on wherethey are, this step alone can slow things down considerably (not to mention making it moreexpensive), so plan your schedule accordingly. If you want your prototype back, considerincluding a return package inside the main one, already addressed to you and with postagealready paid.

Second, you can handle everything over email. Send documents that can be printed out andassembled to make a playable prototype, and include instructions on how to print everything.Do everything you can to reduce the amount of work that must be done at the playtesters endof things. After all, you are already asking for their time in the form of a playtest; asking foranother hour or two to print and cut sheets of cards is adding insult to injury. Along the samelines, try to keep the cost of materials down and availability high if youre expecting yourblindtesters to provide their own. For example, do not insist on printing on heavy card stock(which may require a special trip to a print shop) when printing on plain paper will do.

Contingency Planning

In the field, game companies do not rely on a single blindtest group, but several. Aside fromthe obvious reason that more tests give more data, there is also the problem that blindtestgroups are unreliable. Without you in the same room, they may take awhile to organize aplaytest, and they may take even longer to get back to you. Some groups may lose thecomponents, or forget about their obligation, or they may simply be so busy with other thingsin their life that your game takes lower priority. Or maybe youll send your game through thepost and it will get lost. In this course, when you set up a blindtest, you may find yourself inthe frustrating situation of waiting for test results that are simply not coming or at least, theymay not arrive within the schedule you set for yourself.

You have three options for dealing with this potential hazard. It is up to you which methodbest fits your personal situation.

1. You can set up multiple blindtest groups. As a rule of thumb, three is a fairly safe number.That way, if one or two groups dont show, youll at least have some results. (Note that ifyou are doing a blindtest exchange with other participants of this course, that meansyoull be testing three other projects, so make sure you have time for this.)

2. You can choose a blindtester that you know and trust to be reliable. If you are sending yourgame to a friend who is highly organized and always keeps their promises, you may haveconfidence that they will get back to you when they say they will.

3. You can cross your fingers and hope that something bad wont happen to you. This thirdmethod is not recommended for professional projects.

Homeplay

Your homeplay this past Thursday was to arrange for a playtest session with non-designers.You may have already performed this playtest, or you may have just scheduled it to take placeover the first part of this week, but that playtest session should be concluded before thisThursday, August 20, noon GMT.



In addition, you should find a blindtest group and arrange for a blindtesting session, to takeplace after the non-designer playtest. If possible, you should plan to have blindtest results onor before Thursday, August 27.

Feedback

Do you know of any great articles on blindtesting? As you conduct your own session, oryoure your own personal experience if youve done this before, have you found any helpfultips or tricks that youd like to share? Post in the comments on this blog, or on Twitter with the#GDCU tag.

This entry was posted on August 17, 2009 at 7:00 am and is filed under Uncategorized. You canfollow any responses to this entry through the RSS 2.0 feed. You can leave a response, ortrackback from your own site.

4 Responses to Level 15: Blindtesting

agj Says: August 17, 2009 at 8:23 pm | ReplyBlindtesting is most suitable near the end of development, when you already have a highdegree of confidence in your game.

Im sad to say that Im nowhere near that stage with my game yet. In fact, I have not movedpast the testing by myself stage, since I just keep having to make modifications.

Doctor Mike Reddy Says: August 18, 2009 at 4:20 am | ReplyI would imagine that many people are in your position. Ideally, development of a gameshould stick to the rate of progress implied by the bi-weekly schedule. However, I dontthink that Ian would expect everyone to be able to do this. It is important for the content ofthe course to reflect the order and importance of the various stages of development,particularly iteration after evaluation. As you have discovered, some are more intense/timeconsuming than others. At least now you can understand why games get delayed sofrequently. So, a learning experience!

And remember Ogdens Law (which was cited in the Dedication of my 11 year long PhD!):

The sooner you fall behind, the more time you have to catch up!

Delfar Says: August 18, 2009 at 11:01 am | ReplyI feel too that the course is too fast for me, but I like it too much. I will take the time to tryeveryone step that is mencioned here. But I would need three months holidays to do thecourse as is thought to.

Isi Says: August 18, 2009 at 5:42 pm | Reply



For Blindtesting, I think Vassal (http://www.vassalengine.org) can be a useful tool, but Ihave never used and I dont know if it costs a lot to make a prototype using it. Doesanybody used it?

Thanks

Isi

The Kubrick Theme. Blog at WordPress.com. Entries (RSS) and Comments (RSS).

diseño juegos 15 level 15_ blindtesting _ game design concepts

Documents