up down left right a usability postmortem tony rado – executive director global quality assurance,...

Up Down Left RightA Usability Postmortem

Tony Rado – Executive DirectorGlobal Quality Assurance, [email protected]

Paul Venuti – DirectorDevelopment Services, [email protected]

www.soedevservices.com

Why usability? Usability statistics Organizational relationships Benefits of in-house lab Challenges of in-house lab Usability lifecycle

5 Phases 18 month trial and error Lessons learned

Vivox (EQ, EQ2, SWG) Free Realms DCUO, Agency

Up, Down, Left, and Right. A Usability Postmortem

A Usability Postmortem

Why Usability?

Designers, Developers, Producers, and Executives watch target demographic play games

Increase time to market

Controlled environments

We didn’t have solution in place to extract critical information from target audience

Issues with beta testing:MisinformationCritics of gameNot targeting audience but accepting audience


Usability Stats - May 08 to August 09

44 Usability Tests

1 Focus Test

2 Time-to-Task Data Collection

Total of 473 Candidates

60 Candidates – Competitive Analyses

47 Candidates – Concurrent Vivox23 Candidates – Web Presence18 Candidates – The Agency

177 Candidates – Free Realms

7 Candidates – 3rd Party Contract

36 Candidates – Free Realms First Impression Focus Test

71 Candidates – Free Realms Combat and Racing Data Gathering/Balance

34 Candidates – DCUO


Organizational Relationships:

Development Services

Quality Assurance

Development

Marketing


Benefits of an In-House Usability Lab?

Quality equates to successQuality buy-in from all vested partiesReal time monitoringAbility to record A/V on site

EducativeNo limit on testing iterationsImmediate feedback / Immediate

resolution

“Best practices call for between 8% and 13% of project budget be spent on Usability” *Jakob Nielsen, Alertbox, 7 January 03

Not an ROI in classical senseProject cost measured in terms of moneyUsability measured in terms of increased usage, efficient use, high user satisfaction*


Challenges of Building In-House Usability Lab?

Changing CultureShrinking Budgets - $$$InfrastructureTechnological Challenges

Distance between lab and monitoring room:VideoAudio

Real time synced camera and game play feed support for satellite studios


[Embedded Video]


Consultation PhaseInitial and Ongoing Communication

Job requestPlanning meetingTechnological challengesDefine measurables

ISSUE: Requesting party disagreement:GoalsData pointsCannot please and appease everyone

RESOLUTION: Created one point of contact for which all communication funneled through

Lesson Learned


Recruitment Phase

Generalized recruitment categories for both Internal and External candidates include:

Age

Sex

Hr/Wk playing games (Online/PC/Console)

Favorite games

Specialized – 3rd party contract recruited an ethnic target for European/South American game


Internal & External CandidatesInternal Candidates

External Candidates

Cost Benefit – Recruitment and incentive costsInitial use of internal candidates were employees then employee childrenRecruited internal candidates with same requirements as if recruiting

externally

Free Realms iterative – Created database of employee children and friendsTreated employee’s children as external candidates

Recruited Boy Scout Troops, local kidsExhausted internal children pool and began external recruitingTime saved by utilization of external recruiting company


Candidate Bias

Internal

Caroline Jarrett, in regards to internal candidate bias, states that “they may be your target users, in which case they are perfect” *

**Direct observation doesn’t lie**

ExternalGetting paidDesire to please [Embedded Video]On-site testing

*Society of Technical Communications Usability Special Interest Group – March 28, 2009


Recruitment PhaseLessons Learned - Internal

Lessons Learned - External

ISSUE: Internal bias would taint data due to desire to please or be less than honest

RESOLUTION: Ensured all candidates understood:Setting of quality bar equated to job security!Pride in having input affect changeOpinions would not affect job security!

ISSUE: Candidate no-shows! Sometimes extended testing cycle by one or two days…

RESOLUTION: External recruiting company double confirmation including on testing day, Double book.


Test Plan Creation PhaseFoundation – Plan to Field Measurable Data Points

Lesson Learned

Contact InformationScenario QuestionsDeliverablesDefined Roles

User Profile Information (which ones were used)

Pre-test QuestionnairesScenarios

ISSUE: The “You’re Doing It Wrong” syndrome

RESOLUTION: Test plan approval


Test Plan Example


Testing Phase

Lesson Learned

Don’t give children sugar before a test…

Technicians – Ensure environment stability and smooth technological sailing

Moderators – Conduct testing, exhibit patience

Analysts – Observe, gather data, ensure test follows plan

[Embedded Video]


Testing PhaseLessons Learned

Jakob Nielsen’s first rule of Usability: Don’t listen to users!!!

[Embedded Video]

ISSUE: Communication between monitoring room and Ulab

RESOLUTION: Bluetooth headsets, IM, mIRC, SMS

ISSUE: Not working as intended (Mechanics, New Build)

RESOLUTION: Full test plan walkthroughs before approval process and walkthroughs on test day


Post Test Reporting PhaseLargely Conducted by Analysts

Top Ten Issues List – Executive Summary

Data gathering

Video highlight generation

Design suggestions

Prioritized screenshots of Usability severity issues

Report generation

Deliverable

Postmortem


Post Test Reporting Phase

[Sample Free Realms Mini-Game Report]


What We TestedWhat We Learned from It


Vivox Voice Chat Implementation

[Embedded Video]

ISSUE: Spatial constraints became apparent. Candidates were talking to one another

RESOLUTION: Volume in the headphones was increased

ISSUE: Multiple stakeholders driving different agendas

RESOLUTION: Communication conduit ensuring similar approach to static system across multiple platforms

Lessons Learned


Free Realms

[Embedded Video]


Difference Between Testing Children and Testing Adults

Conducted research prior to beginning testing

External focus groups – observing and studying methods

Reviewed past GDC presentations about testing with children:

Carla Engelbrecht, Teachers College, Columbia UniversityLittle Hands, Foul Moods, and Runny Noses: The Ups and Downs of Researching Games with Kids: GDC 2008


Difference Between Testing Children and Testing Adults

Cognitive and physical limitations in children have been long ago mastered by adults*

Example: Free Realms Target Demographic 8-14

Children Aged 7-12, Concrete Operational Stage

Understand Time, Space, Another Persons Perspective

Still rely on concrete objects and experiments to form ideas

Take off kid gloves*Carla Engelbrecht, Teachers College, Columbia UniversityLittle Hands, Foul Moods, and Runny Noses: The Ups and Downs of Researching Games with Kids: GDC 2008


Lessons Learned

[Embedded User Profile]

ISSUE: Confusion among parties as to whom we were, and should, recruit, against target demographics

RESOLUTION: Created User Profiles based upon 3 candidate profiles created in conjunction with Marketing and Development.


[Embedded Video]


Lessons Learned

ISSUE: Build validation prior to testing

RESOLUTION: Point of contact between studios, 3rd party client, and technicians

ISSUE: Geographic challenges

RESOLUTION: Chat feature for remote users to speak with San Diego office, Remote viewing, Remote logging

Result = Active participation from satellite studios


Conclusion

Greg Spence, Programmer EQ2: “Obvious benefit was ability to identify issues early on that we wouldn’t have caught until we went live. Helped to see where players reacted and stumbled through feature sets.”

Clint Worley, Producer EQ: “Obvious that the way we were presenting it (Vivox) wasn’t compelling. We were looking at the trees instead of the forest.”

Ryan Wells, Assoc. Producer Free Realms: “It was great to be able to make tweaks each week, make predictions about how users would respond and see the results. Our assumptions were often wrong and we learned just how different our audience was from ourselves.”

up down left right a usability postmortem tony rado – executive director global quality assurance,...

Documents

ethnic target

party disagreement

target audience issues

game play feed support

beta testing

efficient use

terms of moneyusability

task data collectiontotal