March 2013

TESTING TECHNOLOGY PRODUCTS

Websites, Apps, Software etc…

OVERVIEWMETHODOLOGIES

- EXPLORATORY TESTING- USABILITY TESTING- “REAL-WORLD” TRIALS

CONCLUSION

Reasons to Test Tech Products

Ensure efficient development of useful and appealing products– Begin with a well-conceived concept– Add features and functions systematically – Conduct periodic “reality checks” with users– Avoid “spider web” syndrome with changes– Increase confidence in reliability, relevance and

appeal of product as release approaches– Be prepared for an effective roll-out: sales story,

user training, competitive awareness etc…3

Reasons for Not Testing Tech

4

Reasons for Not Testing Tech

Lack of time Lack of money Overconfidence

– Unaware of potential problems– Disconnected from users true needs, attitudes or

skill level Sometimes not necessary

– minor changes, simple product, previously well researched, based on something similar, etc…

5

Types of Tech Product Testing

EXPLORATORY: – What digital content or functionality do users need

and/or desire? USABILITY:

– How easy or hard is it to use the digital product? “REAL-WORLD” TRIALS:

– How well does the digital product actually fulfill the users needs?

6

OVERVIEW

METHODOLOGIES- EXPLORATORY TESTING- USABILITY TESTING- “REAL-WORLD” TRIALS

CONCLUSION

7

Methodologies

Exploratory Testing

Usually groups – Live or Online groups– Online Bulletin Board

Typical flow of discussion:– Current technology practices– Speculate on future needs– Response to technology product ideas & options

Concept statements Wireframes or other mock-ups of ideas Existing websites or applications8

OVERVIEW

METHODOLOGIES- EXPLORATORY TESTING- USABILITY TESTING- “REAL-WORLD” TRIALS

CONCLUSION

9

Methodologies

Usability Testing

Most complex but not more important Common practices:

– Multiple iterations– Each consisting of 8-12 In-depth interviews

Depending on types users– 45-60-minute sessions– Conducted live in the early phases

Computer lab provides stable digital environment– Conducted online in later stages

Real-world digital environment - varied OS and browsers

10

Methodologies

Usability Testing

Four “phases” of usability testing– Wireframe– Alpha – Beta– Pre-release or Post-release

Dependent on level of completion and functioning of test digital product

Phases may not be clearly defined Nice but not necessary to test each phase

11

Methodologies

Usability – Wireframe Testing

Participants navigate through hyperlinked screen shots with only minimal content and sketchy design elements

Demonstrates the degree to which the website is organized and functions logically

Best conducted live, in professional market research facilities or computer lab

12

Methodologies

Usability – Alpha Testing

Product functions roughly and is populated with just enough data for participants to complete the most essential tasks

Design is evident but unrefined Evaluate how well the key navigational and

basic design elements align with users expectations

Best conducted live, in professional market research facilities or computer lab

13

Methodologies

Usability – Beta Testing

Product is reasonably stable and approaching full functionality with at least one complete segment of content

Gauges extent to which navigation, design and other conventions will work when product is complete

Good time to test “optional” features This phase of research is typically conducted

online to experience “real-world” conditions14

Methodologies

Usability – Pre or Post Release

Product is fully functional and content is complete

Simulates “real world” use to isolate bugs, impact of combined actions, identify minor refinements to improve user experience, uncover training needs

Usually conducted online– Tasks are more open-ended to see how new

users learn and problem-solve15

Methodologies

Usability

16

Methodologies

Usability – When Are You Done?

Digital products are tools…– Should fulfill the user’s needs in an efficient and

simple manner that is often “unremarkable”– “The absence of feedback is feedback” – High praise = “It worked just as I expected”

Ready for release when… almost all test participants successfully complete the mission critical tasks with minimal problems

17

OVERVIEW

METHODOLOGIES- EXPLORATORY TESTING- USABILITY TESTING- “REAL-WORLD” TRIALS

CONCLUSION

18

Methodologies

"Real-World" Trials

Multiple phase process– Individual trial and reporting of experience

Online Journal– Can use video, audio or text (smartphone saves time)

– Group discussion Live or Online or through Online Bulletin Board

Options for depth and duration– Task specific – 2-3 weeks– Short-term panel – 1-2 months– Long-term panel – 3 or more months1

9

Methodologies

"Real-World" Trials

Options for testing– Individual product– Comparison of competing products

Options for participants (Consider comparisons)– First users of new product– New users of established product– Veteran users of established product– Pre-trained users– Self-trained users– Users of competitive products2

0

Methodologies

Trials vs. Usability Testing

"Real-World" Trials Usability Testing

Documented by posts of users recollections to an asynchronous online bulletin board

Directly observed in real-time during one-on-one interviews

Macro view - Tasks are broad “create a lesson”- Feedback is non-specific

Micro view -Tasks are more specific “find a lesson about cell division”- Feedback is specific

Exposes common problems with only vague or speculative insight into users expectations

Exposes all problems and provides precise insight into users expectations

Helps to determine general functions that are problematic

Specifically identifies the problems related to each tested function

21

Methodologies Learning Curve vs Usability Issues

Learning Curve Issues Usability Issues

Challenge is learning the convention or action

Challenging to learn and to repeat

Once learned – repeated completion of task is easy

Users don’t learn – process is flawed

Improved with tweaks to labeling, design, button position etc… Also, training, tutorials etc…

Requires redesign of process. Sometimes creates a “ripple effect” that impacts other parts of platform.

Can be identified in both usability and "Real-World" Trials

Requires usability testing.Trial testing feedback is generally not specific enough to provide insight into these issues

22

Methodologies

"Real-World" Trials (related)

23

Corporate “Online Presence” Testing– Harvest available data

Online comments Customer communications Technical support data Website metrics

– Enhance and deepen insights with qualitative Live or online groups – primed with pre-group task Online Bulletin Board – users respond to web presence

of various companies and then “discuss”

OVERVIEWMETHODOLOGIES

- EXPLORATORY TESTING- USABILITY TESTING- “REAL-WORLD” TRIALS

CONCLUSION

24

About Dave Kreimer & Next Step Consulting

25

Next Step Consulting consists of a team of professionals supporting the efforts of Dave Kreimer, the company's principal.

Dave Kreimer's background in psychology, and years of facilitation and marketing experience enhance his market research skills. Since 1993, Dave has successfully designed, conducted, and documented roughly 500 research projects involving thousands of focus groups and personal interviews. The company’s most loyal clients include medical, educational, publishing, technology, and finance firms.

Contact Us

26

eMail: 2dave@nextstepconsult.com Phone: (206) 782-6002 NSC Website NSC Blog LinkedIn Facebook Twitter

27

Graphic- Thank You