usability study

Leonardo A. Madariaga MS Human Factors Engineering

Committee: Daniel Hannon - Mechanical Engineering-Tufts UniversityChris Rogers - Mechanical Engineering - Tufts UniversityEthan Danahy - Computer Science- -Tufts University

of the physical environment

in Makerspaces:exploratory study

Implications for the d

esign and

user evaluation of a n

ew workstation-‐syste

m

April 2015

Discussion

Question

Background

Method

Results

AgendaExploratory Study of the Physical Environment in Makerspaces: Implications for the design and user evaluation of a new workstation-system

Problem solving and innovationBackground

Top Down

Bottom Up

People/Places

Events

Vision Resources

Technologies

Some examplesBackground

(Photo by Jonathan Dietz, used with permission)

MIT Museum Makerspace

Background Some examples


Malden High School

Background Some examples

(Photo by Jonathan Dietz, used with permission) NUVU Studio

Background


Some examples

NUVU Studio

Background

(Photo by Author)

Some examples

Artisans Asylum

“Maker spaces support learning in an informal, play-focused environment that can cultivate an interest in science, technology, and design.”Scott Nicholson, associate professor in the School of Information Studies, Syracuse University, NY, 2012

“Despite a flurry of interest and activity around designing and creating Makerspaces, we still know little about the content and processes of learning in Makerspaces”Hakverson & SheridanHARVARD EDUCATIONAL REVIEW, 2014

CommunityShops

LibraryMakerspaces

MuseumMakerspaces

HomeShops

HobbyClubs

CommercialMakerspaces Making

EducationalMakerspaces

MachineWoodShops

Science Labs

Computer Labs

Collaborative spaces

VideoConference

DesignThinking

Educational

50´s 60´s 80´s - 90´s 00´s

A brief timelineBackground

Market analysis: WorkstationsBackground

LocusStanding DeskUS$1400

Ergotron WorkFit-DSit-Stand DeskUS$670

Jesper Office Height-adjustable Standing DeskUS$399

RefoldCardboard standing desk$250

D-School/ Stanford UniversityPeriodicTableUS$493

Market analysis: Storage & Organization systemsBackground

Really GoodStudent BinOrganization rackUS$180

i-RoverIPT102 7 Device Tech CartUS$320

D-SchoolStanford UniversityMobile StorageUnitUS$923

Copernicus3D PRINTINGStationUS$486

CopernicusSTEM MAKERStationUS$599

Paper/Book Author Findings Physical Environment/Usability

The Philosophy of Educational Makerspaces Part 1 of Making an Educational Makerspace

Kurti, Kurti and Flemming (2014)

Flexible by providing open spaces with elements that can capture users attention the first time they visit a space

A Design Case Examining Learning in the d-lab Leigh et al. (2013)

Transparency achieved through visual and acoustical “access” to activitiesEnhanced interconnectivity and socialization spaces inviting collaborationSpatial plasticity through the movability of furnishings

A Multi-disciplinary Design Environment

Carlson and Sullivan (2006)

Providing team storage lockers to organize tools and projects supplies. Larger and flexible storage areas would be desirable features.Architectural concepts, such as flexibility and visibility, should inform most of the design decisions of design lab.

Development of a Multidisciplinary Engineering Learning Center Batill and Gedde (2001)

A flexible and adaptable space. (b) Work areas and benches for project fabrication tools, test and assembly. (c) Storage space for students (d) Movable, student operated equipment (e) Project development and set-up area

Ergonomics product development of a mobile workstation for health care

Toivonen et al (2011) Two Usability tools were used: VAS (Visual Analogue Scales) and System Usability Scale. (SUS). (N=8)The study was able to define 19 design criteria for improvement of the product.

A usability evaluation of workplace-related tasks on a multi-touch tablet computer by adults with Down syndrome

Kumin (2012) Well-defined tasks can be used to account for user error, time to completion and rating of difficulty performing the task with specific devices. (N=10)

Flexibility/Mobility/Display/Storage/Project focused

Literature review

Usability testing/ Well-defined Tasks

Background

How can a Makerspace be more effective for managing larger amounts of projects and people at

the same time?

With limited (shared) spaceWith limited time (class 1/class2..)

With limited number of people to manage the space

Physical Environment

Question

Physical Structure

Symbolic artifacts

Physical stimuli

Open VS Closed OfficesSitting Vs Standing

Formal Meeting Space Vs CasualNatural Vs Artificial Lighting

Incoming email/notes on a wallSecurity signsSmell of coffee

Color of the wallsStyle of furnishing

Logotypes / Images

(Davis, 1984)

ObservationWorkflow& Design

GoalsPrototype

Tufts Tisch/CEEO Jumbo´s Makerstudio/Arts & Crafts Center/

Cambridge Friends/Malden High School

Modelling a general sequential

worklfowDefining design goals

and specifications

DIY-DesignExperimental unit

Contrast with Market

Usability Testing

SimulateSet Up

Design activityClean UpStorage

Method

Classroom needs / College / School

• How do people use the physical environment of Makerspaces?• How does the physical environment help Makerspace users achieve their goals?

Qualitative research method, such as shadowing were taken into account for note taking and understanding actions of users in context (McDonald, 2005)

ObservationMethod

(Photos by author)

Enter

Check on-going activitiesRead announcements

Leave personal belongings near work spotRetrieve on-going project from storage space

Set-up

Gather supplies/tools

Start design

prototyping

Mobility around space

Clean-up

Leave

Functional perimeter

with tools and materials

Other central

workstations

Clean work surfaceDiscard scrap

Recycle

Storage

Retrieve materialsCheck availability of toolsCheck availability of equipment

Leave on-going project and materialsin process in storage area

Bring personal belongings

Collaborationwith peers

Discussion ofproblems

SequentialWorkflow

MakerspaceUsage

User

Initial concept

The on-going project is the center of the making process

Shift from “stationary tables” to a “dynamic set of surfaces”

Allow to free up space

Improve storage / organization / set up / clean up

Method

(Photos by author)

Product goals, specification and prototypingMethod

(Photos by author)


(Drawings by author)

Below Market average (US$ 642)Cost per tray US$25

Does not consider Design/Assembly hours


Below Market average (US$ 350)Does not consider Design/Assembly hours

(Drawings by author)

Experimental prototypeMethod

(Drawings and Photo by author)

Tray RACKMovable Workstation

Researcher

Usability testing

Tray Rack

Materials rack and 3D printer

EntranceDisposal

Can

Workplace

Setting A: Baseline condition

Setting B: Experimental

Experimental layout

Method

•30 Participants (17Female/13Male)

•Age: M=24.6, SD=7.9

•18 sessions, spread over 9 days

•24 in groups of 2

•6 individually

•Performed a series of task using both

settings, two design activities

•Repeated Measures Design(Stangor, 2004)

•System Usability Scale(Brooke, 2996)

•User Experience Questions(Toivonen, 2011)

Usability testingMethod

Study was approved by Tufts University Institutional Review Board Nº 1501002

Task workflow

Usability testingMethod

PredictionsMethod

Increased System Usability Score (SUS)

Easier Storage & Organization

Adjective scale SUS SD

Worst Imaginable 12.5 13.1

Awful 20.3 11.3

Poor 35.7 12.6

OK 50.9 13.8

Good 71.4 11.6

Excellent 85.5 10.4

Best Imaginable 90.9 13.4

Adjective Scale(Bangor et Al, 2009)

Experimental (B)Standard (A)

Standard (A)

Experimental (B)

Faster Exitfrom Makerspace

Time<4 min

Subjective Eval.Subjective Eval.

Set Up/Clean Up

Usability testingResults

Open ended answersResults

Worksurface was equal in both settings (986 sq inches)Sitting still preferred for some detailed prototyping

Familiarity: Baseline condition is present in the majority of Makerspaces

(Coding according to Qualitative Data Analysis, Miles and Huberman, 1994)

Open ended answersResults

Lack of storage spaceShifting positionAccess to powerClean up: Male

Projector size didn´t free-up space

8%#

31%#

8%#

31%#

15%#

8%#

0%# 0%# 0%#

8%#

0%#

41%#

0%#

41%#

0%#

12%#

6%# 6%#

12%#

18%#

0%#

5%#

10%#

15%#

20%#

25%#

30%#

35%#

40%#

45%#

Limited#Space#

Backpack#storage#

Materials#rolling#off#

Shi@#posiAon#

Clean#up#

Access#to#power#

Difficult#collaboraAon#space#

SiIng#

Projector#connecAon#

Projector#size#

Most%difficult%aspects%of%se0ng%A%

MALE#

FEMALE#

Results Open ended answers

Storage directly in the project-trayFree up worksurface with small projector

Improved mobility: gather materials, supervise equipmentDevice-centered users require easy access to power

Results Open ended answers

The project-tray felt “less spacious”Middle separation was perceived as a “separator”

Standing positionThe amount of devices can provoke issues with cords

15%$

8%$ 8%$ 8%$

38%$

8%$ 8%$

0%$ 0%$

8%$

0%$

35%$

12%$ 12%$

47%$

6%$

0%$

6%$ 6%$

18%$

0%$5%$10%$15%$20%$25%$30%$35%$40%$45%$50%$

Tangled$cords$

Middle$separa:on$

Needs$seat$

Tray$access$

Spaciousness$

Locking$Mechanism$

Weight$resistance$

SetEup$

Backpack$support$

Standing$

Most%difficult%aspects%of%se0ng%B%

MALE$

FEMALE$

Hypothesis 1: Increased Usability of the Experimental Setting (setting B)

Setting A can be qualified as between “OK” and

Good”, whereas setting B can be qualifies as between

“Good” and “Excellent”. (Bangor et al, 2008)




Both groups scored setting B with significantly higher scores. Less than 1 year: : t (15)=-3.0365, p=0.0042

More than 1 year: t (13)= -2.0812, p= 0.0289

People working individually (N=6)did not evaluate significantly the workstation as more usable.

People working in groups (N=24) displayed significant higher evaluation of experimental

workstation


Gender-wise difference in evaluation.

Male participants scored significantly higher the new system, t(12)=-4.6209 p= 0.0006.

Female participants didn´t reveal any difference with statistical significance, t (16)=-1.6753,

p=0.1133.



Nº Statement Male Female Diff. P-value(M diff F)

1 I think that I would like to use this system frequently. 4.54 3.94 0.597 0.0305

2 I found the system unnecessarily complex. 2.15 2.24 0.0814 0.789

3 I thought the system was easy to use. 4.31 3.76 0.5429 0.075

4 I think that I would need the support of a technical person to be able to use this system.

1.94 1.54 0.402 0.161

5 I found the various functions in this system were well integrated 4.31 4.24 0.072 0.817

6 I thought there was too much inconsistency in this system 1.94 1.62 0.32 0.252

7 I would imagine that most people would learn to use this system very quickly.

4.61 4.11 0.497 0.055

8 I found the system very cumbersome to use. 1.69 1.941 0.2488 0.305

9 I felt very confident using the system. 4.384 3.823 0.561 0.037

10 I needed to learn a lot of things before I could get going with this system.

1.69 2 0.307 0.188


Participants within the average group (N=17) scored setting B with a higher

difference in terms of its Usability. (Makes sense: design for average)

Participant in the “extremes” (N=13) with either low or high standing elbow height

just reported a 1.1 points lower score in terms of Usability.



User experience: storage & organization

Hypothesis 2: Better storage and organization of projects

Expected value for the workstation system was higher than expected (Statistically significant difference: t (29)= -16.0709, p<0.001)

No gender-wise difference in evaluationStrong perception that the system would make it easier to store and

come back to a project

Results

User experience: storage & organization

Statistically significant , t(29)= -5.3866, p<0.001.

Positive evaluation of the user experience organization of projects

No gender-wise difference in evaluation

Results

Hypothesis 2: Better storage and organization of projects


No statistical significance in differences between timings

Set UP Clean UP

Usability testing

Is it faster to exit a Makerspace with Setting B?

Significant difference between means

t(29)= -10.9958, p<0.001

Results

Results Selected quotes

“I really liked the storage idea. For many Makerspaces, students come and go as they need to, the

hours are less regular. It's so crucial to have a place that is all my own that I can access very quickly

and easily. Having the shelves turn into work stations is a great solution to that. / / Also, in this

environment, when a maker is moving from machine to machine or back to their computer for

adjustments or checking the various projects going on, they are very active. It is much more practical

to be standing up at the standing up at the workstation so moving around and switching tasks is

easier”

Valued aspects: Storage/Mobility/Standing/Projector size and loca6on

Usability study met expectations. Further research is needed to understand why female scores did

not display significant differences. Adoption rate of the new workstation could be more dynamic, not

depending on the level of expertise.

Set up and clean up times did not display significant differences, although subjective perception

supports the idea that exit can be faster, storage and organization easier.

Sense of spaciousness was lower in experimental setting, even though both work areas were the

same. The presence of “edges” and “central division” is probably related to this

Most Valued aspects:

Mobility in the Makerspace: The project-tray decompresses the central zones allowing users to

move around using a functional perimeter.

Storage and organization of materials and projects: During the design and prototyping process

materials and/or projects won´t get mixed up, because the project-tray allows each user to

manage his/her own working surface.

Results Summary

Simplification of the design task due to: time constraints of the experiment, fixed set of materials.

Results Limitations

Carryover effects: Difficult to isolate influences over users when they used one setting and then the other

Practice and fatigue: Continuous assessment in a limited period of time

Define tasks that necessarily require the use of different sizes of materials

Usability evaluation just after testing each setting / Think aloud

Discussion

Developed a design for workstation-system based on current observation of workflow and needs in educational makerspaces

Tested an experimental method comprised of repetitive tasks related to Makerspace operation.

No previous findings related to clean up, set up and storage tasks when using workstations.

Replicate the study with different type of seats

Compare workstation performance against other commercially available standing workstations (adjustable and non-adjustable)

Survey: Perceived usefulness and limitations among teachers

Acknowledgements

Friends

CommitteeFamily

CEEO Administrative Staff

Graduate StudentsParticipants

Questions?

Leonardo [email protected]

MS Human Factors Engineering

Advisor: Daniel J.Hannon

Tufts UniversityDepartment of Mechanical Engineering

April 2015

of the physical environment

in Makerspaces:exploratory study

Implications for t

he design and

user evaluation o

f a new workstatio

n-‐system

mailto:[email protected]

mailto:[email protected]

usability study

Technology

examples photo

d printing station us

jonathan dietz

syracuse university

device tech cart us

educational makerspace

examplesbackground photo

cardboard standing desk