educ evaluation overview

7/29/2019 Educ Evaluation Overview

1/63

Illah Nourbakhsh | CMU Robotics Institute | HRI Summer Course

Projects

Insect Telepresence: HCI formal techniques

Chips in museums: informal learning assessment

Mobot: long-term historical experiment


2/63


Insect Telepresence

Educational telepresence designed using

formal HCI inquiry tools.


3/63


Insect Telepresence Robot

Problem

Increase visitors engagement with andappreciation of insects in a museum terrarium atCMNH.

Approach

Provide a scalar telepresence experience withinsect-safe visual browsing

Apply HCI techniques to design and evaluate theinput device and system Cultural modeling, expert interview, baseline observation

Measure engagement indirectly by time on task

Partner with HCII, CMNH


4/63



Innovations

Asymmetric exhibit layout

Mechanical transparency

Clutched gantry lever arm FOV-relative 3 DOF joystick


5/63




6/63



Evaluation Results:

Average group size: 3

Average age of users: 19.5 years

Three age modes: 8 years, 10 years, and 35 years Average time on task of all users: 60 seconds

Average time on task of a single user: 27 seconds

Average time on task for user groups: 93 seconds


7/63


Insect Telepresence

Time on Task: from 15 seconds to 5 minutes


8/63


Robots in Museums

Educational tours in public spaces.


9/63


A Human-Scale Museum

Edubot Problem Increase visitors engagement and

learning at secondary exhibits in

Dinosaur Hall.

Approach Lead visitors to secondary exhibits and new facts

Design a robot to share the human social space Establish long-term iterative testing over years, not days

Time on task, observation and learning evaluations

Partner with Magic Lantern, Maya, CMNH


10/63


Chips - BBC


11/63


Museum Edubot: Technical

Contributions Required Robot Competencies: Safety, navigation, longevity

Approaches

Property-based control programming

Visual landmark-based SUF (Latombe)

Visual self-docking

h/w and s/w restart diagnostics

Fault detection & communication


12/63


Museum Edubot: Technical

Contributions Required Robot Competencies: Safety, navigation, longevity

Outcome

Zero human injuries

4 years deployment, over 500 km

traversed MTBF converging beyond 1 week

Uptime: 98%

Active diagnosis approaching 100%


13/63


Museum Edubot: Iterative

Design Cycle Design Refinements

Physical Design

Morphological Transparency: designing

informative form

Interaction Design

Behavioral Transparency: affective

interaction model Shortened length of media segments

Two-way interaction, goal-based learning


14/63


Human-scale InteractionGrange, Meyer, Soto, Kunz, Willeke



15/63





16/63





17/63


Robot Components

Nomadic XR-4000 base

Pentium-scale running Linux

Vision, ultrasonic, ir, tactile

8 lead-acid deep-cyclebatteries 12V, ~25AH each

~ 5 hrs endurance

Customized interaction top LCD + DVD media system Interaction button(s)

High-quality speaker system


18/63


Chips

Carnegie Museum of Natural History

Autonomy

5 years, > 500 km navigated, auto-docking MTBF convergence at 1 week

Proactive health state identification


19/63


Sweetlips

Hall of North American

Wildlife

3 years, > 185 kmnavigated

Tour theme interaction


20/63


Joe Historybot

Heinz History Center 2 years, > 160 km

Touchscreen, quiz,

speech generation


21/63


Adam 40-80

Free-agent robot for hire

More than 30 days of work

finished Republican National Conv.

Democratic National Conv.


22/63


Analysis of Evolution

Robot Autonomy & Vision

Interaction and Education Efficacy


23/63


Robot Autonomy Trends

Convergence of safety level Redundant case-based minimal vectors

Path-level control as on-line search in safetyzones

Diagnostic Transparency Vision camera +gdb

Trends toward specificity of state identification

Undo then - Retry methods

Navigation Metrical route-level constraints on safe areas

Topological landmark-based route map

Explicit confidence measured on visual fiducials


24/63


Landmarks: Visual Fiducials

H R b t I t ti


25/63


Human-Robot Interaction

Trends

Lower soliloquoy duration

Caricatured exaggeration of awareness

Interaction bandwidth & specificity

Unidirectional bidirectional - active query

Modifying human behavior

social inclusion experiments


26/63


Human-Robot Interaction:

Awareness Chips: Push my button to start.

Joe: Hey! Hi! Hello!

Adam: Look out, here I come!


27/63


Human-Robot Interaction:

Retention Chips: 3-minute dino-talks

Sweetlips: Which tour do you want?

Joe: Learn to speak Pittsburghese!Guess how much that bell weighs?

Adam: Participate in a poll. Test your

knowledge of presidential politics!

M i I t ti


28/63


Measuring Interaction

Performance

1 Quantifiable Observation

time on task: 74% at 5 15 min. peak ages: 5 12 ; 25 34 gender: 20% more girls


29/63


Measuring Interaction

Performance

2 Educational Efficacy

.5 .9 All dinosaurs lived during same period.

.5 .7 All dinosaurs were huge animals.

.5 .8 Other animals lived on the Earth with din.

.4 .8 All dinosaurs were carnivorous.

.4 .7 Scientists agree how to assemble bones.

.4 .5 All bones in Dinosaur hall are real.


30/63


RAVEN in the National AviaryRobot and educator collaboration


31/63


Influencing Human Behavior


32/63


Conclusion

Autonomy: general methodology leads

to robust behavior

HRI: We influence both robotic andhuman behavior with proper design

Project Closure: the shortcomings of

robotic tour guides.

Robotic A tonom : RI 16


33/63


Robotic Autonomy: RI 16-

162U


34/63


Goals

Empower students toward all robot hardware,

electronics and software

Encourage internalized student goals, both

creative and technical

Collect comprehensive weekly information for

off-line evaluation

Make the course a starting point for learning

create a rover community

give a complete robot to every graduate


35/63


Physical Design: Trikebot

Only 4 DOF

Raised camera POV

CMUcam vision system

Payload contingency

Tricycle configuration

Torsional stress limited

Back-EMF Motor control

Minimize servo torque


36/63


Trikebot Platform

Contributions Bicycle complexity goal Mechanical transparency

Hybrid design concept

Camera gaze design CMUcam vision system



Back-EMF speed control Minimize servo torque
http://www-2.cs.cmu.edu/~rasc/RA/Images/Day1%20Assembly.mov


37/63


Trikebot Platform








38/63


Trikebot Platform








39/63


Trikebot: Snagglepuss


40/63


Trikebot: Powerpuff


41/63


Course Organization

July 1 August 16, 2002 (Repeated in 2003)

30 students: 20 Latino; 9 women; 2 NHU

teachers

Typical age: 16-18 years

3-person teams

Challenge-based curricula

Cumulative trajectory CMU University credit

Web open-source


42/63


Educational Evaluation

Team

Prof. Kevin Crowley, University of Pittsburgh Assoc. Prof. of Education and Cognitive Psychology, U. Pitt.

Learning Research and Development Center, U. Pitt.

Director, Research & Evaluation, Pgh Childrens Museum

Informal science experiences preparing students for

classroom-based science and math education

Katie Wilkinson, Psychology at LRDC

Emily Hamner, Psychology and C.S.

John DIgnazio, M.Des and journalist


43/63


Data Collected

Initial and Final Individual Surveys

Weekly Individual Surveys

Online Documentation of Code

Weekly Interviews and Footage by Onsite

Ethnographer

One Week Formal Ethnography (mid-program)

Follow-up Web-based Monthly Surveys


44/63


Students Were Engaged

Students generally liked the contests and challenges(At least one third of the students rated every

challenge or contest as their favorite, even in later

weekly surveys)


45/63


The students rated the speakers a 4.7 out of 5

and everystudent said the speakers were an

important component.

Students Were Engaged (contd)


46/63


What Students Learned Learning Themes coded

Mechanics Sensors, Motors, etc.

Programming Java, Debugging, Documenting, etc.

Teamwork

Communication, Importance of Teamwork Problem Solving

Patience, Perseverance, etc.

Robot Point of View

Autonomy, Integration of Hardware and Software

Self-Identification with Science and Technology

Self-Confidence, Robotics Community, etc.

The Big Four: not rare to teach, but rare to succeed

Learning themes demonstrate broad acquisition outside the field of focus robotics.

What Students Expected to Learn and


47/63



What They Reported LearningExpected To Learn

0.0%

10.0%

20.0%

30.0%

40.0%

50.0%

60.0%

Team work Program ming Problem

Solving

Mechanics ID W/ Tech robot POV

PercentofStu

dents

Expected To Learn



48/63


0.0%

10.0%

20.0%

30.0%

40.0%

50.0%

60.0%

70.0%

80.0%

Teamwork Programming Problem

Solving

Mechanics ID W/ Tech robot POV

PercentofStudents

Expected To Learn Reported Learning at the End of the Class

What Students Expected to Learn andWhat They Reported Learning

Reported Learning


49/63


Reported LearningQuotes from Final Survey

To have patience.

To open source ones code for the better of robotics.

Document what one does so someone else can repeat theexperiment [and] be just or even more successful.

Developed better skills in working in groups That no matter what the obstacle we have, we can still

overcome it and solve it.

Have more confidence with myself.

Teamwork takes a lot of communication.

I learned that doing something slow is better than doing ittwice.


50/63


Reported LearningQuotes from Final Survey

To really pay attention to what I am doing and try to solve itfirst before asking for help.

Teamwork is hard especially with varying levels of skill and

different personalitiescan be rewarding only through

compromise.

Start with the basics, then make things fancier if youwantsimple is absolutely fine if it works well.

Using states in programming.

Make active decisions. Have the attitude that if I dont do it,

no one will and remember that if you choose something, you

are also choosing not to do other things because you havelimited time, energy, etc. Choose what to do with your talents

wisely and dont waste them!

Robots arent THAT complicated.


51/63


Student Struggles

Teamwork

2%

Programming60%Problem

Solving

2%

Mechanics

24%

ID W/ Tech

0%

robot POV

12%

Robots need to be tested in the same conditions as where they will perform.Teamwork is hard.


52/63


Student Breakthroughs

Teamwork

16%

Programming

33%

Problem

Solving

11%

Mechanics

24%

ID W/ Tech

7%

robot POV

9%

The big discovery was that if I try hard, by working with my teammates,

we could make a lot of things happen.

Don't ever leave anything at the end or else you will be struggling to

finish it on time.


53/63


Learning Trajectories

Statistical analysis of individual themes over

time.

Learning in terms of general to specific

knowledge transitions The Freshman engineering model

The following results all achieve statistical

significance


54/63


Mechanics and Programming

By the end of the course, students mentioned specific robot

technologies (like CMUcam) more often indicating that they had

become comfortable with the parts of the robots.


55/63


Mechanics and Programming

By the end of the course, students were much more specific inhow they talked about programming. Initially students realized

the course would teach them something about programming

but afterwards they were able to talk about the Java

programming environment or states in programming.


56/63


Teamwork

In the Initial Surveys Teamwork was mentioned in very

general terms (e.g. we will work in teams of three and

learn teamwork)

In the Final Survey Teamwork was mentioned much morefrequently and students were more likely to talk about

specific aspects of teamwork.

Teamwork leads to victory.

Teamwork is hard especially with varying levels ofskill and different personalitiescan be rewarding

only through compromise.

T k


57/63


Teamwork

P bl S l i


58/63


Problem Solving

If problem solving was mentioned in the initial surveystudents simply said they expected to learn problem solving

skills.

By the final survey students mentioned problem solvingmore frequently and specifically, often mentioning that the

course taught them patience or a new method of problem

solving.

Really pay attention to what I am doing and try to solve

it first before asking for help.

Always try hard for a long time and try all possible

ways to do something and if you cant then ask for

help.

Evidence That the Class Worked


59/63



For Girls

There were no significant differences in what girlsreported learning in the class as compared to boys.(specific mechanics)

Girls entered the class reporting less confidence withtechnology than boys but reported greater increasesin confidence than boys by the end of the class.

The only other gender difference was that girlsreported struggling more with programming thanboys.



60/63


Analysis:

Girls came in less confident and struggledmore with programming but were notdiscouraged:

Critical introductory engineering retentionissue.


For Girls


61/63


Trikebot: Footage


62/63



63/63

Summary

Learning themes demonstrate broad acquisition outside

the field of focus, with statistical significance

Engineering retention issues appear mitigated by the

Robotic Autonomy curriculum Robotics has a larger role to play in K-12, informal and

college education

Educational robotics as a research field: ripe for

exploration Human-robot interaction

Robustness, diagnostic transparency and intelligence

Minimalist design and control (cost, endurance, simplicity)

educ evaluation overview

Documents