cpen541 human interface technologies instructor: sidney ... · 09/13/2010 !1 cpen541 human...

09/13/2010 1

CPEN541 Human Interface Technologies

Instructor: Sidney Fels Term: W2020/2021

01/01/2021 CPEN541 Copyright 2021, Sidney Fels

• Lots of new interface technologies • Survey of some of the dominant H.I.T. • Look at H.I.T. in context of driving trends • Spark creativity for new H.I.T. and applications • Integrate skills to develop new technologies

Why This Course?

2


• User-centred design issues – introduction to HCI covers this (see CPEN441 for

example) • In depth analysis techniques for H.I.T. • Underlying physics of some H.I.T.

What I don’t cover

3


• The communication of human experience is central to the future of computing

• Techniques needed for: – sensing, encoding, transmitting, storing, indexing,

retrieving, compressing, recognizing and synthesizing • Human body has many I/O channels • Integrate Cognitive, Physical and Emotional

aspects of interaction • Interface should disappear.

Course Basis

4


Course Outline

5

http://courses.ece.ubc.ca/518/


1 Paper review – submit on Canvas website

2 Lecture - group prepared – presentations done towards last half of course as video

lectures plus class discussion – groups formed on Canvas

Assignments

6


• Three phase project to design, prototype and evaluate a new HIT idea – Phase 1: specification

A. 1 page abstract B. deliver specification document

– Phase 2: design A. design outline B. deliver design document (Oct 22)

– Phase 3: implementation and initial user testing A. Experiment Demonstration B. Video C. technical report D. conference paper E. conference style oral presentation

• New interface technology – new use of sensing technology, display tech., new metaphor

• Project done in groups up to 3 people

Project

7


• Use human I/O capabilities as outline • Use examples from literature to illustrate H.I.T.

for given human I/O • Most H.I.T. reports are system oriented • A lot of reading...

Course Structure

8


1. Exposure to tip of H.I.T. iceberg 2. Prototype and Evaluate a novel Human-Machine

interface 3. Develop research oriented thinking about HIT 4. Paper review 5. Write conference/journal style paper 6. Present results in conference style

Course Goals

9


• Introduction to Human Information Processing – Input

• Visual channel • Auditory channel • Position and Motion Sensing Channel • Somatic Channel • Taste and Smell Channels

– Output • Intentional

– neuromuscular, movable, verbal • Non-intentional

– GSR, Heart Rate, Brain, Muscle, other

Introduction

10


• Decisions – Tracking – Memory – Learning – Indviduals vs. Groups

Introduction

11


• Driving Trends for Human Interface Tech. – Virtual Reality, Immersive Environments, AR – Ubiquitous computing/Intelligent Environments – Wearable Computing, Tangible Bits, – Games, Arts, Interactive Theatre, Interactive Art – WWW, Agents, Collaborative work

Introduction

12


• Senses electromagnetic radiation (wavelength = 0.3-0.7 microns)

• 2 eyes for binocular vision • 100,000 fixation points (100deg circular) • Types of eye movement (six muscles):

– compensatory (must have target) – pursuit (must have target) – Tremor, flick and drift – saccadic (jump from one fixation to another)

Visual Channel

13


An Eyeball

14


Field of View

15


• Rods and Cones – Fovea is all cones (6 million) – Periphery is mostly rods (125 million) – interleaved

• Rods activate neurons in groups – higher sensitivity less resolution

• Cones are more one-to-one – lower sensitivity more resolution

Visual Channel

16


Fovea and Periphery

17


• 8Hz gives sensation of motion. • Familiarity helps interpret movement • Movement implies life • Movement links images (strongly)

Perceiving Motion

18


• Seeing depth: – Binocular vision:

• Disparity can be used to determine distance • Frontal plane horoptors (Helmholtz) • non-euclidean space

– Other cues: • overlap, relative size, relative height, atmospheric perspective,

texture gradients, parallel line convergence, motion parallax, accommodation and convergence

• Seeing size: – size constancy

Visual Channel

19


• Colour perception is very complicated - refer to readings

• Adapts to light conditions • lots of illusions to play with size and distance • other interesting things:

– retina is reflective – eye blink does not affect perception – pupil is normally black and circular – attention and gaze direction are correlated – people wear glasses and/or contacts

Visual Channel

20


• Senses mechanical vibration of air molecules – sound travels about 350m/s (1260km/hr)

• Human range is 16Hz to 20,000Hz • Pressure waveform causes hair cells to move

(23,500 cells) • Perceived loudness is approximately logarithmic • Perceived sound is highly dependent upon

environment – without reverberation unusual effects are noticed

Auditory Channel

21


Perceived Loudness

22


The Ear

23


The Ear (ctd.)

24


• Ears can localize sound • Least sensitive on median plane (due to

symmetry) • Perception of multiple pure tones complicated • Ears are well adapted for speech

Auditory Channel

25


• Inner ear has mechanisms for attitude • Body has proprioceptors

Position and Motion Sensing

26


Vestibular Mechanisms

27


• Sense of touch • sensations of:

– heat (temp), cold (temp), touch (pressure), pain (various)

• 7 distinctive receptors • one cold and one warm receptors

– more cold than warm – over 45 deg can activate some cold sensors – sensitive to changes in temperature

Somatic

28


When does Hot feel Cold?

29


• Tactual Sensing – rate is very important

• light touch quickly applied produces sensation – Hair acts as lever – same as proprioceptors – negative adaptation occurs

• high pass filter effect – 20 Hz is maximum for separability

• above 20Hz it is like audio signal

• Pain sensing – mechanical, chemical, thermal or electrical sensitive

Somatic

30


• “Chemical” senses • taste buds for

– sensations of sour, salty, bitter and sweet – receptor issues unresolved – extremely complex and poorly understood

• Olfactory cells for: – different theories: chemical, infrared absorption, … – different perceptual mappings:

• small prism • four odours: fragrant, acrid, burnt and caprylic

– Acuity is great - 10,000 times more sensitive than taste

Taste and Smell

31


Taste Buds

32


How to classify smell?

33


• Usually combination of senses active • We also can sense:

– time (protensity) – probability – intensity

• Break-off phenomenon

Summary of Input Channels

34


• Motor control associated with cerebral cortex • volitional and non-volitional

– can see in facial expression • muscles contract when stimulated by nerves

Output: Neuromuscular

37


A Mapping of Brain Control

38


• Affordances – keyboards, touch pads, phone dials, etc.

• verbal control/non-verbal control • tongue movement • breath control • facial control • gait

Intended Output: Movable Controls

39


Speech Requires Control

40


• GSR • EEG • EKG • EMG

Extracted Output:

41


• Significant constraint for learning and working effectively

• Magic number seven, Plus or Minus Two (Miller, 1956) – Chunking is key

• Capacity: 43 billion bits to 1.5 million bits? • Short term memory and long term memory

– different models for how memory is structured • Brain: 1012 neurons and 1015 connections

– connections change strength as we learn

Human Memory and Learning

42


• Human Cognitive Architecture (Modularity of Mind)

• Basic sensory, motor and cognitive performance characteristics

• Human sensori-motor abilities : good news & bad news

• Attention and Memory

• Models of Human Performance

Human Cognitive Performance

43


• Name the color of the text

• Respond as quickly as possible

• Measure response time

• 2 trials

Proof of automatic processing in brain

44


• Dog • Cat • Fish • Bird • Cow • Horse • Pig

Name the color of the text:

45


• Green

• Red

• Orange

• Red

• Blue

• Blue

• Orange

Name the color of the text:

46


• Subsystems may operate in parallel (theory):

• Input (perception): – Visual subsystem for what we see (most studied) – Acoustic subsystem for what we hear – Haptic subsystem for what we feel – Some subsystems are multimodal (e.g. speech understanding)

• Output (action): – Vocal (articulatory) subsystem for what we speak – Motor subsystem for how we move

Perception & action subsystems

47


• In upcoming images, – Image will blink or flicker – Image changes with each blink – Raise your hand as soon as you identify change

• Images from O’Regan, Rensink & Clark 1999 (ron rensink of this dept)

Visual bottleneck demo

48


Look for change between blinks

50


• The thing that changed wasn’t what we were attending to

• Change is very salient (the no-blink cases) but masked by another change at the same time (blink or flicker)

• Implications: – Mental image is an illusion – Mud-splash on car windshield – Looking away from display, refresh, “busy” interfaces – Same for other senses? Ongoing research…

How does this happen

52


• This phenomenon (also known as “change blindness”) must be taken into account when designing an interface

• Change blindness often dealt with as a constraint – appearance and behaviour of elements of an interface

• Example: error notification when filling in an online form – Instead of suddenly appearing next to the wrong field, leaving the

rest of the page unchanged...

– Use a progress bar to show that the info is being validated and place the notification in a separate pop up window

Why does it matter in HCI?

53


• This phenomenon (also known as “change blindness”) must be taken into account when designing an interface

• Change blindness is often dealt with as a constraint, influencing appearance and behavior of elements of an interface

• Example: error notification when filling in an online form

– Instead of suddenly appearing next to the wrong field, leaving the rest of the page unchanged...

– Use a progress bar to show that the info is being validated and place the notification in a separate pop up window

54

Close this window

x

Please enter a valid zip code for your region


• Limitation of our perception of the world sometimes can be used to overcome technological limitation

• In this case change Blindness allows walking in a Virtual Environment considerably larger than the surrounding physical space!

Change Blindness as an opportunity

55


• Seems like it: – Camera: keep steady, adjust focal lens length – Eye: focal point always moving, yet we perceive the world as

being sharp and in focus.

• But how does it really work? – Camera: film is exposed all at once by light from scene – Eye: Moves to explore a scene as information is needed

Vision system: like a camera?

56

Camera is good metaphor for optics of eye, but a poor metaphor for vision!


• Imagine creating a mental model of a room’s layout & furnishings by touching it when blindfolded or in the dark

• Model is built up serially (over time); process speeded if we start with a memory of what was in the room last time we were there,

• But if the memory is inaccurate or does not reflect current state, may take us longer to find the changes - because we believe in an incorrect model.

Vision is really more like touch:

57


• Vision considered dominant in UI design: this is changing rapidly.

• Audition & touch are critical in our non-HCI information-gathering & interaction with the real world

• Seen less in synthetic interactions because technology hasn’t caught up with our bodies.

• → Hot research area!

Other senses

58


• Multitude of input/output channels – all active at once

• I/O mechanisms usually depend upon – cognitive context – emotional contexts

• All these channels available to assist humans • H.I.T. is about finding ways to manipulate and/or

measure these channels for: – improved performance (cognitive, physical or

emotional) – entertainment and expression

Summary

59


• Tangible Bits • Wearable Computing • Ubiquitous Computing, Pervasive Computing,

Intelligent Environments, IoT • Art, Music and Entertainment • World Wide Web • VR/AR • Information Appliances (see Invisible Computer)

– Don Norman’s design of everyday things • Personal assistants/Personal Robots/self-driving cars

Driving Trends for H.I.T.

60


Is there a future for the Personal Home Computer?

61


• Imagine when computing is as cheap as paper • Computers will be everywhere, for every need

– transparent computing – transparent communication

• Mark Weiser (Xerox PARC) • PARC Tab, Pad, Boards + infrastructure • GUI based direction still

Ubiquitous Computing/Pervasive Computing

62


• Computing should be in background – end of personal computer – not just portable!

• Key concepts: – location

• context awareness – scale

• applications: – doors, preference forwarding, call forwarding, diaries,

daily informational assistance, prosthetics

UbiComp/Pervasive Comp

63



64

R.V.A.R.

V.R.

F.G. attention

B.G. attention

Computer

Computer Physical

Physical


• Dependencies: – H/W

• displays • computers • network infrastructure

– S/W • OS to handle real-time + constantly changing resources • Network protocols


65


• Advantages – computing where and when you need it – contextually aware devices – lower cognitive load

• Disadvantages – privacy – dependency – interaction limited to physical world metaphors


66


• Hiroshi Ishii’s group at MIT and others • Idea: couple virtual world to real, physical objects

– Interactive Surfaces – Couple bits and atoms – ambient media

• Main Goals – grasp & manipulate foreground with physical objects – awareness of background using ambient media

• Dependent upon good metaphor – need to really do user-centred design

Tangible Bits

67


• Leverage affordances from real world – Bricks (Fitzmaurice, Ishii, Buxton) – Clearboard (Ishii & Kobayashi) – metaDESK, ambientRoom, and more – Marble Answering Machine (Bishop), Props (Hinkley),

Live Wire (Jeremijenko) • Can you think of richly afforded physical devices?

– Doors, windows, cars, toys, dishes...

Tangible Computing

68


• People already know what manipulations make sense – as long as metaphor is maintained, life is good

• persistence of data • make abstract concrete • composition is natural • nice match of function, form and augmentation

Tangible Bits: Advantages

69


• Mismatched metaphor – makes task harder

• Limited to real world interactions • Complex interactions may be difficult to express

– looping contructs – boolean operations

• Mechanical failure of physical devices

Tangible Bits: Disadvantages

70


• Smaller and cheaper computers can be embedded in clothing and everyday devices – available all the time – can have first person perspective – augment person’s/devices ability

• MIT/U. of Toronto group including Steve Mann, Thad Starner and others on Wearables

Wearable Computing/IoT

71


• Wearable examples – video camera (glasses) – head’s up display (glasses) – compute device (shoes) – body monitoring devices – communication devices – tracking devices – audio devices – etc.

Wearable Computing

72


• Applications: – altered realities

• freeze frame, colour – augmented realities

• extra information such as people id tags • prosthetics: visual, audio, memory

– devices that know us and respond to us

• Social implications? – New protocols possibly needed – security

Wearable Computing/IoT

73


• Real-time, interactive graphics with 3D models + display technology that gives user immersion in the model world with direct manipulation

• Popular in late 80’s and early 90’s – is changing to interactive information visualization

• drove a lot of HIT – 3D graphics, trackers, gloves, HMDs and more

• Ivan Sutherland (1965), Jaron Lanier, Myron Krueger and lots more...

Virtual Reality/Environments

74


• Applications: – entertainment – vehicle simulation

• airplanes, cars, expensive machinery – physical data visualization

• planet surfaces • NMR data

– information visualization • chemical models • mathematical relationships

VR/VE

75


• Research problems – Visual displays

• field of view, resolution – Audition (speech and non-speech, input and output) – Haptics (forcefeed back and tactile feedback) – Tracking (still) – Emotion – Motion sickness – Software tools and models – Evaluation

VR/VE

76


• Depends on: – high speed computing – high speed rendering – low latency – good engineering design

VR/VE

77


• Cyberspace, Information Space – its own reality – mediates human-human interaction

• Must use H.I.T. to access this space – intelligent agents

• mobile, goal oriented, user context awareness – Computer application interfaces

• Enabling Technologies – browsers, GUIs, direct manipulation devices, – email agents, meeting scheduling agents – face recog. & synthesis, speech synth & recog

World Wide Web/Info. Spaces

78


• Music lead the push for many alt. Controllers – keyboards, wah-wah pedals, pitch benders – Therimin, Sackbut

• Artists often push boundaries of tech. to: – explore human emotion – concepts and philosophy – expression

• Video games drive H.I.T. • Education - web • Medicine - VR, tracking

Entertainment, Art, Music

79


• Technologies – video processing and integration – gesture sensing and recognition

• air guitar – wireless applications – robotics – image processing – high speed graphics – alternate controllers of all shapes and sizes

Entertainment, Art, Music

80


• Main driving trends – UbiComp/IoT

• Tangible interfaces – Information Spaces/WWW

• agents • better GUIs

– VR/AR • immersive experience

– Entertainment/Art/Music/Medicine • explore boundaries of expression

Summary

81


• Look at examples of many technologies in context of use – H.I.T. is generally application oriented initially

• become generally accepted and new appl’s found

Summary

82

cpen541 human interface technologies instructor: sidney ... · 09/13/2010 !1 cpen541 human...

Documents