September 10, 2017 Sam Siewert

CS415 Human Computer Interaction

Lecture 3 – WIMP HCI (Part 1 - GUI Builder Concepts)

Consider … Is the text from page 21 below correct?

What is the value of color vision to humans – at 1/20th resolution of grayscale? (6 million cones, 120 million rods)

Sam Siewert 2

Reading and Notes Summary P. 21 – “the” is repeated twice (…over the the lazy dog.) – Automatic correction based on expectation – Parsing line by line

What causes humans to miss what should be obvious in visual scenes? [page 21 of text]

– Adaptation of cues useful in “nature” used differently for HCI – Limit and focus on what’s “important” (visual), not all details are beneficial – Normalization (habitual) and “seeing what is expected” (lowers cognitive load) – not

noticing a change (missing objects) – Brain forms an image filling in details (that might not be there) – (cognitive model)

What is the value of color vision to humans – about 1/20th resolution of grayscale? (4.5 to 6 million cones, 90-120 million rods)

– Information – “fruit ripe yet?”, dangers such as “poisonous animal” – Same shape, but different color (distinguish), draw focus – Essential “shades” (could be grayscale) – RED means stop or angry (implied semantics),

lighter colors more peaceful. – To distinguish objects (shape, texture, shading). – Osterberg, Gustav. "Topography of the layer of rods and cones in the human retina." Acta

ophthalmol 13 (1935): 6-97.

Sam Siewert 3

A Little Humor on HCI

Sam Siewert 4

WIMP – GUI Builders The 2D GUI (e.g. e-mail, word processors, spreadsheets) is the Killer App for WIMP Floating Window in Main Desktop Window Integrates Various Widgets – Window – Canvas [Background] – Buttons (Radio Selection, Single-Click-Icons, Toggles) – Text Entry and Display – Pop-Up and Pull-Down Menus – Sliders and Scrolling (for large windows) – Minimization and dismissal

Handles Desktop Integration – Hide and Expose Events – Pointer Selection Events – Look and Feel of Existing Windows

Sam Siewert 5

WIMP Windows Icons Menus Pointers State of Practice Introduced in 1980’s!! GTK+, QT, Java, Python, and Numerous More GUI Builders Quality and Efficiency Improvements (Multi-programming and Multi-threading)

Sam Siewert 6

Apple Macintosh Desktop - 1984

Multi-Programming Multi-Programming – Invented for SLOW Storage Devices (Namely Disk and Tape) Waiting on Disk and Other I/O Where p=fraction of time waiting on I/O, n=number of processes, CPU utility = 1 – pn

Sam Siewert 7 Modern Operating Systems (3rd Edition) 3rd Edition by Andrew S. Tanenbaum

Degree of multiprogramming

Disk Arm Motion Random IOPs are the Problem (200 per Disk Typical) Due to Seek time + ½ Rotation on Average (milliseconds) An Eternity compared to Nanoseconds (1, 10, 100’s) for CPU and Memory running at GhZ Rates

Sam Siewert 8

Disk Access Latency Calculations

Sam Siewert 9

Multi-Threading Lighter Weight Concurrency for Multi-core, Load Balancing, AMP and SMP (Asymetric and Symmetric Multi-Processing) For Servers (Many Outstanding Service Requests) For Desktops (E-mail, Edit, Backup, Simulate, … , all in Different Windows at the “same time”) Thread Grids – Divide and Conquer All Embarrassingly Parallel Problems

Sam Siewert 10

Many Core System on a Chip Cluster on a Chip (E.g. Intel MICA – Many Integrated Core Architecture) Real-Time Rendering Graphics Rendering and Inverse Rendering with Computer Vision

Sam Siewert 11

GUI Layout & Code-Gen Tools or VHLLs for GUI Building

Method #1 – Compiled Programming Language (e.g. C/C++) and Widget Libraries (e.g. X-windows Gnome, KDE, etc. or Windows) – Advantages – Compiled for Efficiency, Security, Proprietary IP

Protection (Stripped Binaries) – Disadvantages – Less Interactive Development: Layout,

Generate code, Compile, Test, Repeat

Method #2 – Interpreted Very High Level Language (e.g. Java, Python, C# (both with Common Language Infrastructure)) – Advantages – Interpreted for Quick Test as you Code (Agile),

Rapid Prototyping, Portability – Disadvantages – Security Risks in Runtime (Java Virtual

Machine, CLI) Sam Siewert 12

Elements of a GUI Builder Layout Tool with Code Generation API for Simple Layout Main Window and Realization of it in Main program Canvas (Background) Layout Widget (where to place sub-windows and widgets inside application main window) Active and Passive widgets Active widget event handlers Filling in Event Handling – Behavior State machine or State transition design

Sam Siewert 13

State Machine vs. State Transition Table State Machine Shows “Design Intent” - User “Enters 1,2,3” State Transition Table Shows all Possible Transitions E.g. States S={Start, A, B, C, End} E.g. Input I={select, 1, 2, 3, Enter} Table = S x I 25 Combinations of S with I

Sam Siewert 14

A

B

C

Start

End

1

2

3

Select

State / Input Select 1 2 3 Enter Start A ? ? ? ?

A ? B ? ? ?

B ? ? C ? ?

C ? ? ? End ?

End ? ? ? ? Start

FSM = 20% Coverage [Finite State Machine]

Enter

Mealy FSM Output => State + Input

Next state => Current + Input

Moore FSM Output => State

Next => Current + Input

Assignment #2 Dr. Java + Swing Qt Creator for C++ Visual Studio WinForm C# Xamarin C# for Mobile Apps Python PyCharm Java Example – JDK – Dev Kit – JRE – Runtime

(libs, plugins, etc.) – JVM – Virtual

Machine (Runs programs)

– Somewhat Complex

Sam Siewert 15

http://docs.oracle.com/javase/8/docs/

Java Calculator Example Java OOP – Main program container class – CalculatorGui class – Public and Private classes and methods

Sam Siewert 16

Java Calculator Walkthrough and Demo

Import of Java Swing GUI Widget set Instantiation of Calculator GUI Main window configuration and realization

Sam Siewert 17

import javax.swing.*; import java.awt.*; import java.awt.event.*; public class SwingCalculator { public static void main(String[] args) { CalculatorGui cg = new CalculatorGui(); cg.setSize(250,400); cg.setResizable(false); cg.setDefaultCloseOperation(JFrame.EXIT_ON_CLOSE); cg.setVisible(true); } }

Main widget construction

Main widget configuration

Main widget realization

Java for WIMP GUIs Advantages – Portable – runs on Java Virtual Machine – Interactive development with IDE (DrJava or Eclipse) – Well Designed OOP – Widely Used – Lots of Tutorials and Documentation – Good for mobile systems (e.g. Android OS)

Disadvantages – Interpreted Byte Codes or Just-in-Time Compilation (Slower than

Native Compiled code) – Requires installation of JVM (Java Virtual Machine) – Distribution requires JAR (Java Archvie) – Security Concerns – Oracle Resources

Sam Siewert 18

Deeper Dive

Sam Siewert 19

import javax.swing.*; import java.awt.*; import java.awt.event.*; public class CalcTemplateGui extends JFrame { JFrame frame = new JFrame("My Simple Calculator"); JTextField answerField; JButton add, sub, div, mul, equals; JButton one, two, three, four, five, six, seven, eight, nine, zero; JPanel contentPanel; String stemp1=""; // CalculatorGui default constructor - super long constructor public CalcTemplateGui() { super("My Simple Calculator"); } private class Operations implements ActionListener { public void actionPerformed(ActionEvent event) { JButton src = (JButton) event.getSource(); } } private class ArgumentEnter implements ActionListener { public void actionPerformed(ActionEvent event) { JButton src = (JButton) event.getSource(); } } }

Event handler Classes for specific Active widgets – 0-9 Digits and Arithmetic Operators

Connecting Buttons to Event Handlers Instantiate Handler Classes for Button Types Add Action Listener Using Swing Implement Handler Class and Detect Source

Sam Siewert 20

… // layout with specific dimension for each button one.setPreferredSize(dim); two.setPreferredSize(dim); … div.setPreferredSize(dim); mul.setPreferredSize(dim); equals.setPreferredSize(dim); // create a private class to handle events detected by an action listener Operations op = new Operations(); ArgumentEnter arg = new ArgumentEnter(); // Add a listener to each button to invoke a handler for arguments entered one.addActionListener(arg); two.addActionListener(arg); … nine.addActionListener(arg); zero.addActionListener(arg); // Add a different listener to each button to invoke a handler add.addActionListener(op); sub.addActionListener(op); div.addActionListener(op); mul.addActionListener(op); equals.addActionListener(op); …

GUI Builder & Code Challenges Builders that Generate Code can get Out of Sync with Event Handlers – Generates Separate Handler Files (Classes) – Should not Overwrite Handler Logic – Structure Logic in External Modular Code

GUI Code Becomes Redundant – Similar Code for Each Button (Active Widget) – Many buttons – Similar Code for Layout – Similar Code for Panels, Menus, etc., differing only by Labels

and Use

OOP Can Abstract and Allow for Instances that Inherit and Refine Sam Siewert 21

Interactive Features - Handlers Implements Tasking and State Machine Logic of GUI Function of Current Button Depends on History of Previous Button Events Menus are Context Sensitive (Some Options Grayed Out) Pop-Ups Generated Based on Task Stream and Errors – E.g. – Are you Sure - “Confirm Deletion of File” – E.g. – “Exit without Saving?”

Display Updates Require History Cancel Undo/Redo Context Sensitive Help Pop-Ups … Sam Siewert 22

Event Handlers Ideally Simple State Machines May Require Concurrency – Display Updates (Telemetry, Monitors) – Pointer Drive Selections (Buttons, Menus, Sliders, Scrolling) – Client/Server Interaction – Status Updates [Progress]

Seems Simple, but Challenging Due to Multiplicity of Simplicity Verification and Validation – Use as Intended, Stress Test (Unexpected Use)

Sam Siewert 23

GUI Layers GUI Application Build on API or CLI Simply Generates CLI Commands or API Calls Updates Displays with Return Codes or Screen Scraped Messaging Not Always the Best Idea (but Often Done)

Sam Siewert 24

Prototypes and Mock-ups Early Feedback on Look and Feel and Interaction Usability Engineering Time to Complete Common Tasks Success Rate for Test Tasks by Different Users Cognitive Load for Tasks using an HCI or GUI Focus Groups for Usability

Sam Siewert 25