interactive visualization for computational linguisticsgpenn/esslli09/day5.pdf · 149 matching...

Interactive Visualization for

Computational Linguistics

ESSLII 2009

Evaulation2

Evaluating InfoVis

Purpose?

Information Visualization is the use of computer-

supported interactive visual representations of abstract

data to amplify cognition. (Card et al.)

Evaluating InfoVis: purpose

� Are we trying to make the right vis? (making the right vis / making the vis right)

� Does it do what is really required?

� Have we made what we were trying to make?

� Does it enable some task?

� Is the data represented?

� Does it enable insight?

� Does it enhance cognitive abilities?

Non empirical methods

� Complexity proof

� Verifying algorithmic correctness

� Verifying correct data – representation mapping

� Verifying novelty of the representation

� Demonstrating the match of the representation

to task by case scenarios

Insight

� varies from person to person

� instance to instance;

� hard to define, and consequently hard to measure.

� answering questions you didn’t know you had

� did infovis play a role in discovery

� temporally elusive

� teamwork and social factors

Choosing an Evaluation Approach

three particularly desirable factors:

each methodology favours one or two of these factors, often at the expense of the others

� Generalizability: a result is generalizable to the extent to which it can apply to other people (than those directly in the study) and perhaps even extend to other situations

� Precision: a result is precise to the degree to which one can be definite about the measurements that were taken and about the control of the factors that were not intended to be studied

� Realism: a result is considered realistic to the extent to which the context in which it was studied is like the context in which it will be used.

Choosing an Evaluation Approach

Choosing an Evaluation Approach

Field Study:� in the actual situation,

� observer tries as much as possible to be unobtrusive.

� realism is high

� results are not particularly precise

� likely not particularly generalizable

� generate a focused but rich description of the situation being studied.

Choosing an Evaluation Approach

Field Experiment:

� realistic setting;

� trades some degree of unobtrusiveness for more precision in observations.

� realism is still high, it has been reduced slightly by experimental manipulation.

� results may be more readily interpretable

� specific questions are more likely to be answered

Choosing an Evaluation Approach

Laboratory Experiment: � experimenters fully design the study.

� can provide for considerable precision.

� measurements possible - when and known

� less realistic – can provides more information

� introducing more realism will likely reduce the possible precision

Choosing an Evaluation Approach

Experimental Simulation: � experimenter tries to keep precision

� introduces some realism via simulation.

� examples

� studying driving under influence

� ‘Wizard of Oz’

� can provide considerable information while reducing the dangers and costs of a more realistic experiment.

Choosing an Evaluation Approach

Judgment Study:� person’s response to a set of stimuli

� creating ‘neutral conditions’.

� perceptual studies often use this approach.

� Examples

� Speed of recognition

� Setting can have imapct

Choosing an Evaluation Approach

Sample Survey: � discovering relationships between a set of variables in a

given population.

� proper sampling of the population can lead to considerable generalizability.

� responses are hard to calibrate.

� despite difficulties, much useful information can be gathered this way. We as a community must simply be aware of the caveats involved.

Quantitative Methodology

Qualitative Evaluation

holistic approaches that consider the interplay among factors

Observation Techniques� unobtrusive� notes are taken as observations occur� observations include

� setting, time, people, tasks, data, subtlies .� include both the overt and covert � include both the positive and negative

� be concrete whenever possible.� distinguish between verbatim accounts and paraphrased

and/or remembered.

Inspection Qualitative Methods

Usability Heuristics: � Well established

Collaboration Heuristics: � communication and coordination, awareness, territoriality, Mechanics

of Collaboration

Information Visualization Heuristics: � knowledge and task, Tufte’s, Bertin, cognitive (Ware)

Common Method: � first pass - gain an overview� second pass - asses interface components in detail

� akin to the design term guidelines

Qualitative Methods as Primary

� to develop a richer understanding through holistic approach. � enables full, rich descriptions rather than to make statistical

inferences � may be factors that can be numerically recorded� can be used at any time in the development life cycle. � as a preliminary step in the design process.

In Situ Observational Studies:

Participatory Observation:

Laboratory Observational Studies:

Contextual Interviews:

Albert Einstein

‘Everything that can be counted does not necessarily

count; everything that counts cannot necessarily

be counted’

Usability evaluation if wrongfully applied

� stifle innovation by quashing (valuable) ideas

� promote (poor) ideas for the wrong reason

� lead to weak science

� ignore how a design would be used in everyday practice

Greenberg, S. and Buxton, B. (2008) Usability Evaluation Considered Harmful (Some of the Time). In Proceedings of ACM Conference on Human Factors in Computing Systems - CHI'08, ACM Press, pages 111-120

Early design as sketches

Sketches are innovations & valuable

Method23

� Generative study

� Analysis of existing context (data, tools, work

environment, collaboration...)

� Derive rich understanding of needs and context

� Design sketching

� Discussions with data experts

� Prototype design

� Implementation

� Deployment and evaluation

design

evaluation implementation

Generative Study25

� Understand visualization context:

� How people work without information visualization or

with pre-existing visualizations

� How information work is situated in existing workplace

practices and environment

� How teams work together

� Domain-specific nuances of information use

� Goal is to describe meaning not make statistical inference

Observational Study

Preview image

Puzzle solution board

Puzzle piece

Russell Kruger

1. Comprehension

� Ease of reading, ease of task, alternate perspective

2. Coordination

� Establishment of personal spaces

2. Coordination

� Establishing group orientation

3. Communication

� Intentional communication

Analyzing Observations

Personal Territories

Storage

TerritoriesGroup

Territory

pW

Group 2

pNE

Rotation, translation &Mobile Storage

Solutions from the Real World

� Organizing items

� Passing and sharing items

� Storing items

Currents - sharing

Picture from (McGee, 2001)

Petra Isenberg

Real world information

Real world information

Petra Isenberg

Petra Isenberg

Observational Study

Browse Parse Clarify Strategize

Discuss Collab Validate Select Operate

Petra Isenberg

8 Processes

Temporal Sequence

Browse

Parse

Strategy

Select

Operate

Validate

Clarify

Collab Style

Petra Isenberg

Information visualization

workbench

Petra Isenberg

Information visualization workbenchPetra Isenberg

Information visualization

workbench

Petra Isenberg

Lark’s collaborative information visualizationenvironment

Pipeline Representation

Visualization Pipeline Branch

Joint Work

References

1. P. Isenberg and S. Carpendale. Interactive tree comparison for co-located

collaborative information visualization. IEEE Transactions on Visualization

and Computer Graphics, 13(6):1232–1239, 2007.

2. P. Isenberg, A. Tang, and S. Carpendale. An exploratory study of visual

information analysis. In CHI ’08: Proceeding of the twenty-sixth annual

SIGCHI conference on Human factors in computing systems, pages 1217–

1226, New York, NY, USA, 2008. ACM.

3. M. Tobiasz, P. Isenberg, and S. Carpendale. Lark: Coordinating Co-located

Collaboration with Information Visualization.

Matrix chart MT viz

2 Uncertainty in statistical NLP Collins et al., EuroVis, 2007

147

Gorman and Curran, Scaling Distributional Similarity to Large Corpora

Schmid, Trace Prediction and Recovery with Unlexicalized PCFGs and Slash Features

Ayan and Dorr, Going Beyone AER: An Extensive Analysis of Word Alignments and Their Impact on MT

Visualization for Presentation, Examples from ACL 2006

148 Exploration of Indigenous Languages Manning et al., Literary and Linguistic Computing, 2001

149 Matching Semantic Maps Ploux and Ji, Computational Linguistics, V. 29, pp. 155-178, 2003

150 Lexical Semantics Kamp et al., LREC 2004

151 Translation Parse Trees Voeckler et al., Personal Communication, 2008

152 Rule Productivity in Spelling Variation Kempken et al., SPIE-IS&T/ Vol. 6495, 2007

131 Cross-lingual IR Leuski et al., ACM Trans. on Asian Language Information Processing, 2003

132 Document Structure and Content Comparison Rembold and Späth, Total Interaction, 2006

133 Document Contrast Diagrams Clark, www.neoformix.com, 2008

134 Visual Search Engine kartoo.com, 2008

141

Two occurrences of a sequence is suspect

Plagiarism Detection Ribler & Abrams, InfoVis 2000

142 Literary Analysis: Affect Gregory et al., ACL Workshop on Sentiment & Subjectivity in Text, 2006

143 Literary Analysis: Patterns Feature Lens, Don et al., CIKM 2007

Open Research Problems187

188

CL Expertise for InfoVis

Improving Document Visualization189

Incorporating WSD, detection of multi-word entities, idioms

Enabling cross-language comparisons

Document “difference” visualizations on a semantic level

Deriving document structure to aid document navigation

Abstracting document visualization to a level useful and usable for information retrieval (next generation search engine interface)

e-Discovery190

A specialized form of document visualization for lawyers:

Thousands of documents classified individually

Clustering speeds things up drastically

More accurate keyword detection

Auto-classification with measures of confidence

... Very profitable sector already!

Attenex.com, 2008

Navigating Email and IM Chat191

Existing visualizations use only surface characteristics (letter/word counts, punctuation, meta-data)

Imagine navigating your email/chat history thematically

Thread Arcs (Kerr, 2003)BubbaTalk (Tat and Carpendale, 2002)

Managing Streaming Data192

RSS feeds from news and blogs

Facebook/Twitter updates

Academic journals/library update services

Social vis community is very active here, appropriating whatever CL methods they can figure out!

193

InfoVis to Further CL Research

Structural Comparisons194

Visualization to show similarities and differences in data structures:

Comparing parse trees and parse representations

Comparing ontologies, other knowledge sources

Language change over time

Lexical semantic distance measures

Others?

Exploratory Data Analysis195

Visualizing corpora

Quality control

Deep investigation of inter-annotator agreements

Discover areas of imbalanced data coverage

Interactive exploration of parameter spaces

“What changes when I adjust this parameter?”

Scented Widgets Willett et al., InfoVis 2007

Understanding NLP Processes196

“Live” visualization of automata

Dialogue system construction

Visualizing non-determinism

Visualizing uncertainty in parametric models

Visualization of chart pruning and beam search

Hypothesis tracking

Machine translation

Speech recognition

Others?

197

http://www.infovis-wiki.net Research & Education Linguistic Visualization

or Search “linguistic visualization wiki”

CHRISTOPHER COLLINS

CCOLLINS@CS.UTORONTO.CA

GERALD PENN

GPENN@CS.UTORONTO.CA

SHEELAGH CARPENDALE

SHEELAGH@UCALGARY.CA