InkSeine: in situ search for active note

taking

Ken Hinckley, Shengdong Zhao, Raman Sarin, Patrick Baudisch, Edward Cutrell, Michael Shilman &

Desney Tan

orontor

In situ search for digital ink

InkSeine : Ink + Seine

-n: a fishing net -v: to fish with a seine

= insanely cool ink search!

InkSeine lets the user fish for useful information directly from ink notes.

Fishermen catching salmon on the Columbia River using a seine.(courtesy Wikipedia)

Active note takingXLibris—pen-based system for active reading

(reading + critical thinking + learning)

InkSeine combines pen-based note taking with searching, linking, collecting & sensemaking:

ActiveNote Taking

Sketch designs Reflect on a topic Capture & extend

creative ideas

Core interaction pattern: Ink-Search-Gather

Ink – excels at capturing user’s thoughts & ideas with minimum distraction from formatting issues

Ink

Sear

ch

Gathe

r

Ink-Search-Gather interaction pattern

Search – digging through the file system on a tablet is really a pain Active note taking benefits

from low overhead access to supporting materials & related docs

Focus on personal search, but support Web as well

Ink

Sear

ch

Gathe

r

Ink-Search-Gather interaction pattern

Gather – Make it easy to use what you find by pulling content directly into unstructured notes Thumbnail hyperlinks to

documents Clippings from documents Queries that can be revisited

later

Ink

Sear

ch

Gathe

r

Ink-Search-Gather interaction pattern

InkSeine unifies these three elements in a fluid user experience that is integrated directly with inking

Ink

Sear

ch

Gathe

r

In situ searchLeverage preexisting ink to initiate search

No context switch to “search app” – stay in the flow No tedious transcription of text to a “search box”

Queries as first class objects, commingled with ink notes Flag some ink for search for later Visible and salient search history as part of notes Queries can be copied, pasted, moved, etc.

Interleave inking, searching & gathering No barrier between inking / searching – note ideas while

searching, side-by-side searching, span app boundaries for content gathering

Tightly couple queries with application content Queries naturally persist in originating context Easy re-visitation of prior queries Results of queries become new content

Rich tradeoffs in cost structure of

sensemakingContinuum of time cost – can defer at any point:

1 second: Capture thought “I should search on this” 5 seconds: Trigger search & see initial results list 10-20 s: Open a doc from result list or drag result into

notes 20 s-2 minutes:

Scroll through list and inspect details of results Apply filters, revise query Grab a snapshot from a short document Correct occasional ink reco errors – Tablet’s reco is amazingly

good! Longer: sufficient depth for common info needs; not mired

by last 10% of complex queries that require full featured search tools Inspect result documents to see if they meet information needs Sideways searches based on Title, Date, or Author of a result

Video

Paper Prototype

Paper Prototype

Paper Prototype

Findings

Existing desktop search tool difficult to use Difficult text entry

Enter and correct text

Findings

Existing desktop search tool difficult to use Difficult text entry

Enter and correct text Annoying context switch and flow interruption

Switch among apps and context

Findings

Existing desktop search tool difficult to use Difficult text entry

Enter and correct text Annoying context switch and flow interruption

Switch among apps and context Tedious information gathering

Difficult to link to documents and gather pieces of information

Notebook Control

In Situ Search

Breadcrumb

Breadcrumb

Trigger Personal Search

Search Panel

Query Area

Query Area(CueTIP)

Result List

Result List

Result List

Result List

Focus Result

File and Timeline Filters

Filters

Gestures for Filters

Tap - Select Cross – Multi-select

Disjunctive cross – Multi-select Cross & Turn Right – Exclude

Directed Work FlowSelection

Correction

Search

Occlusion

Occlusion

Examine Document

Document & Tracking Menu

Multiple Search Panels

Design Issue 1: Discovery

Gesture discovery problem

Highlighter hints

Design Issue 2: Occlusion

Context preservation & occlusion problem

Smart positioning of search panel

Design Issue 3: Poor Reco

Recognition problem for out-of-vocabulary words

Personalized vocabulary from desktop search indexUIST

List?, Dist?, Gist?, Mist?, Hist? ?

Summary

In situ search experience

Tailored to pen input

Optimum workflow / maximum flexibility

Multiple ways to gather content

Minimize search screen real estate

Span application boundaries

Questions?