Designing Knowledge-assisted Visual Analytics Systemsfor Organizational Environments

Xiaoyu WangCharlotte VisCenterxwang25@uncc.edu

Thomas ButkiewiczUniv. of New Hampshire

tbutkie@gmail.com

Wenwen DouCharlotte VisCenterwdou1@uncc.edu

Eric BierPalo Alto Research Center

bier@parc.com

William RibarskyCharlotte VisCenterribarsky@uncc.edu

ABSTRACTWe present research focused on designing knowledge-assisted visual analytics (VA) systems for workers inorganizational environments. We focus on business an-alysts and asset managers, who work collaborativelyto analyze information and make decisions. Throughextensive investigations in two organizational environ-ments, we found that these users struggle with manag-ing and analyzing information from multiple perspec-tives. Their current tools lack support for aggregat-ing, organizing, and sharing such information. To ad-dress their needs, we characterized their analytic work-flows, extracted specific key knowledge actions for eachtask commonly found in these workflows, and designedand evaluated two visual analytics systems that supportand encapsulate these knowledge actions. We providedesign guidelines that should be used when designingknowledge-assisted visual analytics systems, and illus-trate their effectiveness with two systems built by fol-lowing them.

ACM Classification KeywordsH.5.2 User Interface: Graphical user interfaces (GUI)

General TermsDesign

INTRODUCTIONWe present the design for knowledge-assisted visual an-alytics systems in organizational environments. Ourtargeted users are business analysts and asset managers,who comprise the task force that handles informationanalysis and decision-making for companies and gov-ernment agencies. These users focus on fusing multi-ple streams of data, retrieving information for context-dependent tasks, analyzing and sharing their findings,and finally collaborating with others to reach decisions.

Permission to make digital or hard copies of all or part of this work forpersonal or classroom use is granted without fee provided that copies arenot made or distributed for profit or commercial advantage and that copiesbear this notice and the full citation on the first page. To copy otherwise, orrepublish, to post on servers or to redistribute to lists, requires prior specificpermission and/or a fee.VINCI 2011, August 4–5, 2011, Hong Kong, China.Copyright 2011 ACM 1-58113-000-0/00/0010.

Various systems have been developed to provide datamanipulation and information analysis [12] [16]. How-ever, as Gile et al. [8] pointed out, a major shortcomingof these systems is that they do not associate informa-tion analysis with the analytical process, and are there-fore limited in providing context for decision-making.Bucher et al. [4] further suggested that information anal-ysis is generally isolated from the user’s analytical work-flow, leaving a significant amount of data and informa-tion detached from an interpretation context. There-fore, it is necessary to design a system tailored to theseprofessionals’ workflows, while supporting more system-atic and purposeful information analyses.

Following Zimmerman et al.’s [21] definition of designresearch, instead of intending to produce a commercialproduct, we focus on producing design considerationsthat support the analytical process within a knowledge-assisted visual analytics system. We consolidate the re-sulting design considerations into more general guide-lines, which can be applied to the wide range of visualanalytics applications currently being developed and de-ployed in today’s organizational environments.

Our design study was conducted through extensive col-laboration with two groups of users. From them, welearned their actual analysis needs and workflows, andwith them, we concurrently designed prototype systemsto iteratively identify tangible design considerations fortheir user class.

This work makes three primary contributions: We presenta characterization of the analytical workflow of usersand key knowledge actions that are required to performindividual tasks in the workflow. We describe designguidelines for visual analytics systems that facilitate theworkflow through support for the above key knowledgeactions. Finally, to illustrate the effectiveness of ourguidelines, we introduce and evaluate two systems de-signed using them as a basis. (Further details on thearchitectures and implementations of these systems canbe found in our companion papers [19] [18].)

We grounded our design based on studies with two groupsof professionals in different organizational settings: bridge-

asset managers in The U.S. Department of Transporta-tion, who propose and execute strategic bridge main-tenance plans; and business analysts from Xerox, whoretrieve and analyze documents for information essen-tial to the operation of the business. We closely ex-amined these users’ analytic workflows and interviewedthem to learn the knowledge work required for achiev-ing each analytical task. In general, members from bothgroups must utilize and analyze information from multi-ple channels, and are required to generate shared prod-ucts effectively (e.g., a maintenance proposal or ana-lytical report). Subsequently, they need to coordinatewith multiple colleagues in different locations to agreeon strategic decisions.

Specifically, through iterative prototyping with domainusers, we summarized six task activities essential forthese professionals’ decision-making workflows. As shownin Figure 1, these six tasks are recurrent and central injobs involving foraging and analyzing relevant informa-tion, and enable these workers to update statuses andcoordinate progress with other individuals and groups.Currently, these tasks are handled dispersedly in an in-dividual’s workflow with little support for systemati-cally aggregating, organizing, or analyzing the informa-tion.

In the following sections, we present the procedures andfindings of our characterization for this domain, and itsrelated analytical workflows. We describe the use ofactionable knowledge (Section ) to transform the tasksfound in these workflows into tangible visual analyticsdesign guidelines. These design guidelines include re-quirements that support the essential analytic tasks, aswell as advanced functions. Finally, we evaluate twosystems designed with these guidelines in mind.

CHARACTERIZING ORGANIZATIONAL ANALYTICS PRO-CESSESTo produce appropriate design guidelines for an effec-tive knowledge-assisted visual analytics system, we closelystudied our targeted users and characterized their domain-specific analytical processes. As shown in previous re-search [14] [5], an organizational analytical task is aprocess of handling multiple channels of informationthrough the utilization of trained knowledge and cur-rent resources. Characterizing the analytical processin an organizational setting, such as a company or agovernmental agency, is a complex process and requirescommitment from all parties to maintain long-term col-laboration.

We are very appreciative to our collaborators from US-DOT and Xerox Corporation for their devotion to help-ing us pursue our research goals and generously provid-ing invaluable resources. Both organizations grantedus the opportunity for close, in-depth interactions withtheir users and to conduct surveys and interviews, whichwere crucial in studying their analytic processes. Withthe input we collected, we were able to create schemat-

ics detailing their workflows and identify the analyticaltasks used in each organization.

Our design study involved two separate investigationswith users from each of the two organizations. Par-ticipants varied in number, depending on the availabil-ity of these busy professionals at each time. Duringeach investigation, data was collected using online ques-tionnaires and/or semi-structured interviews. The datacollected was used to characterize these workers’ taskactivities within analytical processes, and further usedto develop the design requirements for a knowledge-assisted visual analytics system. In the following sec-tions, we describe the procedures and results for eachinvestigation:

Depicting Tasks in Bridge Maintenance ProcessStarting in January 2008, our university formed a re-search partnership with the USDOT and The NorthCarolina State Department of Transportation (NCDOT)to investigate novel approaches in assisting the bridgemanagement process. One of our first actions under thisresearch partnership was to conduct a nation-wide sur-vey [18] regarding professional profiles, tool usage, andtool preferences. The surveys were designed to providea baseline and statistics for comparisons between nor-mal tools used in bridge management, and to identifypotential areas for improvement.

Thirty-five out of the 50 state DOTs responded to oursurvey. The results clearly indicated that current bridgemanagement systems are often insufficient in support-ing effective bridge analysis. Almost all the respondingstates expressed the need to have a management systemthat would enable them to be more effective at analyz-ing their bridges, and that such a system needs to becustomizable to assist their individual workflows.

Based on their feedback, we further conducted semi-structured interviews with bridge managers on a reg-ular basis (every two weeks), in order to iterativelyidentify and propose features that can better supporttheir analyses. Through our interviews, we learnt thatbridge maintenance workflow is a process of decidingthe severity, trending, relevance, and benefits of main-tenance work on specific bridges, as well across as en-tire networks of bridges. Bridge managers hold the roleof knowledge manager and are attuned to informationanalysis and sharing practices.

As shown in Figure 2, the first essential analytical taskin the bridge analysis process is to gather all the relevantdata about a particular bridge, including any knowndamage, previous maintenance history, and typical de-terioration patterns of the materials involved. Bridgemanagers then analyze the obtained information, iden-tify any need for maintenance, and write up propermaintenance plans. We also note that bridge managersoften need to develop their own custom analysis rou-tines. Depending on available resources, a bridge man-

Figure 1. An overview for our design guidelines. An organizational workflow is characterized into six common taskactivities. Each activity is disseminated into fine-grained actionable knowledge. VA design guidelines are consolidatedby transforming this actionable knowledge into practical functions. Note: Given the different degree of completeness,only a subset of the listed actionable knowledge is typically used in accomplishing each task.

ager’s strategy can be very different from their peers’,requiring a different combination of the above analysisprocesses. In addition, sometimes even a single man-ager needs to utilize multiple alternative analytical ap-proaches due to changes in priorities. At the heart ofthese individual routines are different combinations andsequences of the above analytical processes. Therefore,it is important for a system to provide bridge managerswith the flexibility to combine and sequence these ana-lytical processes to fit their own, customized workflows.

Understanding Business Information AnalysisWe further carried our momentum and analysis method-ologies into our project with Xerox Corporation in thesummer of 2009. In an organizational environment,such as Xerox, employees’ document-centric activitiesresult in the creation of many diverse information streams,including email threads, calendar entries, web brows-ing histories, and versions of office documents. Manyof these documents contain information essential to theoperation of the business, such as project proposals andemails capturing product discussions. Thus, our goal inthis project is to investigate and design a system thatis effective to assist corporate employees in both man-aging these information streams, and extracting desiredbusiness information from them.

To understand this particular information analysis pro-cess, we conducted 30 semi-structured interviews withXerox employees. The interviewees held a broad rangeof positions, including product researchers who neededto write proposals and research papers, managers whowere in charge of business planning and marketing, andadministrative staff members who oversee hiring. Theseinterviews were designed to provide us with baselinestatistics about the general information analysis meth-ods that were being used in managing business infor-mation.

The results of our interviews showed that the most chal-lenging problems for the corporate employees was han-dling large amounts of content and, more importantly,managing information from multiple channels simulta-neously.

As shown in Figure 2, the analytical tasks of findingbusiness information often include content aggregation,information organization and correlation, and sharingand collaboration. To analyze certain business informa-tion, an employee often starts with aggregating content,such as possibly relevant documents, into a single loca-tion. They will then filter this large collection of data,and attempt to organize it in a clear and consistentmanner to support the awareness and sensemaking pro-

cess. We noticed that sharing their analysis findings andproviding status updates are crucial activities in theseemployees’ workflows. Because most current tools lacksupport for these critical functions, employees will oftenresort to paper formats or email to communicate withother colleagues about the business information whichthey have found or their need for help finding it.

Identifying Six Common Task Activities in OrganizationalAnalytics ProcessesTo further characterize the common task activities foundwithin an organizational environment, we consult theThink Loop model [15], grounding the usage of visualanalytics in a theory of information flows through theusers’ analysis processes. We found that, while differ-ent organizations shared diverse tasks, each’s analyticalprocesses constituted a series of similar, loosely defined,and collaborative task activities. Users accomplishedanalytical goals via subtasks, had focused targets, andaccessed a range of services and resources [7]. As shownin Figure 2, we identified six task activities common toorganizational analysis processes:

• Content Gathering and Aggregation: users identifyappropriately-scoped content to form basic analyti-cal contributions. They seek and extract informa-tion from multiple channels relevant to the analyticaltasks.

• Content Filtering and Customization: users use fil-tering to familiarize themselves with content theyhave collected. They also personalize the analysisenvironment in which this content is filtered.

• Content Organization and Information Analysis: usersorganize the collected content and examine it frommultiple perspectives to look for data patterns anddesired information.

• Evidence Collection and Hypothesis Generation: userscreate hypotheses regarding their analyses, and col-lect related supporting evidence.

• Report Generation and Status Update: users increasevisibility to others regarding analysis status, by pro-viding notification and updates on the progress oftheir analyses,

• Post-Analysis and Summarization: users focus onvalidating project achievements and introspecting work-flows, after accomplishing an analytics process.

TRANSFORMING ORGANIZATIONAL ANALYSIS PROCESSINTO VISUAL ANALYTICS DESIGNDesigning a knowledge-assisted visual analytics systemrequires supporting the analytical workflows of the users.While the aforementioned task activities are useful indescribing a general analytic process, they are often toogeneral to provide any specific guidelines in actual sys-tem designs. Therefore, the first step in our design re-search was to search for tangible artifacts that could

help breakdown these high-level semantic tasks. Thesetarget artifacts must meet two basic requirements: (1)they need to be concrete enough for practical visual an-alytics system designs, and more importantly (2) theymust be consumable for the users, who need to decidehow to make use of them, without introducing a con-siderable cognitive overhead.

In Search of Tangible Design ArtifactsMany approaches have been used to denote such ar-tifacts. We examined previous research in the intel-ligence analysis and knowledge management communi-ties, and focused on understanding the use of the knowl-edge process (e.g. knowledge creation, consumption,and transfer) within the analytical process. Similar toHeuer’s [10] perspective on knowledge as a dynamic ex-pectation of information, we emphasize the importanceof knowledge actions to support the transitions betweendifferent analytical task stages, and further integrationwithin the analytical process as a whole. In addition,Nonaka et al. [11] also explored the general knowledgeconversion processes that can guide design of organi-zational decision support systems. Again, these knowl-edge processes are too high-level to be useful in directinga specific design for a visual analytic system.

Enlightened by the Theory of Action [2], we followedAnrigyri et al.’s definition, and described our target ar-tifacts as a series of Actionable Knowledge. Action-able knowledge is explicit symbolic knowledge,typicallypresented in the form of tradeoffs for action or rules [13],which allows the decision maker to recognize some im-portant relations and perform an action, such as target-ing a direct marketing campaign, or planning infrastruc-ture maintenance aimed at reparing those assets withlowest health. The nature of actionable knowledge fitswell with our two requirements in that: (1) it representsthe fine-grained elements of each analytical task, andthus is quite instructive for the design of a knowledgeassisted visual analytics system; (2) it is extracted fromdomain users’ knowledge actions, and therefore can beconsumed without additional cognitive overhead.

Representing Organizational Analytics Processes usingActionable KnowledgeAs illustrated in [2] [4] [6], there are many approachesto model actionable knowledge. Given our advantage ofa close working relationship with actual domain users,we adopted the domain-driven modeling process, andgrounded our search for actionable knowledge on theinterviews and surveys with our two interviewee groups.

During the interviews, we asked the participants to en-vision the hypothetical process of carrying out theirusual tasks with their regular tools and working en-vironments. We encouraged them to also think aboutadditional functions that might be useful but not yetavailable in any of the tools they typically used. Specif-ically, we asked our participants about the fine-grainedknowledge actions they used in their daily practices, the

Figure 2. This chart describes the workflow in each organizational environment. (Top) The six task activities commonto both organizational analysis processes. (Middle) A typical analytical workflow for bridge maintenance planning.(Bottom) A typical analytical workflow for a business information analyst.

essential tools they have, and how they utilized thesetools to execute each action. In doing so, we were ableto identify key actionable knowledge that a tool shouldsupport to improve productivity and reduce workload.

In their responses, our interviewees expressed the im-portance of actionable knowledge to the organizationaldecision making process. In their analytical process,actionable knowledge is followed to respond to differentsituations, and illuminates potential action paths forovercoming obstacles. The use of actionable knowledgefurther directs these professionals to discover certain in-formation or data patterns, and helps them to react tothe advantages of a specific task. For example, for thecontent aggregation task, a bridge manager often needsto check multiple sources of information (e.g. struc-tural, financial, and historical) prior to their responsefor a new bridge maintenance request. During this pro-cess, actionable knowledge regarding where to look forinformation, and how to examine the information, playsa significant role in addressing this task.

Tools, in this context, are considered as means to trans-form the knowledge into desired task actions. Users pri-marily use tools such as email/documents/local folders,to produce and communicate task related contents andinformation. In the process, their domain knowledge(i.e. the expertise) is employed, and further results incontext-dependent actions that are used in their analyt-ical process. These professionals currently posses anduse a number of different tools; however, we found thatboth groups were severely lacking tools that were ac-tually designed to support to their analysis workflows.This finding pointed to the need for a tool that encap-sulates the users’ actionable knowledge and helps themeffectively perform necessary actions.

Based on the feedback from our interviews, we sum-marized a set of selected actionable knowledge that de-scribes the six common organizational task activities.As shown in Figure 1, every task in an analytical pro-cess is decomposed into a set of fine-grained actionableknowledge. Note that, this list contains only a subsetof all the collected actionable knowledge; some of the

stated actionable knowledge is unclear, ambiguous, orcontradictory, and is therefore excluded from this list.Also as seen in Figure 1, we have constructed a clearmapping between high-level tasks and their fine-grainedtangible artifacts. This mapping provides clear insightsinto the organizational workflow. More importantly, itis further transformed into a range of important de-sign requirements for creating an effective knowledge-assisted visual analytics system.

Transforming Actionable Knowledge into VA Design Guide-lines though PrototypingOur next step was transforming this list of specifiedactionable knowledge and requirements into proper vi-sual analytics designs. We followed the design theoryfor enterprise-knowledge-processes [13], and conductedseveral iterations of prototyping in close collaborationwith our users to encapsulate their actionable knowl-edge into functions. Although both groups shared sim-ilar common analytical tasks, our prototyping methodswith them were quite different (considering their diverseworkspaces and time constrains). Specifically, we useda more frequent, throwaway prototyping [1] method forour collaboration with Xerox. Given the shorter designcycle (three months), the throwaway prototyping guar-anteed us more design iterations and, more importantly,allowed us to explore broader options for transformingthe actionable knowledge into visual analytics designs.A sample of the intermediate prototypes can be seen inour companion media.

Based on our iterative prototyping with business ana-lysts, we found a clear preference for a unified, intuitive,and less intrusive system that can help effectively re-trieve and manage desired information. Therefore, wefinalized our prototype and implemented Taste [19]; aninteractive visual analytics system that enhances em-ployees’ capabilities to search and share business in-formation. As shown in Figure 3, Taste is structuredto embed information retrieval cues into a coordinatedmulti-level visualization system. At a high level, Tasteencodes these cues with a set of three visualizations,a Facet view (A), a Temporal view(B), and a EntityTag view (C). Each view presents a particular aspect of

Figure 3. An overview for both the Taste and IRSV systems. (Middle) The design guidelines actually incorporatedwithin each system are indicated by marked checkboxes. (Right) Taste consists of (A) Facet view, (B) Temporalview, (C) Entity Tag view, (D) Detail view, and (E) Storytelling view. (Left) IRSV contains multiple analysis views,including (F) Detailed structural view, (G) High-level structural view, (I) Geospatial view, and (H) Temporal analysisview. In addition, IRSV provides two variations: (K) a knowledge base integrated system and (J) a web-based system.

document activity information across entire collections.In lower-level views, Taste presents visualizations thatintegrate related activity information for single docu-ments(D). Using this multi-level structure, Taste helpsusers to cohesively depict document activity from dif-ferent points of view, and effectively find the desiredinformation.

Thanks to a long-term collaboration plan, we were ableto conduct a longer-cycle, more functionality-based pro-totyping process with the bridge managers at both US-DOT and NCDOT. This iterative functional prototyp-ing [9] simulates application behavior and helps to en-sure that more of our design system is understood ateach step of the collaboration. In each iteration, weinvited the bridge managers to test and evaluate ourprototypes by working with the system to perform ac-tual bridge analysis. Based on their suggestions and re-quests, we then refined, re-designed, and re-implementedthe prototype system to increase its effectiveness to sup-port the bridge analysis process.

During a nine-month period, we generated over ten func-tional prototypes, including various changes to the visu-alization and interface designs. Over the course of pasttwo years, our prototyping has resulted in a final set ofvariations of the system. These all focus on providingsupport for bridge management using integrated remotesensing and visualization, so we generally refer them asIRSV. While each of the systems is designed to accom-modate requirements for different use cases, all followa similar set of underlining actionable knowledge, andwere designed to achieve the same goal: to provide ex-amination of heterogeneous data sources and facilitateeffective bridge maintenance planning.

At the heart of IRSV, we designed a set of visualizationsto help bridge managers organize and analyze their as-sets from the multiple perspectives essential to theirdecisionmaking process. As seen in Figure 3, these vi-sualizations were designed to perform the three high-level analyses: structural analysis (G), temporal analy-sis (H), and geospatial analysis (I). For lower-level tasks,we designed a structural detail view (F) to automati-cally link information between each bridge component,and provided bridge managers with an intuitive visu-alization to interactively analyze specific correspondinginformation. All of these visualizations are tightly coor-dinated together in such a way that an action performedin one view affects all other views. Implementation de-tails can be seen in our companion papers [18] [20] [17].

EXAMPLES FOR SYSTEM DESIGN AND EVALUATIONBoth Taste and IRSV were designed following our guide-lines. These knowledge-assisted visual analytics sys-tems are implemented to support the analytic processesencountered in organizational environments. Throughiterative prototyping processes, each was tailored to theanalytical workflow of its target domain. As shown inFigure 3 (Middle), the design guidelines actually incor-porated within each system are illustrated separatelyby marked checkboxes. We also conducted user-studiesto evaluate the utility of these systems.

Instead of emphasizing technique details, our discussionbelow focuses on evaluations for the effectiveness of oursystems to support domain analysis processes. Specifi-cally, we summarize the users’ feedback and comments,and use these to assess the performance of our systemsin facilitating the common task activities. We have alsoplanned future improvements for the systems based on

the users’ suggestions.

Taste: Supporting Business Information AnalysisTo evaluate Taste, 21 Xerox employees participated inboth lab and field studies using the tool. In the fol-lowing subsections, we describe how Taste was foundto be useful and effective in facilitating each of the sixcommon task activities in the domain analysis process.Detailed statistical results for this evaluation can befound in our previous report [19].

Gathering content into a unified visual interfaceAt the heart of Taste is a transparent, real-time, con-textual data capturer, which was designed to capturethe user’s activities around office documents, calendars,emails, etc. Taste creates an index of documents on auser’s machine, and logs information about the user’sactivities with these documents. Taste stores this in-formation, along with copies of the documents, in aunified repository. All captured information is then in-dexed and grouped with its related documents, and isinteractively presented to the user through Taste’s vi-sualization interface, as shown in Figure 3 (Right).

All participants indicated the usefulness of this unifiedinterface. They agreed that integrating multiple infor-mation streams into a single interface sufficiently en-capsulates their actionable knowledge, reducing searchtimes for related information. They believe this couldgreatly assist them in gathering and aggregating con-tents from multi-channels

Enable facet search for content filteringAs shown in Figure 3 (A), Taste utilizes the Facet viewto aggregate both the documents and the people withwhom a user has previously interacted. This visual-ization allows the users to filter and sort informationbased on automatically extracted data facets, includ-ing type (person or document) and format (email, textdocument, etc.). Facet view further sorts and displaysdocument activity by importance, which is measured byfrequency and users’ dwell time.

When presented to the participants, they spontaneouslyformulated a variety of facet filters to find information.They were generally satisfied with the efficiency of usingTaste to ’slice and dice’ information, and appreciatedthe flexibility to perform customized analysis.

A common suggestion was to be able to also create for-mulas to sort the documents with customized measures.One analyst indicated that introducing customized timefactors (such as increasing the importance of a more re-cently created documents over older documents) wouldbe especially useful for filtering.

Interactive Information AnalysisBesides the facet view, Taste also supports high-levelcontent analysis based on both temporal informationand content keywords (See Figure 3 (B) and (C)). Taste

utilizes the temporal view to show how a user’s activi-ties unfold over time, and presents the temporal trendsand patterns of a user’s document activities. This viewallows the user to interactively drill down to a specifictime, and helps the users examine the content, whichoccurred in that time span. In addition, an entity tagview is used to enable fast entity browsing. This is im-plemented using an automated entity extractor, whichextracts entities, such as company name, contacts, etc.,from all of previous documents. As shown in Figure 3(C), Taste enables users to focus on a specific entity,and examine any information related to it.

In the low-level view, Taste incorporates a detail view(Figure 3 (D)) for depicting a single document frommultiple perspectives, such as its related temporal in-formation and other versions of the document. All viewsin Taste are coordinated, such that updates in one vieware immediately reflected in the others.

In our studies, Tastes was compared with other existingtools to assess its analysis capabilities. The participantswere generally positive about Taste’s effectiveness forretrieving and analyzing business information. All par-ticipants agreed that the ability of viewing informationfrom different granularities can largely help them filerand analyze information.

One suggestion was to provide finer-grained categories,and display more information for entities. One partic-ipant suggested that the current categorization is toobroad by referencing a common expectation: Instead ofgeneral, high-level categories like browsing, email, etc,usually the categories of interest are more narrow like“email with Bob” or “browsing about JAVA”)

Using Storytelling to generate and share reportsBy utilizing an interactive storytelling view, shown inFigure 3 (D), Taste allows users to interactively collectevidence, annotate it, and share it with others. Thestorytelling view allows the user to take a more activerole in information tracking, and enables them to ex-press the information relationship based on their ownknowledge. Whenever a user comes across an interest-ing information object in Taste, they can directly addthat object to a new or existing story view. Once anelement is in a storytelling view, the user can further an-notate or tag it, and can group different story elementsbased on their reasoning logic.

The story created by one user around a collection ofpeople and documents may be of interest to other usersas well, so Taste allows stories created in one instance ofthe system to be shared with users in another instance.Analysts who receive these shared stories, are able tomodify them based on their understanding of the top-ics, and add or suggest removal of story elements. Bysharing their stories about document activities, groupsof employees can now understand those activities bet-ter, and improve information analysis for all members

of the group.

While the story feature is new, many participants foundthe idea of collaboratively searching for information in-tuitive, and felt that the feature was practical and use-ful. Although we didn’t set up a collaborative environ-ment for participants (due to privacy concerns), partic-ipants were still interested in utilizing the story viewand tried to share findings between different instancesof Taste.

In summary, while Taste has so far only been eval-uated by a limited number of participants (albeit ac-tual target users), it appears to be a promising technol-ogy and a successful design. Based on the feedback wehave received, we believe the design of this visualizationsuccessfully encapsulates the actionable knowledge andsupports the analytical workflows that are essential forbusiness information analysis. Through our on-goingcollaboration, we are further refining its basic functionsand enriching it with more advanced features.

IRSV: Facilitating Bridge Maintenance PlanningOur evaluations of IRSV and its variations were per-formed iteratively throughout the collaboration, andwere mainly conducted with a group of bridge man-agers from both North Carolina DOT and CharlotteDOT (CDOT). These 12 (10 male, 2 female) bridgemanagers participated in at least three sessions of on-site evaluations

In the following subsections, we summarize feedbackfrom these evaluations and assess the systems (multi-ple IRSV variations collectively) for their effectivenessin facilitating each task activity encountered in bridgemaintenance planning.

Integrating heterogeneous data into one interfaceAs shown in Figure 3, IRSV provides bridge managerswith a unified content interface that combines multi-ple streams of bridge information. It can incorporate arange of data sources, including National Bridge Inspec-tion Standards (NBIS) datasets, high-resolution aerialimages, and Light Detection and Ranging (LIDAR) scans.In addition, IRSV provides an advanced feature, incor-porating knowledge contents from an ontological knowl-edge structure. As detailed in our previous report [20],using a service-oriented-architecture, IRSV has been ex-tended to communicate with the knowledge base, accessand fetch the inference results, and present them in acohesive visual interface.

Through comparisons to existing bridge managementsystems, it was clear that IRSV was appreciated for itsefficiency in contents aggregation. All participants con-sidered the visual interface well addressed their infor-mation retrieval needs, representing cohesive and use-ful for bridge information. Moreover, they were excitedabout the ability to access and follow prior practices andguidelines that were embodied in the knowledge base.

Customizing analysis workflowsBecause it was built with a modular architecture, IRSVallows bridge managers to extend the system to in-corporate advanced visualizations and more effectivedata models. Each visualization component integratedwithin IRSV was designed to be interchangeable withother equivalent visualizations. Furthermore, IRSV pro-vides bridge managers with the flexibility to combineand sequence different visualizations to fit their indi-vidual analysis routines.

All participants appreciated the flexibility of the in-terface, finding it useful for customizing the system toonly utilize the necessary visualizations in their partic-ular practices. They spontaneously formed a variety ofvisualization combinations in order to find bridge as-sets. The most common strategy used was to combinea geospatial window with scatter plot view to gain in-formation for the most recent changes of a particularbridge. A manager from NCDOT further pointed outthat,“[IRSV] will greatly shorten the catch-up time be-tween my learning to use the system and my actual useof it.”

Analyzing information from multiple aspectsAll participants noted that IRSV provided a visual ex-ploration environment to help them analyze informationfrom multiple aspects. The capability to perform notonly geo-temporal analysis, but also structural analysiswas of great value to their decision-making process (SeeFigure 3 (G)(I)(H)). One of the managers commentedthat, “[the] linked visualizations provide me with a co-hesive understanding about the data that I am workingon. It reduces the time I spent on manually searchingfor information, and helps me focus more on the taskitself.”

In particular, seven out of the 12 bridges managerspointed out that the temporal analysis in IRSV pro-vided them with the capability to effectively monitorchanges in bridge conditions and identify maintenancecandidates. In addition, after familiarizing themselveswith the concepts and usage of the visualizations, mostbridge managers (9 out of 12) noted that the capabil-ity to examine bridge structures simultaneously frommultiple levels (overview and detailed view) allowed foreffective transitions from examining large amounts ofdata to inspecting bridges one at a time.

Evidence collection and report generationAs shown in Figure 3(J), IRSV also supports interac-tively collecting, annotating, and sharing analysis find-ings between different collaborators. Using a web inter-face, IRSV treats individual visualizations and groupworkspaces as collectable items. It enables bridge man-agers to directly drag and drop these items into a sand-box, designed to collect all the findings and sort themtemporally. IRSV further allows bridge managers touse the collected evidence to support their analysis hy-potheses and create analysis reports. The bridge man-

agers can directly combine findings that can supporttheir reasoning and share them with colleagues, throughbuilt-in sharing channels or emails.

Most participants found the idea of collaboratively man-aging bridge information intriguing. They consider ourapproach practical and useful for creating preliminaryanalysis reports. There was significant interest in uti-lizing the features that allowed evidence to be reportedand shared with others. While we are still refining thesefeatures, we have seen great potential for IRSV to sup-port the inherently collaborative nature of bridge main-tenance planning.

In summary, IRSV was designed by following our de-sign guidelines set forth earlier in this paper. It has beendeployed to USDOT for daily use and testing. Basedon feedback from bridge mangers, IRSV appears to bea successful design and a useful visual analytics sys-tem that effectively supports the bridge maintenancemanagement process. The effort to enrich IRSV is stillon-going; we are working closely with bridge mangersto identify new actionable knowledge that requires ad-vanced features, including web-based collaboration andpost-analysis.

LIMITATIONSWe undertook this research to better understand thepragmatic analytical processes in an organizational en-vironment, and identify practical design guidelines forvisual analytics systems. To this end, we consolidatedour design guidelines into characteristics for the six com-mon analytical task activities, their related actionableknowledge, and interactions between the two. We foundthat actionable knowledge plays a unique role in ad-dressing important problems in organizations, and af-fects users’ performance. Therefore, we transformedthis knowledge into design guidelines for visual analyticssystems. We hope that our guidelines will help othersprovide better support for domain analytical processeswithin their visual analytical applications.

There are limitations to our research which must shouldbe addressed. Generalizability of our design guidelinesis limited because this research was conducted withinonly two organizations. We attempted to mitigate localbiases by increasing the number of participants. Nev-ertheless, different training backgrounds, personal pref-erences, and project time constraints could engenderdifferent analytical conditions.

Moreover, our research characterizes the domain ana-lytical workflow through interviews and surveys, whichgenerally are self-reported by participants. Our researchwas also limited, in that it modeled the analytical work-flow from a retrospective perspective, whereas Brows etal. demonstrated that problem spaces and solutions areestablished and change dynamically in interactions withpeople and the environment [3]. Therefore, our under-standing of domain analysis and actionable knowledge

is constrained to the users’ general way of performingtasks.

Finally, our research is limited by its evaluations withdomain experts. We evaluated Taste with formal stud-ies and IRSV with informal case studies. Developingevaluations, strategies, and methodologies to accuratelyassess the effectiveness of a knowledge-assisted visualanalytics system is challenging. At this point we do nothave a clear outline on the best evaluation approach; thedesign of guidelines for evaluating a knowledge-assistedvisual analytic system would be one interesting futuredirection for our research.

However, while we recognize these limitations in ourwork, we believe supporting organizational analysis pro-cesses is important visual analytics research. Our de-sign guidelines (Figure 4) illuminate the role that aknowledge-assisted VA plays in such complex problem-solving environments.

CONCLUSIONSThis paper has presented two years of iterative designefforts to explore and advance the design of knowledge-assisted visual analytics systems. Based on our exten-sive interactions with domain users, we identified andconsolidated six common task activities that are gen-erally used to perform organizational analysis. To de-compose these high-level tasks to implementable arti-facts, we further reframed the problem and dissemi-nated these tasks into actionable knowledge that illus-trates the fine-grained functional requirements for eachtask. Using these requirements, we designed and imple-mented two knowledge-assisted visual analytics systemsfor our collaborators.

Our primary contribution is the resulting set of de-sign guidelines that, when implemented, allow visualanalytics researchers to effectively collaborate with do-main users, and to empower users in organizational en-vironments to effectively accelerate their analytical pro-cesses. These guidelines provide design considerationsfor both high-level task activities and low-level func-tional requirements. In addition, we have summarizeda set of evaluations that show the effectiveness of vi-sual analytics systems designed using our guidelines asa basis.

We hope that by proposing these general guidelines,we can begin a serious discussion of design considera-tions critical for producing effective knowledge-assistedvisual analytics systems. We will continue to evaluateand refine our guidelines with current and future col-laborators. In addition, we hope that these guidelineswill lead to potential impacts in today’s organizationalenvironments.

REFERENCES1. S. Andriole. Fast, cheap requirements prototype,

or else! Software, IEEE, 11(2):85 –87, Mar. 1994.

Figure 4. Our design guidelines in checklist form.

2. C. Argyris and D. A. Schon. Theory in Practice:Increasing Professional Effectiveness. Jossey-Bass,1992.

3. J. S. Brown and P. Duguid. OrganizationalLearning and Communities-of-Practice: Toward aUnified View of Working, Learning, andInnovation. Organization Science, 2(1):40–57,1991.

4. T. Bucher, A. Gericke, and S. Sigg. Process-centricbusiness intelligence. Business ProcessManagement Journal, 15(408-429), 2009.

5. G. Convertino, S. Kairam, L. Hong, B. Suh, andE. H. Chi. Designing a cross-channel informationmanagement tool for workers in enterprise taskforces. In Proceedings of the InternationalConference on Advanced Visual Interfaces, AVI’10, pages 103–110, New York, NY, USA, 2010.ACM.

6. R. Cross and L. Sproull. More than an answer:Information relationships for actionableknowledge. Organization Science, 15:446–462,August 2004.

7. P. Geczy, N. Izumi, S. Akaho, and K. Hasida.Analytics and management of collaborative

intranets. In Collaborative Computing:Networking, Applications and Worksharing, pages623–631. Springer Berlin Heidelberg, 2009.

8. K. G. K. Gile, P. Russom, C. Moore, andC. Teubner. The Emergence Of Process-CentricBI. Forrest Research, Cambridge, MA, 2004.

9. D. E. Gray and T. R. Black. Prototyping ofcomputer-based training materials. Computers andEducation, 22(3):251 – 256, 1994.

10. R. Heuer. Psychology of Intelligence Analysis.Pherson Associates, 2007.

11. N. I and T. H. The Knowledge Creating Company.Oxford University Press, 1995.

12. IBM. Business analytics and optimization.

13. M. L. Markus, A. Majchrzak, and L. Gasser. Adesign theory for systems that support emergentknowledge processes. MIS Quarterly, 26(3):pp.179–212, 2002.

14. M. E. Nissen and J. Espino. Knowledge processand system design for the coast guard. Knowledgeand Process Management, 7:165–176, 2000.

15. P. Pirolli and S. Card. The sensemaking processand leverage points for analyst technology asidentified through cognitive task analysis. Proc.Int’l Conf. Intelligence Analysis, 2005.

16. SAP AG. Business management software.

17. X. Wang, S.-E. Chen, E. Hauser, andW. Ribarsky. imonitor: Architecture of web-basedcollaborative visual analytics system for bridgemanagement. In Transportation Research Board 90Annual Meeting, 2010.

18. X. Wang, W. Dou, S.-E. Chen, W. Ribarsky, andR. Chang. An interactive visual analytics systemfor bridge management. Computer GraphicsForum, 29:1033–1042, 2010.

19. X. Wang, B. Janssen, and E. Bier. Findingbusiness information by visualizing enterprisedocument activity. In Proceedings of theInternational Conference on Advanced VisualInterfaces, AVI ’10, pages 41–48, New York, NY,USA, 2010. ACM.

20. X. Wang, D. H. Jeong, W. Dou, S.-W. Lee,W. Ribarsky, and R. Chang. Defining andapplying knowledge conversion processes to avisual analytics system. Computers and Graphics,33(5):616 – 623, 2009.

21. J. Zimmerman, J. Forlizzi, and S. Evenson.Research through design as a method forinteraction design research in hci. In Proceedingsof the SIGCHI conference on Human factors incomputing systems, CHI ’07, pages 493–502, NewYork, NY, USA, 2007. ACM.