IN DEGREE PROJECT MEDIA TECHNOLOGY,SECOND CYCLE, 30 CREDITS

, STOCKHOLM SWEDEN 2020

Visualizing partitioned data in Audience Response Systems

A design-driven approach

OSCAR WIIGH

KTH ROYAL INSTITUTE OF TECHNOLOGY

SCHOOL OF ELECTRICAL ENGINEERING AND COMPUTER SCIENCE

Abstract

Meetingsandpresentationsareoftenmonologicalintheirnature,creatingabarrierof

productivityinworkplacesaroundtheworld.Byutilizingmoderntechnologiessuchas

aweb-basedAudienceResponseSystem(ARS),meetingsandpresentationscanbe

transformedintointeractiveexerciseswheretheaudience’sviews,opinionsand

answerscanbeexpressed.Visualizingtheseaudienceresponsesandrelatingquestions-

specificpartitionedanswersbetweeneachother,throughvisualizationstructures,was

thetopicofthisreport.Thethesisprojectwascarriedoutincollaborationwith

Mentimeter,creatorofaweb-basedARSandonlinepresentationtool.

TheDoubleDiamonddesignprocessmodelwasusedtoinvestigateandgroundthe

designanddevelopmentprocess.Toguidetheimplementationoftheprototypes,a

focusgroupwasheldwithfourvisualizationanddesignprofessionalsknowledgeable

aboutARSs,togatherfeedbackonhighfidelitysketches.

ThefinalprototypeswereevaluatedwiththeextendedTechnologyAcceptanceModel

(TAM)forinformationvisualizationtosurveyend-users'attitudesandwillingnessto

adoptthevisualizationstructures.Eightend-userstestedthefinalweb-based

prototypes.Thefindingsoftheusertestsindicatethatbothvisualizationsprototypes

showedpromiseforvisualizingpartitioneddatainnovelwaysforARSs,withan

emphasisonacircleclustervisualizationasitallowedforthedesiredexploration.

Theresultsfurtherimplythatthereisvaluetobegainedbypresentingpartitioneddata

inwaysthatallowsforexploration,andthataudienceswouldlikelyadoptafull

implementationofthevisualizationsgivensomeaddedfunctionalitiesandadjustments.

Futureresearchshouldfocusonfullyimplementingandtestingthevisualizationsin

frontofaliveaudience,aswellinvestigatingothercontemporaryvisualization

structuresandtheircapabilitiesforvisualizingpartitionedARSdata.

Sammanfattning

Mötenochpresentationeräroftaseddasomettproduktivitetshinderpåarbetsplatser

runtomivärldenpågrundavderasmonologiskanatur.GenomattanvändamodernatekniskalösningarsåsomwebbaseradeAudienceResponseSystems(ARS)såkanmöten

ochpresentationeromvandlastillinteraktivamomentdärenpubliksperspektiv,åsikter

ochsvarkanuttryckas.Attvisualiseraenpublikssvarochrelaterafrågespecifikapartitioneradesvarmellanvarandra,genomvisualiseringar,vardennarapports

huvudämne.ProjektetutfördesisamarbetemedMentimeter,skapareavettwebbaserat

ARSochdigitaltpresentationsverktyg.

DoubleDiamond-modellenanvändesförattundersökaochförankradesign-ochutvecklingsarbetetiprojektet.Förattguidautvecklingsarbetet,ochfåfeedbackpå

designförslag,genomfördesenfokusgruppmedfyravisualiserings-ochdesignexperter

sombesattkunskapomARS.

DeframtagnaprototypernautvärderasgenomdenutökadeTechnologyAcceptanceModel(TAM)förattundersökaslutanvändaresinställningochvillighetattanvända

visualiseringarna.Totalttestadeåttaslutanvändaredeframtagnawebbaserade

prototyperna.Resultatetavanvändartesternaindikeradeattbådavisualiseringsprototypernaharpotentialattvisualiserapartitioneraddatapånyasätti

ARS,menattenklustervisualiseringvaröverlägsenfrånenutforskningssynpunkt.

Resultateninnebärvidareattdetfinnsettvärdeiattpresenterapartitioneraddatapå

sättsommöjliggörutforskningavpublikenssvar,ochattpublikentroligenkommerattantaenfullständigimplementeringavvisualiseringarnaförutsattnågraextra

funktionerochjusteringar.Framtidaforskningbörfokuserapåattfullständigt

implementeraochtestavisualiseringarnaframförenfaktisktpublik,samtundersökaandrasamtidavisualiseringsstrukturerochderasmöjligheterattvisualisera

partitioneradARS-data.

Visualizing partitioned data in Audience Response Systems A design-driven approach

Oscar Wiigh Royal Institute of Technology

Stockholm, Sweden wiigh@kth.se

ABSTRACT Meetings and presentations are often monological in their nature, creating a barrier of productivity in workplaces around the world. By utilizing modern technologies such as a web-based Audience Response System (ARS), meetings and presentations can be transformed into interactive exercises where the audience’s views, opinions and answers can be expressed. Visualizing these audience responses and relating questions-specific partitioned answers between each other, through visualization structures, was the topic of this report. The thesis project was carried out in collaboration with Mentimeter , creator of a web-based ARS and online presentation tool.

The Double Diamond design process model was used to investigate and ground the design and development process. To guide the implementation of the prototypes, a focus group was held with four visualization and design professionals knowledgeable about ARSs, to gather feedback on high fidelity sketches.

The final prototypes were evaluated with the extended Technology Acceptance Model (TAM) for information visualization to survey end-users' attitudes and willingness to adopt the visualization structures. Eight end-users tested the final web-based prototypes. The findings of the user tests indicate that both visualizations prototypes showed promise for visualizing partitioned data in novel ways for ARSs, with an emphasis on a circle cluster visualization as it allowed for the desired exploration.

The results further imply that there is value to be gained by presenting partitioned data in ways that allows for exploration, and that audiences would likely adopt a full implementation of the visualizations given some added functionalities and adjustments. Future research should focus on fully implementing and testing the visualizations in front of a live audience, as well investigating other contemporary visualization structures and their capabilities for visualizing partitioned ARS data.

AUTHOR KEYWORDS Information Visualization; Audience Response Systems; Technology Acceptance Model; Double Diamond Design Process Model

ACM CLASSIFICATION KEYWORDS H.5.m. Information interfaces and presentation (e.g., HCI): Miscellaneous

1. INTRODUCTION

Meetings and presentations are an essential part of the modern workplace and school environment, yet many see them as a “barrier of productivity” [7] due to their often inefficient structure and monological nature. However, “when meetings are conducted effectively, they can be a positive outlet for employees to share opinions, collaborate and create” according to Gallup’s State of the American Workforce Report [7]. Unnecessary presentations or meetings should be eliminated and “instead use technology and collaboration forums to connect team members.”

One way of remedying inefficient meetings and following Gallup’s advice [7] is to utilize an audience response system. Audience response systems, hereafter referred to as ARSs, is a technology in which both large and small groups of people, typically in a meeting, lecture or presentation setting, interactively vote on presented topics or questions. The audience responses are then tallied up and presented by the system in the form of visualizations of varying complexity such as bar charts, pie charts, line charts etc. [12]. Several modern ARS applications, such as Kahoot, Mentimeter and Slido, utilize smartphones or other handheld devices for the voters to vote with, and a separate screen to display the results on [12]. ARSs typically include multiple questions or topics that the audience can vote on in succession, but can contain a single question or topic as well.

At its core, an ARS is a data collection and presentation tool. This means that an ARS is dependent upon how clearly it can summarize and convey an audience's responses back to the audience. This presentation of data is often limited to tallying up the responses in bar or pie

1

charts, but can with the help of modern web technologies encompass more advanced visualization methods that allow for exploration of the data. Visualizing audience responses that have been partitioned based on similar characteristics, can be used to reveal insights about how these subsets relate back to the overall dataset. In an ARS, relating audience responses between different questions in the same presentation is a common goal as this can show insights that would not be revealed while examining the responses of a single question in a visual structure. However, audience response systems have seen little research in this domain.

Of the mentioned visualization structures, typically only bar charts are partitioned into subgroups i.e. stacked bar charts. These subgroups are primarily used to display the audience’s earlier voting decisions (e.g. 33 % of those who voted on x also voted on y). The prevalence of other visualization structures in audience response systems is low, and users’ impressions of them is unknown.

1.1 Case Description

The thesis project was carried out in collaboration with the Stockholm based company Mentimeter, creator of a web-based presentation tool and audience response system. Mentimeter was interested in seeing how ARSs could introduce new ways of visualizing partitioned data for different question types. In practise this meant developing a validatable proof-of-concept of suitable visualization structures as a tool to understand how to visualize partitioned data in an ARS.

1.2 Research Questions

This degree project seeks to answer the following two research questions:

1. How can information visualization and partitioning facilitate the exploration of audience responses through relationships between questions in an ARS?

2. What are the end users’ attitudes towards the developed visualization structures and are they willing to adopt them?

1.3 Delimitations

For this thesis project, the Javascript library D3.js will be used because of its flexibility and usage in other ARSs [13]. The developed prototype will not be directly implemented in a live ARS, but rather be developed as a standalone web-based prototype as part of the design exploration process. Several lo-fi sketches will be ideated and two high

fidelity visualizations will be incorporated into the proof-of-concept prototype.

The ARS data will be delimited to answers from multiple choice questions and open-ended questions, as these are the most common question types in ARSs [8]. This report will not focus on the presenter’s interaction with the prototype but instead on the audience’s impression of the visualizations in an ARS setting. As such, real time voting will not be included in the prototypes.

2. BACKGROUND

This section details information visualization as a field and partitioning as a way of grouping data. It also describes how contemporary audience response systems use visualization structures to present data and what constitutes a good visualization.

2.1 Information Visualization

Information visualization is a discipline within computer science where datasets are transformed into visual structures that can reveal insights and patterns that would otherwise be hard or tedious to discern [4]. It is important to note that these insights are not “explicitly stored within the dataset but are inferred through visual pattern recognition” [14].

2.1.1 Visualization task overview According to Ben Shneiderman’s task by type taxonomy (TTT) within information visualization, there are seven high level actions that users wish to perform [18]. Below is a selection of these actions that are relevant for this thesis:

● Overview: Gain an overview of the entire collection.

● Zoom : Zoom in on items of interest. ● Filter: filter out uninteresting items. ● Details-on-demand: Select an item or group and

get details when needed. ● Relate: View relationships among items.

Relationships, correlations and multiway interactions, are types of complex insights that only can be revealed with the right type of visualization structure [14].

2.1.2 Partitioning data There are several definitions of what partitioning data can entail within the field of information visualization. In the context of this thesis, my definition of partitioning data is: The process of dividing and organizing objects into groups whose members are similar in some kind of way. In Audience Response Systems, visualizing partitioned data is often denominated as “segmentation” (see section 2.3) and

2

is represented by visualization structures that divide the separate answers to two questions into subgroups that relate back to each other. 2.1.3 Visual information processing Information visualization relies on the visual senses and cerebral processes to find meaning in the visualization structures. According to Pineo and Ware, “a good data visualization is one that triggers neural activity that encodes the desired information in the brain” [15]. According to Ben Shneiderman, the “bandwidth of information presentation” in the visual domain has the potential of being the highest out of all other mediums that appeal to our senses [18]. Humans can without effort “scan, recognize, and recall images rapidly, and can detect changes in size, color, shape, movement, or texture” [18]. As such, information visualization can be seen as a communication problem that should be decoded [15].

2.1.4 Aesthetics and usability A computerized system’s aesthetics and usability are closely connected according to research in the HCI field [19]. More specifically, the initial perception of an interface’s “beauty” affects the perception of a system’s usability positively. These early impressions endure both during actual system use and post-use. Conversely the actual usability of a system had no bearing on the post-use perception of usability and aesthetics [19].

2.2 Audience response systems

Modern ARSs are digital tools designed to collect and present responses from an audience. Audience response systems are largely prevalent within learning environments to increase student engagement [12], and have similarly been used in work meetings and presentation settings to achieve audience participation. The collection of audience responses is typically done through app or web-based interfaces where users can vote, react or input text. In audience response systems, audience responses are “instantly collected, summarized and presented” in a visual format on the presenter’s screen [12]. The visual format, or more accurately visualizations, are presented in front of the participating audience where the presenter has control of the presentation. 2.3 State-of-the-art (STAR) of ARSs

There are many examples of digital audience response systems, each with different ways of visualizing audience responses. Below is a summary of which visualizations popular ARSs include in their products, as of May 2020, 1

1 Alexa Global Rank above 500,000 as of May 2020.

and whether they have any functionality that visualizes partitioned data.

ARS/Visualization Bar

chart* Pie Chart* Word

cloud Scatter plot Radar

Chart Dot Matrix Chart Density

Plot Image Heatmap Partitioning

visualization Slido Yes No Yes No No No No No No

Kahoot Yes Yes Yes No No No No No No

Mentimeter Yes Yes Yes Yes Yes Yes Yes No Yes

Top Hat Yes No Yes No No No No Yes No

Socrative Yes No No No No No No No No Poll

Everywhere Yes Yes Yes Yes No No No Yes Yes

Klaxxon Yes Yes Yes No Yes No No No No

Wooclap Yes Yes Yes No No No No Yes Yes

Voxvote Yes No Yes No No No No No No

Directpoll Yes No No No No No No No No Pigeonhole

Live Yes Yes Yes No No No No No No

Swipe Yes No No No No No No No No

4 Screens Yes No No No No No No No No Figure 1: STAR matrix over ARS functionality * Pie charts include donut charts. ** Bar charts are used here as an umbrella term regardless of chart orientation or direction (e.g. column chart).

Out of the 13 ARSs, three included visualizations that were related to partitioning. These ARSs and their respective ways of visualizing partitioned data are described below:

Poll Everywhere has a feature called Segmentation (Figure 2a) that takes answers from one multiple choice question and partitions them according to team belonging. Users can through stacked bar charts view relationships between question answers and team belonging. The respective length of the bars correspond with the answer frequency of the current question, while colored segments divide the bars and constitute the respondent frequency of answers to the previous question. The segmentation feature is limited to data from multiple choice questions.

Wooclap has a feature called Comparison (Figure 2b) where it is possible to partition results from previous presentation sessions and compare them with the live audience responses in two separate donut visualizations. The feature works for multiple choice questions and open ended questions.

3

Figure 2: a) Poll Everywhere feature Segmentation b) Wooclap’s feature Comparison

Mentimeter has a similar feature, also named Segmentation , that is able to partition results according to what participants have voted for on another question (Figure 3). It has an identical implementation for stacked bar charts as Poll Everywhere, see Figure2.a, but also includes a dot matrix chart variant that is able to partition the data. The dot matrix chart variant groups dots along the y-axis based on answers to the current question, and then color codes the dots based on the answer of the selected partitioning question. The segmentation feature is limited to data from multiple choice questions.

Figure 3: Mentimeter’s partitioning feature Segmentation

2.4 Technology Acceptance Model

The Technology Acceptance Model ( TAM) [6] was used to evaluate the developed prototypes. The model is used to assess the users’ intention and willingness to adopt new technologies or systems. This is mainly achieved through analysing two main aspects: perceived usefulness and perceived ease of use. The perceived usefulness is defined as the user’s belief that the technology will improve their performance in completing a task, while the perceived ease of use is defined as the user’s belief that using the technology will be free of effort [6]. The TAM framework has been, in the field of Information Systems, “widely applied, validated and cited” [15, p. 47]. Figure 4 shows the workflow of the TAM framework.

Figure 4: TAM framework depiction [5]

The TAM framework has been extended to the field of information visualization [2], with additions that further make it possible to evaluate if a visualization technique will be adopted by its potential users. One such addition is the perceived authority aspect which focuses on soft factors and “represents the degree to which a person is confident that using a particular information visualization technique is a good choice” [2].

The TAM framework has been used in previous studies as the theoretical framework for examining, for example, users’ intention to adopt business information visualizations [17] and as such was deemed to be a fitting evaluative framework despite its relatively novel approach within the field of information visualization.

2.5 Double Diamond Design Method

For the prototype development, the Double Diamond design process model was chosen as the design method. The design method fit the project's iterative workflow boundaries, and ensured “an initial assessment of project type and the approach needed to address a specific challenge” [3]. The Double Diamond method has been successfully employed in previous projects [3] and was also chosen because of familiarity with the method. The model is divided into four distinct phases and “it maps the divergent and convergent stages of the design process” (Figure 5) [3].

The Discover phase is the beginning of the design process and includes information gathering through for example market research, literature studies and requirement analysis. The Define phase is where the findings from the Discovery phase “are analysed, defined and refined as problems”. Reviewing, selecting and filtering out ideas and information is essential for this stage of the process. It is succeeded by the Develop phase where “design-led solutions are developed, iterated and tested”. Concluding the design process is the Deliver phase where the end result is finalized. [3]

Figure 5: Double Diamond design process model [3]

4

3. METHOD & IMPLEMENTATION

This chapter introduces the methodology of the study including how the pre-study, prototype development, focus group, user testing and data analysis were conducted and structured.

3.1 Double Diamond Design Method

This section presents the development of the visualization prototypes that were then evaluated in the user tests. The development process is presented in accordance with the structure of the Double Diamond phases (Figure 5): Discover, Define, Develop, Deliver [3].

3.1.1 Discover The Discover phase consisted of a literature study in the field of visualization (section 2) along with a STAR-analysis (section 2.3) where audience driven visualization products and services were compared and the design space explored [10]. During this requirement analysis phase, it was important to identify the two primary inputs to design: (1) the characteristics of the information to be visualized and (2) the types of insights that the visualization should ena ble [14].

The data to be visualized was not directly provided by an ARS, but rather inspired by presentation questions from the Segmentation feature in Mentimeter. The data was focused on two questions, and their answers, in one specific scenario that was deemed typical for current partitioning implementations. The data mainly consisted of answers to open-ended questions and multiple choice questions (as delimited in section 1.3). While data from ARSs contains many data points, the relevant data points for this thesis are described below.

● Primary question - The primary questioned to be partitioned.

● Secondary question - The question that partitioned the answers to the primary question.

For each voter:

● Unique participant id - An alphanumeric id ● Answer to primary question - Either a multiple

choice answer, or free text answer. ● Answer to secondary question - Either a multiple

choice answer, or free text answer.

The data was thus multidimensional as the answers of the primary question were subsets of the answers of the secondary question. The specific dummy data used is outlined in section 3.1.6.

3.1.2 Define During the second step of the design process, the Define phase, the knowledge acquired during the Discover step

was condensed and sorted which resulted in several design hypotheses and guidelines. Key findings included that a visualization within an audience response system cannot be too labour intensive, i.e. visually complex, for the audience to interpret. Since audience response systems typically only show the results, and by extension the visualizations, for a short period of time, as opposed to other more exploratory visualization tools, the visualizations embedded in an ARS have to be easily digested. This was clear from the state-of-the-art analysis as contemporary ARS mainly limit their partitioning approaches to basic pie charts and stacked bar charts. In addition, the visualizations should not be developed without their aesthetics in mind, as this is an important factor for the perceived usability. Another key aspect that the visualizations should enable is the ability to give a clear overview of the dataset and at the same time being able to relate items between each other.

These were the general design guidelines that were distilled based on the Discover phase:

● The visualizations have to be aesthetically pleasing to encourage the perception of usability.

● The visualizations have to be easily digested by the audience in a presentation setting due to limited time.

● The visualizations should enable the audience to relate items in the visualization.

● The visualizations should require little prior experience to be properly understood, as end-users of ARSs are a diverse target audience.

● The visualizations have to include an overview and be able to convey details on demand.

● The visualizations should not be too general in their implementation, and should rely on the ARS presenter to find fitting use cases.

3.1.3 Develop - Low fidelity sketching At the start of the Develop phase of the design process, low fidelity sketching, as shown in Figure 6 and 7, was used as a way of interpreting the findings and setting a “design challenge” [3]. The low fidelity sketches were influenced by the STAR-analysis as well as “The Data Visualization Catalogue” [16]. The low fidelity sketches detailed, on a conceptual level, visual structures and interactions, that allowed for the visualization of partitioned data, but had no embedded interaction by themselves. The low fidelity sketches were then evaluated based on their potential to gene rate the expected insights in accordance with the Define phase.

In Figure 6, a layered bar chart, a stacked bar chart on cards, and a tree map are presented. The leftmost layered bar chart displays the answers the participants had given to

5

a previous question as bar charts inside larger bar charts. The stacked bar chart shows previously given answers as a bar chart stacked upon a text card. The tree map displays data hierarchically through using nested rectangles.

Figure 6: Preliminary sketches of a layered bar chart, a tree map and a stacked bar chart

Figure 7 displays two other visualization structures. The leftmost sketch showcases an outline of a Marimekko chart where both axes are in percentual scales, thus visualizing the partitions in a mosaic-like manner. The last visualization was the sunburst diagram. The answers the participants give to the different questions become nodes from the origin point in the middle, which gives the graph it’s characteristic “sunburst” look. Ultimately, the Marimekko chart and the tree map visualization were discarded in this phase after evaluating them based on the criteria set in the Define phase (section 3.1.2).

Figure 7: Preliminary sketches of a Marimekko chart and sunburst diagram

3.1.4 Develop - High fidelity sketching During the high fidelity sketching, a circle cluster visualization replaced the discarded low fidelity sketches, as it better aligned with the design guidelines from the Define phase, and was made high fidelity in preparation for the focus group. The resulting high fidelity sketches are presented in Figure 8 and 9.

Figure 8: a) Stacked bar chart on cards b) Layered bar chart

Figure 9: a) Sunburst diagram b) Circle cluster

3.1.5 Develop - Focus Group The focus group, with the resulting high fidelity sketches, was conducted with four visualization and design professionals from Mentimeter . During the focus group session, which was conducted in March 2020, four different sketches of visualizations were presented to the participants. The focus group included the following:

● An introduction to the topic area and a clear description of partitioning by the focus group leader.

● A brainstorming session of what visualizing partitioned data can enable for users of an ARS.

● A presentation and discussion of the created design proposals. To guide the discussion, the participants had the ability to vote on the two design proposals they found most promising. The two most voted upon proposals were discussed more thoroughly.

● A design exercise where the participants, in groups of two, could tweak and combine the design proposals in sketches. The design exercise was concluded by a group discussion where each pair explained their creations.

● The participants were asked to send all sketches and notes digitally at the end of the focus group.

The collected data from the focus group session was analyzed through thematic analysis [1], and used to guide further development. The results of the focus group are summarized below.

The first sketch (Figure 8a) was seen as a good addition to contemporary visualizations of open-ended questions as stacked bar charts are used elsewhere in ARS but not in this context. Several participants mentioned that a criteria for this visualization is that the audience responses are either simple by nature or grouped by similarity and not exact spelling, as this would likely result in an ineffective visualization structure. The unanimous opinion of the layered bar chart (Figure 8b) was that the same functionality is achieved with a stacked bar chart more concisely and this structure would likely struggle to give a clear overview of the data when the number of options is large.

6

The sunburst diagram (Figure 8a) was deemed to be a potential solution for a small number of options but the connection between the inner and outer segments were regarded by the participants as non intuitive. The circle cluster visualization (Figure 8b) was voted by the focus group participants as another promising visualization structure. The size legend in the corner of the visualization was expressed to be irrelevant as the exact size of the circles was secondary to the comparison between the circles. The outer ring, delineating the visualization was criticized for being superfluous and not adding any additional information. Additionally it was suggested during the design exercise that the large circles answered the secondary question rather than the primary question and should be swapped.

Based on the focus group feedback, the stacked bar charts on cards and circle cluster visualizations were the two visualizations chosen for further development.

3.1.6 Deliver The final prototypes were created with D3.js and React . The visualizations were populated with dummy data, inspired by presentations in Mentimeter, that simulated real world questions seen in contemporary ARS systems. The questions in the dummy data were chosen to exemplify the functionality of the visualizations clearly and were structured as follows:

● Primary question (e.g. “What is your favorite animal?”)

● Secondary question (e.g “What is your role?”)

For each voter:

● Unique participant id (e.g. 27) ● Answer to primary question (e.g “Dog”) ● Answer to secondary question (e.g “Sales”)

Figure 10: Prototype 1: Stacked bar charts on cards

Figure 10 shows the first prototype, based on the stacked bar charts on cards visualization. It was directed at the open ended question type with the second question being a multiple choice question. The answers in the dummy data (dog, cat, etc.) were simple, but answers grouped by similarity would work in the same manner. Each card represented an answer by the audience while the stacked bar chart, adjoining each card, showed the color coded distribution of answers from the second question (“What is

your role?”). When updating the data of the prototype, the bars were smoothly animated to resize and reorder accordingly. Toggling between the percentages or actual votes in numbers was done by clicking the bars of each card.

Figure 11: Prototype 2: Circle cluster

Figure 12: Prototype 2: a) First zoom level of the circle cluster b) Second zoom level of the circle cluster

The second prototype is displayed in Figure 11 and 12. It was based on the circle cluster sketch and subscribed to the notion of “Overview first, zoom and filter, then details-on-demand“ [18]. This final iteration removed the legend, the redundant border, added a gray background to each cluster for improved readability, and added another zoom level to individual responses in accordance with feedback from the focus group. The clusters represented answers to the multiple choice primary question (“What is your favorite animal?”), and was partitioned by answers to the second question (“What is your role?”).

These two prototypes (Figure 10,11,12) were the subjects of the final user tests, the results of which are presented in section 4.1.

3.2 User Tests

The user tests of the final prototypes were conducted remotely via the video meeting software Zoom . As the study focuses on how audience response systems can visualize partitioned data, the main point was how the audience perceived the visualizations. Thus, the user tests did not focus on how the presenter experienced the visualizations

7

but rather how the audience did. As such, the user tests were conducted by a test leader who interacted with the web-based prototype and acted as the presenter, while the test participant was seen as part of an audience. The user tests did not include real time smartphone voting, as this was deemed non-essential to the perception of the partitioning visualizations. This setup mimicked the real world use of an audience response system when the audience had voted. The user tests of the high fidelity prototypes allowed eight end-users to individually observe the visualizations being interacted with and answer survey questions.

Eight people participated in the final user tests. Six of the eight participants defined themselves as male and two as female. The age of the participation spanned between 19 and 60. All participants had limited prior experience with ARSs.

3.2.1 Visualization task overview The participants were first asked to explain how they thought the visualizations worked while mistakes in their explanations were noted. Secondly, they were asked to interpret the visualizations by questions related to the dummy data. A simple example of such a question was:

“ Which animal is the second most popular amongst marketers? ” 3.2.2 Prototype survey questions Following the visualization tasks, the participants were asked to evaluate statements about the perceived usefulness, ease of use and authority in accordance with the extended TAM framework for information visualization [2]. The survey statements were repeated for each of the two visualizations. The evaluation answers were graded on a seven point Likert scale from “strongly disagree” to “strongly agree”. The prototypes were presented in random order to the test participants to avoid anchoring bias . Statements regarding perceived ease of use:

1. It was easy to follow and understand the interaction of this visualization.

2. Previous knowledge is not required for using and understanding the visualization.

3. The visualization is easy to manipulate and to adapt to specific purposes.

4. The information visualization can be used at its full potential swiftly

Statements regarding perceived usefulness:

1. The visualization helps to focus on relevant aspects and speeds up the analysis process.

2. With the information visualization tasks can be achieved promptly.

3. The information visualization leads to the emergence of new insights from the data.

Statements regarding perceived authority:

1. The visualization is fun and a pleasure to the eye. 2. I would recommend this visualization to a friend. 3. I would gladly use this visualization in a

presentation setting. 4. People in a professional setting would use this

visualization (academia, industry, etc.) At the end of the user tests, participants were asked to answer three concluding questions:

1. Did you find these kinds of visualizations fitting for an audience response system?

2. Would you use a full implementation of this prototype in an audience response system?

3. Was there any functionality lacking in this prototype that you would like to add?

3.3 Data Analysis The user test data consisted of audio and video recordings of each participant's impression of the prototype, alongside survey data in the form of text and Likert scale answers, and was analyzed through the content analysis framework Thematic Analysis [1]. The first step of the analysis was familiarization where the user test recordings were transcribed. From these transcriptions then came the task of coding phrases and sentences into shorthand labels that were used to gather an overview of how the visualizations had been perceived by the participants. The codes were then combined and further abstracted into overarching themes that were finally reviewed by their relevance to the thesis. None of the themes were predetermined before the user tests in accordance with Conventional Content Analysis [9] but rather generated from the collected data.

The quantitative data from the Likert scales in the survey was passed through a fitting mean value analysis as a complement to the qualitative data. Furthermore, the number of mistakes made when explaining the visualizations were averaged.

4. RESULTS

This section summarizes the results of the user tests

4.1 User Tests

The findings of the user tests are graphed but also categorized into three categories: Suitability, usefulness and added functionalities. Numbers reported in this section are from the likert scales (1-7). The stacked bar chart on cards visualization will be referred to as the first prototype, and the circle cluster visualization as the second prototype.

8

Figure 13: Likert score averages from the perceived ease of use statements

Figure 14: Likert score averages from the perceived usefulness statements

Figure 15: Likert score averages from the perceived authority statements

4.1.1 Suitability All respondents (R1 through R8) perceived both of the visualizations as suitable for an audience response system. R3, R4, R6, and R7 commented that the visualizations enable the user to quickly understand the results with ease. R8 commented on the second prototype “You don’t have to decipher numbers and can just get an intuitive sense of the partitions quickly”. In addition, the models made finding new correlations an easy process, according to R3, R4, and R8.

Several of the respondents also mentioned how practically anyone could interpret the visualizations with ease without prior knowledge. For the first prototype, R5 and R8 did however make the mistake of thinking that the bar charts were connected to the overall dataset and not unique to each answer. For the second visualization, no errors when attempting to explain the visualization were made by the participants.

In the survey section, the statement regarding whether professionals in academia or industry would use the prototypes resulted in a mixed questionnaire responses and relatively low averages for both prototypes (Figure 15). In response to the willingness to use the first prototype in a group presentation-situation, the average score was 6.6 (Figure 15). Recommending the first prototype to a friend was given an average score of 6 (Figure 15). In regard to the statement about the first protoype’s visual design; “The visualization is fun and a pleasure to the eye ”, the average score was 5.4 with no outliers (Figure 15). R5 commented however that “the [first] visualization benefitted from being barebones and simplistic”. The majority of the participants were likely to recommend the second prototype to a friend, with an average score of 6.5 (Figure 15). Regarding the statement “The visualization is fun and a pleasure to the eye ”, the second prototype received an average score of 6.6 (Figure 15).

4.1.2 Usefulness All respondents agreed that the first visualization prototype gave an overview of the audience responses. The majority of the respondents also reported that the prototype sped up their analysis process (Figure 14). Furthermore, the average score relating to the statement: “The information visualization leads to the emergence of new insights from the data” was 6.5 (Figure 14). In regard to specific functionalities, seven out of the eight participants noted that the toggle between percentages and number of votes was useful, with one participant having no opinion. The respondents also found the first visualization to be easy to manipulate and adapt to specific purposes, with an average score of 6.3 (Figure 13). Several participants noted however that the number of responses could become a limiting factor, with R2 noting: “I would use this but it assumes that you don’t have too many respondents as this might make the visualization less clear”. The willingness of using the second prototype in a group presentation-situation was identical to that of the first visualization prototype (Figure 15). Similarly, the second prototype was also seen by the respondents as useful in giving a good overview of the audience responses. Pertaining to the statement of whether the second visualization prototype leads to new insights, the average score was 6.8 (Figure 13).

9

The majority of the respondents answered that they would use a full implementation of the presented prototype. R6 reported a willingness to use the prototype in an academic as well as in a work setting, referencing its ease of use. R3 admitted to rarely using such systems but said that they saw the visualizations’ purpose and potential. R2’s and R1’s answers highlighted that the prototype is useful in situations when larger audiences are divided into subgroups. R1 expressed a preference for the second prototype when it comes to larger group gatherings sizes and a preference for the first prototype when presenting data from a smaller group of respondents.

4.1.3 Added functionalities The responses to whether any functionality was lacking from the prototype were varied. R1, R3, R5, and R7 all expressed the need for percentages and a total number of votes for each answer; especially in the first prototype. R6 expressed that filtering options were lacking from both of the prototypes, where filtering out answers from the partitioning question would be useful. R4 expressed a similar concern where the ability to organize the order of the colors in the bar charts would be a “nifty” functionality. R1 found that a functionality where the larger cluster bubbles in the circle cluster-model could fade out as you hover over them was missing. R7 added that a greater selection of colors and symmetry could enhance the second prototype. R4 voiced privacy concerns in regards to the circle cluster visualization suggested a feature of toggling anonymized results. Finally, R6 suggested that the placement of text in the second prototype could be moved above each cluster to improve usability.

5. DISCUSSION

This section discusses the user tests results, their implications for the research questions and future research, as well as suggestions for methodology improvements. 5.1 General Discussion

The goal of this report was to investigate how visualizations for partitioned audience response data could be constructed to allow for exploration, and survey end-users’ attitudes and willingness to use a full implementation of said visualizations. The first prototype was graded as easy to understand, but when asked to explain its functionality, several participants wrongly assumed that the bar charts were connected to the overall dataset and not unique to each answer. This indicated that although the first visualization was positively perceived as simplistic and barebones, which aligned it with

contemporary ARSs as evident by the STAR-matrix (Figure 1), the understanding has to be correct upon first time use. In regard to how much exploration the first visualization allowed for, the users expressed it as limited as indicated by the lack of details on demand. Displaying the total amount of votes per answer on demand and filtering the answers was noted as ways of remedying this to some extent. The second prototype was perceived to be more pleasing to the eye than the first visualization, while also being rated higher in perceived usability. This visualization did however elicit feedback on colors, text placement and hover functionality which was thought to enhance the usability. This does, to some extent, fortify the claims that the aesthetic qualities of a computerized system’s impacts its perceived usability as laid out by Tractinsky et al. [19]. The second prototype was also found to be more intuitive in regards to presenting an overall summary of the results. These results go in line with Shneiderman’s principles, that a clear overview is paramount when visualizing data [18]. The results indicated that some participants thought the dummy data was simple in nature and that the applicability to more complex question pairs might be limited. However, the intended use for these visualization structures requires the presenter to identify the characteristics of the data [14], and the visualizations were not developed to become all encompassing solutions as laid out by the Define phase guidelines. Overall, the user test participants often echoed said guidelines that were laid which confirmed their importance, for example clear overviews, little prior knowledge needed, and the importance of aesthetics . An addition to both prototypes would be a filtering functionality, that the participants expected to be there and follows Shneiderman’s task by type taxonomy [18]. Both visualizations worked for open ended questions but this required some sort of grouping by similarity function, or questions that collected simple answers. The likelihood of using a full implementation of the developed prototypes, was further indicated by the soft factors as addressed by the extended TAM framework [2]. However, the statement regarding academica’s and industry’s potential usage of the visualizations collected a mixed response and was deemed difficult to draw any conclusions from, partly because the statement was interpreted as confusing. Despite this, the overall implications of the user tests were that both visualizations showed promise for visualizing partitioned data in novel ways for ARSs, with an emphasis on the circle cluster visualization as it allowed for more exploration. The results further indicated that the participants’ attitudes were mainly

10

positive to the developed prototypes, specifically when addressing the participants’ willingness to recommend them to a friend, their perception of the prototypes’ aesthetics and their opinion that these visualizations could provide new insights into the data. These findings imply that there is value to be gained for an audience by presenting partitioned data in a way that allows for exploration, and that end-users would potentially adopt a full implementation of the visualizations given some added functionalities and adjustments. Introducing visualizations for partitioned data could possibly shift ARSs from being just data collection and presentation tools, to becoming effective aids for audiences to explore connections and trends in their responses and gather insights. 5.2 Method Criticism

5.2.1 General areas This report resulted in two types of visualizations that were deemed fitting for an ARS. As the pre-study showed, there are multiple ways of visualizing data for ARSs and by taking a design-driven approach [3] to the problem, other potential solutions, design guidelines and visualizations are bound to be omitted. By extending the Double Diamond design process model to include other criteria, such as familiarity and decoding time [15], a narrower selection of potential visualization structures could have been presented. The resulting prototypes were rudimentary in their nature, which could be seen as a drawback. However, it was deemed important for the visualizations to be non-ambiguous and conceptually precise. Relating items in the visualizations could, however, be expanded to allow for more individual exploration of the data. The data was inspired by ARS questions from Mentimeter , and questions that are typically asked in these systems. However, utilizing actual questions and answers from real presentations could have ensured that the visualization structures worked for other scenarios as well. In addition to this, the dummy data from open-ended questions was decided to be virtually identical to the open-ended question answers. Investigating how to partition open-ended answers that are not identical or grouped by similarity could also have been done. The focus group of the high fidelity sketches was conducted with design and visualization professionals, yet the final prototypes were tested on end-users. Involving novice or experienced end-users earlier on in the development process, such as the focus group or a pilot study, could potentially reveal important aspects earlier in the design process. The TAM framework, while established in other domains, is relatively novel within the field of information

visualization[2]. The extended TAM framework for information visualization did work well as an evaluative framework for this report but the model needs to be validated through systematic studies to ensure transferability between domains. A substantial part of the results in this report consisted of qualitative data, that were analyzed through thematic analysis . Thematic analysis can miss nuanced data, which requires thoroughness from the researcher.

5.2.2 Ongoing pandemic As a consequence of the Covid-19 outbreak the focus group and user tests were held remotely. The video conferencing tool Zoom was used where each respondent was invited to a private session. The focus group-meeting was originally thought to be a collaborative and hands-on design exercise. The participants were intended to be paired into groups of two and would through the use of scissors and pencils extract and improve elements from each design sketch. Instead, the participants in the focus group were instructed to discuss and illustrate, in groups of two, potential improvements to each sketch. The results of the remote focus group may have been impacted by the absence of physically manipulating print-outs of the visualizations. Furthermore, the ideation process might also have been hampered by not having the focus group in person. During the user tests the test leader shared their screen, ensuring that the prototypes and questions were displayed correctly. One downside from having to perform the user tests remotely is that the user engagement can be lower and the qualitative data not as rich [11]. 5.3 Ethical Aspects

During all of the conducted user tests, the participants were asked to give their informed consent concerning their verbal responses and their screen actions being recorded. When using visualization techniques, it is important to consider the ethical aspects of the information visualizations. Audience response data could easily be represented in a way that indirectly, or directly, invites the user to misinterpret it. During this degree project, it has been of importance to create visualizations that do not skew the data in a misleading way. Additionally, in the circle cluster visualization, users could be identifiable by their self-chosen usernames which could cause privacy issues.

5.4 Future Work

This thesis delimited itself to multiple choice and open ended questions while future research could focus on different question types such as ranking questions, word clouds etc. In addition to this, Figure 1 suggests that there

11

are a plethora of visualizations used in ARSs, and their ability to visualize partitioned data could be investigated.

The prototypes could additionally be tested more extensively with a larger amount of user tests. Furthermore, expanding the functionality of the prototypes to receive real time voting data and conducting the tests in an actual presentation setting with a live audience, could lead to deeper insights. This thesis only focused on the audience’s perspective, but testing the presenter’s ability to understand the visualizations is also paramount for its adoption and usage. 6. CONCLUSION

In this thesis report, two visualization prototypes were developed in accordance with the Double Diamond design process in an attempt to visualize partitioned data from an ARS. The prototypes were to varying extents perceived as usable and able to visualize partitioned data, with a cluster circle visualization being perceived as the most fitting approach. The results indicated overall that the interactive visualizations were perceived as being useful and comprehensible by an audience. Furthermore, the results, based on the extended TAM framework, indicated that there is potential for end-users to adopt a future full implementation. Going forward, the prototypes that emanated from the design process show that there are many different possible tweaks and implementations choices, but that an audience can gather new insights from such exploratory ARS visualizations. Future prototypes should be extended to include real time audience voting and be tested in a presentation setting to corroborate and expand on the findings of this report. The area of visualizing data in an ARS in general, and partitioning data in particular, is still underresearched but this report shows a direction for future research. 7. ACKNOWLEDGEMENTS I am especially indebted to Taylor Plante and Charlotte Ristiniemi who were always eager to help and made the completion of this thesis possible. I want to express my gratitude to everyone at Mentimeter, especially those who came with design ideas and participated in my design workshop. My supervisor Björn Thuresson at KTH deserves much praise for providing well-needed guidance in several stages of this thesis project. Finally I want to thank everyone who participated in my user tests, and my classmates and friends who have tirelessly helped me proof-read this thesis. REFERENCES 1. Richard Boyatzis. 1998. Thematic Analysis and Code Development.

Sage.

2. Bresciani, Sabrina & Eppler, Martin J. (2015) Extending Tam to Information Visualization: A Framework for Evaluation. Electronic Journal of Information System Evaluation , 18 (1). 46-58.

3. British Design Council. 2005. Eleven lessons. A study of the design process. Retrieved February 10, 2020 from https://www.designcouncil.org.uk/sites/default/files/asset/document/ElevenLessons_Design_Council%20(2).pdf

4. Katy Börner, Andreas Bueckle, and Michael Ginda. 2019. Data visualization literacy: Definitions, conceptual frameworks, exercises, and assessments. Proceedings of the National Academy of Sciences 116, 6: 1857–1864. https://doi.org/10.1073/pnas.1807180116

5. Davis FD, Venkatesh V. A critical assessment of potential measurement biases in the technology acceptance model: Three experiments. International Journal of Human-Computer Studies.1996;45(1):19–45. https://doi.org/10.1006/ijhc.1996.0040 . https://doi.org/10.1371/journal.pone.0216007.g001

6. Davis, F.D. 1989. Perceived usefulness, perceived ease of use, and user acceptance of information technology. MIS Quarterly 13, 3, 319-340.

7. Gallup. 2017. State of the American Workplace Report. State of the American Workplace Report, 144.

8. Christian Haintz, Karin Pichler, and Martin Ebner. 2014. Developing a Web-Based Question-Driven Audience Response System Supporting BYOD. JOURNAL OF UNIVERSAL COMPUTER SCIENCE 20: 39–56. https://doi.org/10.3217/jucs-020-01-0039

9. Hsieh, H. and Shannon, S. 2005. Three Approaches to Qualitative Content Analysis. Qualitative Health Research 15, 9,1277-1288.

10. Eunsuk Kang, Ethan Jackson, and Wolfram Schulte. 2011. An Approach for Effective Design Space Exploration. In Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) . 33–54. https://doi.org/10.1007/978-3-642-21292-5_3

11. Korey Johnson. 2020. Remote moderated testing: An alternative ffor face-to-face user research during COVID-19. Retrieved May 28th, 2020 from https://www.uintent.com/blog_article/remote-moderated-testing-an-alternative-for-face-to-face-user-research-during-covid-19

12. Robin H. Kay and Ann LeSage. 2009. Examining the benefits and challenges of using audience response systems: A review of the literature. Computers & Education 53, 3: 819–827. https://doi.org/10.1016/j.compedu.2009.05.001

13. Maxwell McGee. 2018. Updated poll chart technology delivers speed and high impact visuals. Retrieved May 15th, 2020 from https://blog.polleverywhere.com/updated-poll-charts/

14. Don Norman. 2016. The Design of Everyday Things . Vahlen. https://doi.org/10.15358/9783800648108

15. Daniel Pineo and Colin Ware. 2012. Data visualization optimization via computational modeling of perception . IEEE Transactions on Visualization and Computer Graphics. https://doi.org/10.1109/TVCG.2011.52

16. Severino Ribecca. 2019. The Data Visualization Catalogue. Retrieved February 10, 2020 from https://datavizcatalogue.com/index.html

17. Schelkle, M. 2017. 'An Assistance System for Business Information Visualization'. In: Maedche, A., vom Brocke, J., Hevner, A. (eds.) Designing the Digital Transformation: DESRIST 2017 Research in Progress Proceedings of the 12th International Conference on Design Science Research in Information Systems and Technology. Karlsruhe, Germany. 30 May - 1 Jun . Karlsruhe: Karlsruher Institut für Technologie (KIT), 46-54

18. Ben Shneiderman. 1996. The Eyes Have It : A Task by Data Type Taxonomy. In Proc. IEEE Visual Languages ’96.

19. N. Tractinsky, A. S. Katz, and D. Ikar. 2000. What is beautiful is usable. Interacting with Computers . https://doi.org/10.1016/S0953-5438(00)00031-X

12

www.kth.se

TRITA-EECS-EX-2020:491