26 th annual management information systems (mis ) conference february 14, 2013

Fulcrum IT Services Company | 7403 Gateway Court | Manassas, VA 20109 703.331.1500 | http://www.fulcrumit.com

1

Fulcrum IT Services, LLC | 5870 Trinity Parkway, Centreville, VA 20120 703.543.0000 | http://www.fulcrumit.com

26th Annual Management Information Systems (MIS) Conference

February 14, 2013

National Assessment of Educational Progress (NAEP) Observable Data From Computer-Based Assessments

Gregory Anderson, Ph.D.Senior Usability Expert

Fulcrum IT, LLC


2


Outline

What are observable data?

What can we observe in the data?

How have we used what we’ve observed?

How can we observe more?


3


Outline






4


What are Observable Data?

Computer actions recorded into a database

Possible with Computer-Based Assessments (CBA)

Automatic recording of low level information – Initially

• Number of key presses• Number of word processing functions (cut, copy, etc.)• Button clicks (Next, Back, etc.)• Screen coordinates (mouse click)

– Recently added:• Time stamping• Extended data (keys pressed, etc.)


5


What are Observable Data?CBA2011 (Math Computer Based Study MCBS Grade 8 – Adaptive)1 Enter Item2 Exit Item3 Change Volume4 Change Language5 Keypress Count6 Click Choice7 Click Progress Navigator8 Keyboard Cut9 Right Click Cut10 Keyboard Copy11 Right Click Copy12 Keyboard Paste13 Right Click Paste15 Text-To-Speech17 Collapse Prompt18 Expand Prompt19 Collapse Response20 Expand Response21 Zoom22 Admin Move-To-Next-Block23 Hide Timer24 Show Timer25 Change Skin26 Application Recovery28 Swap Mouse Button29 Play30 Pause

31 Reach End of Tutorial32 Backspace33 Delete34 Clear Answer35 Mouse Cut36 Menu Cut37 Mouse Copy38 Menu Copy39 Mouse Paste40 Menu Paste41 Mouse Bold42 Keyboard Bold43 Menu Bold44 Right Click Bold45 Mouse Italic46 Keyboard Italic47 Menu Italic48 Right Click Italic49 Mouse Underscore50 Keyboard Underscore51 Menu Underscore52 Right Click Underscore53 Mouse Indent54 Mouse Outdent55 Mouse SpellCheck56 Menu SpellCheck57 Mouse Misspellings

58 Misspellings Identified59 Misspellings Corrected60 Menu Thesaurus61 Right Click Thesaurus62 Thesaurus Replacement63 Item Scroll64 Response Scroll65 Stimulus Scroll


6


What are Observable Data?CBA2012 (2012 WCBA Pilot Grade 4)66 Drop Choice67 Keypress68 Highlight Mode On69 Highlight Mode Off70 Highlight Text71 UnHighlight Text72 Eliminate Mode On73 Eliminate Mode Off74 Eliminate Choice75 UnEliminate Choice76 Mouse Select All77 Keyboard Select All78 Menu Select All79 Mouse Undo80 Keyboard Undo81 Menu Undo82 Mouse Redo83 Keyboard Redo84 Menu Redo85 Right Click SpellCheck86 Mouse Thesaurus87 Click Word Definition88 Display Reminder89 Pause Workstation90 Resume Workstation

91 Enter Student Control Panel92 Leave Student Control Panel93 Menu Indent94 Menu Outdent

TEL2012 (2012 TEL Tryout Grade 8)95 Close Word Definition

CBA2013 (2013 TEL Pilot Grade 8)96 Reset Booklet97 Next98 Back99 Yes100 No101 OK102 Media Interaction


7


Event ID

Event Type ID

Time Stamp Booklet ID Item ID

Block ID

Extended Data a (misspelling)

Extended Data b (corrected to)

66784 59 21:40.2 4069 41 36 exersize exercise

68085 59 42:47.8 4069 35 10 begining beginning

68255 59 45:05.3 4069 35 10 ningas ninja

74559 59 39:41.2 4071 70 24 lemonad lemonade

74969 59 44:53.9 4071 70 24 fliyers fliers

75609 59 57:18.7 4071 70 24 arived arrived

75614 59 58:25.7 4071 70 24 haelp help

77638 59 28:15.8 4071 37 12 monkey.You monkey. You

Putting the “Data” in Observable Data

Event Type ID Event Type58 Misspelling Identified

59 Misspelling Corrected


8


Observable Data Sources

2011 Writing Computer-Based Assessment Pilot – Grade 8: 24,281 Students– Grade 12: 28,238 Students

2011 Math Computer-Based Study– Grade 8: 8,427 Students

2012 Writing Computer-Based Assessment– Grade 4: 13,005 Students


9


Outline






10


What Can We Observe?

Frequency/Proportion (event ID, booklet ID, total number of participants)

End State (event ID, booklet ID, order of events)

Time Course (event ID, time stamp)

Duration/Intervals (event ID, 2 time stamps, math)

Qualitative (event ID, extended data)

Big Picture Qualitative (all data, other data sources)

Easy

Difficult


11






Duration / Intervals (event ID, 2 time stamps, math)



Easy

Difficult


12


Observing Frequency/Proportion

How many times did the student use the mouse to cut? How many times did the student use the [CTRL+X] cut keystroke combination? How many times did the student use the menu to cut? How many times did the student cut using the right-click context menu? How many times did the student use the mouse to copy? How many times did the student use the [CTRL+C] copy keystroke combination? How many times did the student use the menu to copy? How many times did the student copy using the right-click context menu? How many times did the student use the mouse to paste? How many times did the student use the [CTRL+V] paste keystroke combination? How many times did the student use the menu to paste? How many times did the student paste using the right-click context menu? How many times did the student use the mouse to click the bold button? How many times did the student use the [CTRL+B] bold keystroke combination? How many times did the student use the menu to boldface? How many times did the student boldface using the right-click context menu? How many times did the student use the mouse to click the italicize button? How many times did the student use the [CTRL+I] italic keystroke combination? How many times did the student use the menu to italicize? How many times did the student italicize using the right-click context menu? How many times did the student use the mouse to click the underline/underscore button? How many times did the student use the [CTRL+U] underline/underscore keystroke combination? How many times did the student use the menu to underline/underscore? How many times did the student underline/underscore using the right-click context menu? How many times did the student use the mouse to click the spell check button? How many times did the student use the menu to check spelling? How many times did the student check spelling using the right-click context menu? How many times did the student right-mouse click misspelled words? How many times did the spell check identify misspellings?


13


Scrolled Help Zoom Highlight Txt2Spch ShowTimer Hide Timer0%

10%20%30%40%50%60%70%80%90%

100%

70%

15%

33%

47%

72%

53%56%

60%

7%

16%

32%

53%

34% 36%

WCBA 2011System Tools

Grade 8 (n=24,281)

Grade 12 (n=28,238)

Perc

ent o

f Stu

dent

s

Available: WCBA 2010



14


What Proportion Told Us

All tools were triggered at least once by a higher percentage of 8th graders than 12th graders

Text-to-Speech and scrolling were used by a greater percentage of students than other system tools

Help was used by a smaller proportion of students than other system tools

What Proportion Doesn’t Tell Us

Which students used multiple tools

If tools were used in some sections more than others

When and for how long each tool was used

The impact of tool use on performance– Exploration vs. need



15









Easy

Difficult


16


Grade 8Blue–72%High Contrast–21% Gray–7%

Observing End StateAvailable: WCBA 2010


17


Observing End State

What End State Told Us

Blue was the end state skin for 72% of participants

Grey was the least used (7%)

What End State Doesn’t Tell Us

When they changed to the final state (beginning? end?)

How long they spent in the end state

Why they changed

How many blue users surveyed the other skins first (i.e., kept the default vs. choosing it)


18









Easy

Difficult


19


Available: MCBA 2011Booklet #6037

Observing Time Course

10 min 13 min 16 min 19 min 22 min 25 min 28 min 31 min0

5

10

15

20

25

30

35

Timeline - Item 71

Corr

ecte

d M

issp

ellin

g Ev

ent #


20


What Time Course Told Us

When Corrected Misspelling events occurred

How long between Corrected Misspelling events

What Time Course Doesn’t Tell Us

Which words were misspelled

Number of words that were not identified as misspelled

If spellcheck was used to good effect

What was happening in between spellcheck events

Observing Time Course


21









Easy

Difficult


22


Duration(2 time stamps, math)

Observing Duration/Intervals

VE521278c16 VE521279n160

3

6

9

12

15

1817.5

15.16

MCBS 2011Average Time Spent on Task Directions

Seco

nds


23




Students Senior Usability Expert

05

101520253035404550

17.5

47

MCBS 2011Average Time Spent on These Di-

rections

Seco

nds

YOU ARE PERMITTED TO USE A CALCULATOR IN THIS SECTION.

This section has 16 questions. Respond to questions on the computer. You will have to select an answer or type your response as directed. In those questions where you type an answer, it is important that your answer be clear and complete since partial credit may be awarded.

Remember:

• You are permitted to use a calculator in this section; either your own, or the one provided by NAEP. If you need help using the one provided by NAEP, read the instructions on the back of the scratch paper or look at the quick reference card on the calculator lid.

• If the calculator does not work or you do not know how to use it, raise your hand and ask for help.

• You will have to decide when to use the calculator. For some questions using the calculator is helpful, and may even be necessary. However, for other questions the calculator may not be helpful.

• This section is 25 minutes. If you finish before time is up, you may review any question within this section. You will not be able to go to the next section until 25 minutes are up.

• You may use the scratch paper provided, but be sure to enter all of your responses on the computer.

Click NEXT to continue.


24




Students Senior Usability Expert

05

101520253035404550

15.16

27

MCBS 2011Average Time Spent on These Di-

rections

Seco

nds

YOU ARE NOT PERMITTED TO USE A CALCULATOR IN THIS SECTION.


Remember:

• You are NOT permitted to use a calculator in this section.





25


What Duration Tells Us

Students spent from 15-20 seconds on the directions for different blocks

Students on average spent more time on the directions for VE521278c16

Senior Usability Experts are really slow readers!

What Duration Doesn’t Tell Us

What students were doing while on that page

How much of the directions they read

How well they understood the directions



26









Easy

Difficult


27


Observing Qualitative Information

No Correction 7%

Spelling 73%

Spacing 20%

Spelling Examples (includes punctuation)– “befor” vs. “before”– “ninjas” vs. “ninja’s”

Spacing Examples– “monkey.You” vs. “monkey. You”– “books,none” vs. “books, none”

No Correction Examples– “stonger,faster,and” (no alternate spelling

suggested)– “orrhjrejfjhhrhgjshjdnewjmfnffjnendndsms

andbfnnwnnfgbnemsnfcnfnsehgfnbsjehdnbvfndbfjxdnnebndfnfbfbfbbdbdnbsnsnnnsnsnsnnssnsnnsnnsnssssssssdjejkwjennbrbbfbbfbfcbfcbcvbbcxbcxblolooooooooooooooooooooooobbxbbx”

Identified Error Types


28


What Qualitative Told Us

What was modified

How it was modified

When it was modified

What It Didn’t Tell Us

The impact of the modification on the final output

Re-creation of the final product

Observing Qualitative Information


29


Outline






30


Major Takeaways

Students used accommodations and system and WP tools substantially during CBAs.– Scrolling: 60%-70%– Text-to-Speech: 53%-72%

Feature or Accommodation

4th Grade

8th Grade

12th Grade

Spanish 55% 56%

Text-to-Speech 82% 72% 53%

Show Timer 72% 53% 56%

Hide Timer 78% 34% 36%

Zoom 64% 33% 16%

Scrolled 63% 70% 60%

Help 22% 15% 7%


31


Major Takeaways

Some accommodations and tools were not used as might be expected– Accommodation – Directions written in Spanish– Used by - 55%-56% of eligible students

Used56%

Did Not Use44%

Grade 12

Used55%

Did Not Use45%

Grade 8


32


Major Takeaways

Observable data helped to identify and validate usability issues with CBAs.

Misuse of Next button– To open writing panel– To check on time remaining

Pausing tutorials (only 8%-9% of pausers didn’t finish)

Scrolling was not used by 7%-14% of participants on items where it was necessary in order to show all the information.


33


Major Takeaways Insight into student response patterns

– Number of words used for different writing purposes


34



– Types of errors identified by spell check

No Correction 7%

Spelling 73%

Spacing 20%


35



– Distribution of spellcheck corrections by student


36


Major Takeaways

MCBA Participant Average

Senior Usability Expert

100 200 300 400 500 600 700 800

Short 515

Short 289

Long 761

Long 283

National 204

National 280

Words per Minute

Insight into student response patterns– Compare silent reading rates


37


With Calculator Without Calculator YOU ARE NOT PERMITTED TO USE A CALCULATOR IN

THIS SECTION. This section has 16 questions. Respond to questions on

the computer. You will have to select an answer or type your response as directed. In those questions where you type an answer, it is important that your answer be clear and complete since partial credit may be awarded.

Remember:• You are NOT permitted to use a calculator in this section.• This section is 25 minutes. If you finish before time is up,

you may review any question within this section. You will not be able to go to the next section until 25 minutes are up.



Major Takeaways

YOU ARE PERMITTED TO USE A CALCULATOR IN THIS SECTION.


Remember:• You are permitted to use a calculator in this section; either your

own, or the one provided by NAEP. If you need help using the one provided by NAEP, read the instructions on the back of the scratch paper or look at the quick reference card on the calculator lid.

• If the calculator does not work or you do not know how to use it, raise your hand and ask for help.

• You will have to decide when to use the calculator. For some questions using the calculator is helpful, and may even be necessary. However, for other questions the calculator may not be helpful.





38


Written Instructions?

Written instructions are not being read for comprehension

Possible Causes / Consequences– The students suffered for not having read the

instructions– The students didn’t need the instructions– Some students didn’t need the instructions and

skewed the average duration data


39


What Do We Know?

How students use system tools

How students use word processing tools

What keys they are pressing and when

What buttons they are clicking and when

What words are identified by spell check, which are corrected and how they are corrected

How much time students spend on each screen


40


Outline






41


Descriptive vs. Experimental

Observable Data are by definition descriptive

Don’t answer causation or other complex questions

Causation needs experimental data– Isolate variables– Manipulate conditions– Compare groups

Complex questions– Pair Observable Data with other information


42


The More You Know, The Less You Know

Simple Question: Can Students Access the Features of CBA?– Simple Answer: Yes

Complex Questions– Did the changes made to interface have any impact on their

performance or comfort during assessment?

– Did everyone who would have benefited use them?

– Do kids even know what benefits them?

– What is impact of age/grade on beneficial impact of interface changes?


43


More Questions

Why do some students not use system tools?

Why do some students not use word processing tools?

Which tools result in better performance?

Are there other tools that might prove beneficial?

Do written instructions help? Everyone?


44


Better Questions = Better Answers

1. Start with questions to be answered

2. Determine sources of information necessary to answer question (research design)

– Obs. Data (OD) + Usability testing

– OD + Background Question Answers

– OD + Performance Scores

– OD + Qualitative Analyses

3. Interpret interactions among data


45


Data Sources

Observable Data (behavior)– Easy to collect (now)– Every participant (not just a sample)– Many, many parameters (many, many)– Single source

Profile Data (outcomes, background)– Also easy to collect on every participant– Fewer parameters– Multiple sources

Qualitative Data (impact, usability)– Requires interpretation– Difficult to collect– Requires sampling (research design)


46


Data: The Final Frontier

Level I – CountingSingle variable

Frequency /Proportion

Level II – StructureExtended information

Time course / Duration

Level III – MeaningMultiple variables

Interactions among variablesQualitative information

ValueValue


47


Fulcrum IT Services, LLC

National Assessment of Educational Progress (NAEP) Observable Data From Computer-Based Assessments

Gregory Anderson, Ph.D.Senior Usability Expert

Fulcrum IT, LLC

26 th annual management information systems (mis ) conference february 14, 2013

Documents

services company

llc fulcrum

gateway court manassas

trinity parkway

cut10 keyboard copy11

copy12 keyboard paste13

mode on73

mode off74