Card Sort TestCogQuiz Neuropsychological

Assessment Tests

Figure 1

CONTENTS:

1) CogQuiz Card Sort Test…………………………...………...…. .3

2) A Brief History of Sorting Tasks…….………………………..…. 3

1

3) Registering Your New Test………………………………………8

4) Executing a Card Sort Test…………………..………………….5

5) Analysis of Results………………………………….…………..12

6) Creating and Editing a Card Sort Test…..…….………………20

7) Normative Data………………………………………………….29

8) References………………………………………………….……36

2

Card Sort TestCogQuiz Neuropsychological Assessment Tests card sorting task provides considerable

flexibility in task definition. The number of stimulus cards can range from two to five and

the number of sort dimensions, from two to five. One of the categories can be user

defined, allowing for the definition of classes of stimuli. The card definition tool allows test

authors to create their own stimulus and response card definitions. Cards can display

alpha/numeric characters, any image that can be realized as a “dingbat” or words.

Response card decks can be of any size and their presentation can be controlled either

by the participant or the computer. Category (stage) shifts can be either signaled or

unsignaled. For example, in the Nelson (1976) Modified Card Sorting Test after

successfully completing a criterion run the participant is signaled that the matching

criterion will shift. All test messages and instructions can be tailored to meet the needs of

the test author. Participant response modalities include “point and click” and touch.

Tests can be analyzed using any one of three analysis protocols or the test user can print

a basic analysis sheet displaying basic participant performance for “hand” analysis. The

package comes with normative data collected on over 3000 participants grouped in age

cohorts from five to nine years, 10 to 14 years, 15 to 19 years and in 10 year groups from

the twenties through the eighties on the Card Sort Test. As Tien, et al., 1996, Feldstein, et

al., 1999, and Steinmetz, 2010 have all shown significant outcome differences between

manually administered card sorting tests and tests administered using a computer. This

normative data set on one of the six card sorting tests that comes with this highly flexible

set of computerized card sort tests represents a considerable step forward in computer

based card sorting tasks.

A Brief History of Sorting Tasks

The use of sorting tasks has a long history in the assessment of cognitive performance.

At the beginning of the last century the development of concepts by children was thought

to be governed by an “associative” process similar to ones described by Hume, Locke,

John Stuart Mill and later Bain and Wundt. Narziss Ach, however, was convinced that

mere association was not sufficient to account for the complexity of acquired concepts or

3

the interrelationships among those concepts (e.g., abstract properties, “hierarchal”

organization, etc.). He was also uneasy with the methodological

technique of “scientific introspection” (Wundt, 1873) then favored by “experimental

psychologists.” As part of his research program he developed a sorting task in which

subjects were asked to assign objects that varied in shape, size, color and weight to a

series of nonsense syllables that were assigned arbitrary meanings (e.g. “all pyramids,”

“all large, red, heavy balls”). Cards with these arbitrary syllables printed on them would be

placed at various locations on a table along with a collection of objects. The Subjects first

memorized the nonsense syllables and were then instructed to place the objects on the

card they thought named the characteristics of the object. They were given feedback

indicating whether they had correctly assigned an object to a card. The procedure

allowed Ach to observe directly the subject’s development of competency in making

assignments rather than depending on their “introspective” description of that

competency.

The use of sorting tasks in neuropsychology can be traced to the work of Gelb and

Goldstein (1920) in their work on color amnesia in head injured veterans of the First

World War. He used as his sorting task the Holmgren Test for Color Blindness

developed by Fithiof Holmgren (1831-1897) in 1876 and subsequently modified by

William Thomas. The test consisted of three differently colored, labeled standard stimuli

(small skeins of yarn), and forty labeled skeins of varying color. The Subject was first

asked to select ten skeins from the collection of forty that most closely matched the first

standard, next to select five skeins from the remaining thirty that most closely matched

the second standard and finally select five skeins from the remaining twenty-five that

most closely matched the third standard. Although this task was designed to detect color

blindness, it also proved useful in distinguishing neurological and psychiatric patients

(who were not color blind) from normal individuals. Normal individuals could select colors

that approximated the standard while impaired individuals could not (e.g. could not

classify a light green or a dark green as green when the standard was green). This

series of observations would lead Goldstein to formulate his distinction between

concrete and abstract attitudes; Unimpaired individuals can abstract the relevant

characteristics of the match, while impaired individuals continue to focus on its concrete

4

characteristics, precluding the proper selection of skeins that only approximately

matched the standard. Gelb and Goldstein continued to develop variations on this basic

color sorting test in their work with neurological patients.

While resident at the Institut zur Erforschung der Folgeerscheinungen von

Hirnverletzungen, an institute for the study of brain-injured soldiers, found in 1916, by

Goldstein in collaboration with Gelb, Egon Weigl (1901-1979) extended their work to

include the ability to shift classification categories. His work was first published in 1927 in

the German publication Zeitschrift Psychologie under the title “Zur Psychologie

sogenannter Abstraktionprozesse”, but because of its importance it was translated into

English and republished in 1942 (Weigl, 1942). Weigl was interested in the ability of

neurological patients to “shift” their sorting perspective from one dimension to another.

L.R., a patient with frontal lobe injury, exhibited the same inability to sort color samples

based on a standard color as had been seen earlier by Gleb and Goldstein in

neurological patients. In addition, L.R. was unable to “shift” between two possible sorting

dimensions on a relatively simple sorting task. Weigl presented L.R. with 12 cardboard

shapes (four equilateral triangles, four squares and four ovals). Each member of a shape

type was colored red, green, yellow or blue. The 12 colored shapes were randomly laid

out before R. and he was asked to arrange them into groupings that made sense to him.

L.R’s. solution was first to create a column with the red triangle at the top, the red oval

immediately below the triangle and the red square below the oval. L.R. then proceeded

to replicate this arrangement of shapes for each color, yellow then green and finally blue

in columns positioned immediately adjacent to the first column. When asked if he could

think of another way of sorting the colored shapes, he after some consideration swapped

the row containing the oval with the row containing the squares. He was not able to shift

to “shape” as the sorting category, although he was able to sort correctly on shape when

the 12 objects were turned over revealing a homogenous white surface. Normal subjects

tested by Weigl showed no difficulty in shifting between color and shape as the relevant

sorting dimension and with one exception simply piled the shapes together in the

relevant category. The one exception when sorting to color, organized the four groupings

in much the same way as L.R. Subsequent variations on this study and other studies

5

using three dimensional everyday objects produced similar outcomes which are

fascinating (particularly the ones with children who seem to classify on the basis of an

invented narrative rather than categories), but are beyond the scope of this discussion.

Harry Harlow’s (1905-1981) work with primates on “learning how to learn efficiently in the

situations the animal frequently encounters,” or “learning sets” in part set the context in

which Esta Berg developed a card sorting test. Harlow’s interest in learning sets grew out

of his concern that the study of learning as it was then pursued by “theoretical

psychologists” was conducted in such rigorously controlled conditions that obtained

experimental results had little or no utility in a world outside the laboratory. In a learning

set paradigm the subject is present with two (sometimes more) stimulus objects. Under

one of the objects is to be found a reinforcing stimulus (in the case of infra-human

primates a morsel of tasty food and in the case of humans either food or more likely

some desirable object). The stimulus objects can be arranged in several ways but the

reinforcer in the basic paradigm is always to be found under the same object. The

dependent variable of concern is the number of trials required to identify and reliably

choose the object concealing the reinforcer, and the change in the value of that variable

over a number of sequentially present problems (a problem is defined as a stimulus set).

Typically subjects show a decrease in the number of trials required to reliably identify the

object under which the reinforcer is hidden. A variation on this basic paradigm is to

sometimes consistently conceal the reinforcer under one of the objects in the object pair

(object–quality discrimination) and at other times conceal the reinforcer under the object

in a particular location (either left-position discrimination or right-position discrimination).

Zable and Harlow (1945) showed that monkeys (rhesus) could reliably learn both object-

quality and position discriminations and also learn to reliably shift between discrimination

types (object or positions) when the problem set changed. Settlage, Zable, & Harlow

(1947) extended those findings to rhesus monkeys following “bilateral removal of the

frontal areas” showing them to be capable but less efficient than normal animals on such

a shift task.

Harlow eager to extend his findings into the human realm approached his colleague,

David Grant. Together they designed the framework for extending the work of Harlow to

human subjects and then turned the project over to Esta Berg as the topic for her

6

master’s thesis (Eling, Derckx & Maes, 2008). The design of that study resulted in the

development of a 60 card deck (response cards), where each card contained from one

to four identical figures (stars, crosses, triangles, and circles) all identically colored in

one of four colors (red, yellow, blue, and green). Each card was unique in its

configuration of figure, figure color, and figure count. Four cards (one red triangle, two

green stars, three yellow crosses and four blue circles) were duplicated to serve as

stimulus cards. The stimulus cards were laid out in row such that from the subjects left to

right the one red triangle appeared to the far left of the row, the two green stars

immediately to its right, the three yellow crosses to the right of that, and the four blue

circles on the far right end of the row. The subjects were told to place “these cards [from

the response card deck] into four groups, underneath the ones lying on the table. I will

tell you whether you are ‘right’ or ‘wrong’.” The initial “correct” category was arbitrarily

chosen from among the three dimensions, figure, color, and count on which each card

varied. After sorting correctly five successive cards the “correct” category was shift to

one of the two remaining categories and then following five successive correct sorts to

that category the “correct” category was shifted to the remaining category. A new

category was then arbitrarily chosen and the procedure just outlined was followed again

and then followed for third time for a total of nine successive sorting categories. If the

subjected exhausted the response deck, the deck was shuffled and given back to the

subject for a second time.

Grant and Berg (1948) modified this procedure, extending the size of the response deck

to 64, allowing all combinations of figure, color, and count to appear exactly once. The

number of categories used was reduced from nine to six. Both the “arbitrary” order of

category selection and the randomized (“shuffled”) order of the response deck were

preserved. While Berg had recorded only response errors for each category, this study

introduced the concept of a “perseverative” response and divided errors into

“perseverative” and “non-perseverative”. The card sorting test also got its name in this

article: University of Wisconsin Card-Sorting Test.

7

The use of card sorting tasks similar to the Wisconsin Card-Sorting Test (WCST) did not

find their way into neuropsychological testing until Brenda Milner (1963) citing deficits of

“Weigl” style tests (“too easy a task for young adult subjects” and the

“disadvantage of yielding only dichotomous data”), “decided, instead, to use the

Wisconsin Card Sorting Test of Grant and Berg, which is based partly on the Weigl test

and partly on experimental techniques used successfully with lower primates [Harlow].”

Milner standardized the order of category presentation to color, figure, count and

enlarged the definition of a “perseverative” error to include responses in the first

category “as a continued response in terms of the patient’s initial preference.”

She, in a pre-, post-operative design, tested 71 patients using the WCST who underwent

unilateral cortical excisions to relieve epileptic seizures. In addition, she also tested 23

patients post surgically only. She found a marked increase in the errors committed and a

decrease in categories achieved when post-surgical test results were compared with

pre-surgical test results for dorsolateral frontal-lobe excisions, but no similar decrements

for patients whose excisions were more posterior. For the 23 patients tested post-

surgically only, the performance of patients with dorsolateral frontal-lobe excisions was

markedly inferior to patients with more posterior excisions.

In a recent review Nyhus and Barceló (2009) identified the WCST as “one of the most

distinctive tests of prefrontal function.” They go on to say, however, that “clinical

research and recent brain imaging have brought into question the validity and

specificity of this test as a marker of frontal dysfunction.”

Registering Your New Test

When first loaded, whether from an online download or a physical storage device,

execution of the installed program will prompt for the registration of the newly installed

CogQuiz software. This prompt will continue to appear each time the program is run until

the registration procedure is completed.

8

If the computer on which the software has been installed is connected to the Internet, the

software can be automatically registered by providing the information requested (i.e., the

name of the purchaser and the purchaser’s email address) and clicking the Register button.A Product Key Code will be sent to the email address provided within two working

days. After receipt of the Product Key Code, enter it in the provided fields and click the

Enter button to complete the registration procedure. (Note: Each time a program is

installed and executed for the first time it is issued a new Serial Number so each of the

two allowed installations will have its own unique combination of Serial Number and

Product Key Code.) Be sure to note both the Serial Number and Product

Registration Key as they may be required in subsequent interactions with CogQuiz

Neuropsychological Assessment Tests. Once registration is completed the Product

Registration screen will not appear again and the nags in the corners of the screen will

disappear. The software is fully functioning with nags prior to Registration and no data

will be lost following Registration.

Executing a Card Sort Test

On the WELCOME screen (see Figure 1), select a test from the pull down “Select a

Test” list. This will produce the SELECT PARTICIPANT window (Figure 2). Select a

participant from the “Select a Participant” pull down list. This assumes that the

participant has already been added to the Participant Database; if the participant has not

yet been added, then before proceeding you will need to return to the CogQuiz Menu

screen (by clicking the Cancel button) and add the participant to the database (see the

section on adding new participants). Once a participant has been selected there is an

option to then enter the tester’s name; if entered, the name of the person administering

the test will be added to the participant’s test record. After selecting the participant and,

optionally, adding the tester’s name, click the Run Test button.

9

Figure 2

The first display a participant will see is a set of instructions for the test. Figure 3 shows

a typical instruction display. Once the instructions have been read and understood the

test is initiated by clicking or touching the OK button. A display similar to one of the two

tests shown in Figure 4a and 4b will be displayed. The exact configuration of the display

depends on the definition of the selected test. The tests shown in Figure 4a and 4b are

for adults and children respectively.

10

Figure 3

Depending on whether the production of a Response Card is specified as ”Dealt” or

“Paced” (See Creating and Editing a Card Sort Test for an explanation of the

difference), the participant first clicks the single card at the bottom of the screen to

display a new Response Card (Dealt) and then clicks on the card immediately below the

chosen Stimulus Card displayed at the top of the screen. If “Dealt” is the specified option

the computer will automatically display a new Response Card at the appropriate time.

The participant then clicks the card under the chosen Stimulus Card. Following a

response the participant will be provided with either visual, auditory or both feedback on

the correctness of their response and the next trial will begin. At the end of the test the

participant will be thanked for their participation and control of the test will be returned to

the WELCOME screen.

Figure 4a and 4b

As with all CogQuiz tests, the Card Sort test can be terminated at any time by pressing

Ctrl x. On initiating this sequence the user will be asked to confirm the choice to

terminate the test; if the choice to terminate is affirmed, the test will be terminated and

no data from the test will be saved.

11

Analyzing a Card Sort Test

The Analysis Screen shown below has two components. On the left side is displayed

relevant participant information and the summary data for the test selected for analysis.

The right side provides for selecting a participant, the test to analyzed and the “event

record” of the test selected for analysis.

Figure 5

Analyze a test by first selecting a participant from the Get Participant pull-down list.

Selecting a participant name from the list (by clicking on the name) will populate the Card Sort Tests Taken with all of the Card Sort Tests taken by the selected participant. Select

a test for analysis by clicking on it. Doing this will result in the display of the event record

(more about the event record later) for the selected test and the appearance of a small

blue box at the bottom of the screen. Select an analysis protocol from the Select Analysis Protocol pull down list. There are four including Heaton et.al. (1993), Grant

and Berg (1948), Lezak (1995) and Basic Score Sheet only. Each of these protocols

define the 13 summary measures in different ways in some cases yielding different

values on a particular test in the summary data table.

Category: The ordinal value of the current category (stage).

Resp. Count: The total number of responses made in this category.

Avg. Resp. Time: The average latency of responses made in this category in

milliseconds (latency is calculated as the time from the initial display of the

response card until the selection of a stimulus card).

Correct Resp. Count: For the Lezack and Heaton protocols this is simply the

total number of correct responses made for this category. For the Grant protocol

it is the number of correct responses in excess of the correct responses making

up the final run in this category.

Avg. Correct Resp. Time: The average latency (as defined above) of correct

responses (as defined in “Correct Resp. Count”) in this category.

Error Resp. Count: For the Lezack and Heaton protocols this is simply the total

number of erroneous responses made for this category. For the Grant protocol it

is the number of erroneous responses made in this category excluding the first

error made following a category shift.

Avg. Error Resp. Time: The average latency (as defined above) of error

responses (as defined in “Error Resp. Count”) in this category.

Persev. Resp. Count: Grant & Berg define a perseverative response as “a

response to a new or shifted category which would have been correct for

the immediately preceding category” (Grant, et. al., 1948). For Heaton it is

defined as a response appropriate to the “perseverate-to-principle” (See

the Heaton scoring rules outlined below).

Avg. Persev. Resp. Time: The average latency of a perseverative response in

this category.

Persev. Error Count: Grant & Berg define a perseverative error response as a

response that would have been correct to the previous category. For Heaton it is

defined as an error response appropriate to the “perseverate-to-principle” (See

the Heaton scoring rules outlined below).

Avg. Persev. Error Resp. Time: The average latency of a perseverative error in

this category.

Unique Error Resp. Count: The number of responses where the stimulus card

and the response card did not match on any dimension.

Avg. Unique Error Resp. Time: The average latency of a Unique Error

Response.

Failure to Maintain Set: A count of the number of times a participant made a run

of consecutive correct responses greater than or equal to half the number of

consecutive correct responses required to initiate a shift to a new category, but

less than the number to initiate a shift to a new category.

Basic Score Sheet: shown below is the output from selecting the “Basic Score Sheet

Only” option. No summary analysis is done, providing only a listing of the responses

made. This option has been provided in the event the test administrator wishes to score

the test manually or using a scoring protocol not captured by the three provided ones.

The score sheet can be either saved to a file or printed.

14

Figure 6

Basic test identification information is provided -- including the test version taken, the

internal identification number of the participant, the date and time the test was taken, the

date and time the score sheet was generated and the name of the person administering

the test.

15

The Category in effect is shown in capital letters and underlined. So for example, here the

first category of the displayed score sheet is SYMBOL COLOR. The characteristics of the

Participant’s responses are displayed in from seven to nine columns (depending on

selected parameters of the analyzed test). The first column is simply the response

number. The second column contains either - - indicating an error response or a number

indicating a correct response. The next several columns (between 2 and 4) display the

match dimensions used in the analyzed test. So in the example shown above three

dimensions Color C, Form F, and Number N were used. If Card Color had also been used

a fourth column B (for Back color) would also appear. The next column following the

match dimension columns contains an O for Other. If the current response card (a card

from the deck) matches the selected stimulus card on a particular dimension then that

dimension is bracketed. So, for example, the first card from the response card deck

matched the selected stimulus card on Form [F] and number [N] but not on color C. On

the second trial the selected stimulus card did not match the response card on any of the

match dimensions and has been scored as an Other [O]. It is not until trial five that the

stimulus card and response card match for Color [C]; it also, incidentally, matched on

Form [F]. The number, 1 is recorded indicating the first correct response in what might

potentially be a run of correct responses. The next column is simply a blank space

allowing for the recording of the response as “perseverative” according the criteria in use

by the investigator. The last column is the time in milliseconds between the presentation of

the response card and the selection of the stimulus card.

Lezak: The simplest protocol, described by Lezak (1995), counts total, perseverative and

unique (Other) errors. The total error count is simply the total number of errors made on

the test. A perseverative error is defined as an error response which would have been

correct for the previous category. A unique error is a response error where the response

card does not match the stimulus card on any match dimension (for example one green

triangle and three red stars).

The Lezak Score Sheet is similar to the Basic Score Sheet with two differences (see the

Heaton Score Sheet). The column displaying response latencies is now the third column

in the display and the column displaying blanks has been replaced with a column

16

displaying either a blank or a “P” for Perseverative, in the case of Lezak a perseverative

error.

Grant and Berg: The “Grant and Berg” protocol scores “Correct Responses” as the

number of correct responses made minus the number of consecutive correct responses

required to complete the current category. The column “Correct Resp. Count” in summary

data table is number of correct responses in excess of the number of consecutive correct

responses required to initiate a category shift. This protocol also does not count the first

error following a category shift as an error.

Heaton: The “Heaton” scoring protocol is based on TERMINOLOGY and SCORING

RULES as described by Flashman, et al. (1991). “A perseverative response is one that

would have been correct in the immediately preceding stage.” It is a response that

matches the Response Card with a Stimulus Card on the “perseverate-to principle,” that

is the stimulus dimension thought to be the relevant one in determining a correct match.

The “perseverate-to principle” is usually the stimulus dimension that was correct in the

immediately preceding stage, but not always as it can be modified by Rule #5, #6, and #7

(See below). “An unambiguous response is one that matches the stimulus cards on only

one criterion.” The following scoring rules coupled with the definitions given above were

used to define the “Heaton” scoring algorithm.

1. Any unambiguous response that matches the “perseverate-to principle” is considered a perseverative response, even if there are intervening responses that do not match this principle.

2. An ambiguous response may also be scored as perseverative, according to the“sandwich rule” (our terminology). If the ambiguous response matches the “perseverated-to principle,” and the closest unambiguous responses both preceding and following it are perseverative, it is scored as a perseverative response.

3. A qualification to Rule #2 may apply if there is more than one ambiguous response “sandwiched” between the two unambiguous ones. All of the “sandwiched” ambiguous responses must match the “perseverate-to principle” in order for ANY of them to be scored as perseverative.

4. The incorrect response immediately following completion of a stage [category] (i.e., following 10 [Correct run Length] consecutive correct responses) is scored as a perseverative response, but only if it is unambiguous.

17

5. Once the first incorrect unambiguous response has been made, that sorting principle becomes the “perseverate-to principle” in the first stage [category].

6. If the subject makes three consecutive, incorrect, identical, unambiguous responses ..., this new sorting principle becomes the new “perseverate-to principle.” The second unambiguous response in the sequence is the first one to be scored as perseverative.

7. A qualification to Rule #6 may apply if there are ambiguous responses “sandwiched” between the three consecutive, incorrect, identical, unambiguous responses. All of these “sandwiched” ambiguous responses must match the same category as the unambiguous ones in order for the “perseverate-to principle” to change. As in Rule #6, the second unambiguous response in the sequence is the first to be scored as perseverative.

8. Responses that are scored as Other [O] may not be scored as perseverative responses.

The “Heaton” Analysis Sheet adds a final column not included in the other scoring sheets

giving the Flashman rule used to determine the perseverative status of a response.

18

Figure 7

Event Recorder: All CogQuiz Neuropsychological Assessment Tests keep a “running”

event record of all salient events occurring during a test’s administration. This event

record, displayed in the upper right corner of the Analysis display, has four columns. The

first column displays the ordinal position of the event; the second column, the event

name, reflecting the nature of the event; the third column contains argument(s) describing

parameters of the event; and the fourth column contains a “timestamp” indicating in

milliseconds the time that has elapsed since the recorder was started (the start of the

test) when the computer detected the occurrence of the event.

19

The event recorder for Card Sort captures eight separate events. The list of events and their arguments are shown in Table 1.

Event Name Event (Description) Argument(s) Argument(s)

Description

Start Test Start of the test. Date/Time The date and time thetest started

Category The category currently in effect

Category, Run length

The category in effect name & the number ofconsecutive correct responses required for a category shift.

Display CardDisplay the current response card.

Icon, Icon Color, Count, Card Color

The displayed value of Icon is the equivalent alpha/numeric character of the displayed icon

Response+A correct response

Card, Icon, IconColor, Count,Card Color

Card is the 0 relative stimulus card selected. Icon is the equivalent alpha/numeric character of the displayed icon

Response- An incorrect response “ “

CompleteSignals the completion of a category.

Category, Run length

The category in effect name & the number ofconsecutive correct responses required for a category shift.

TerminateSignals thetermination of thetest

Reason for termination

Test Complete (all categories achieved) Deck Exhausted (all response cards used)

End Test Signals test completion Date/Time The date and time the

test endedTable 1

Creating and Editing a Card Sort Test

Create a new or edit an existing Card Sort test by first clicking the Create/Edit Test button on the Welcome screen. Doing so will produce a test definition/edit screen similar

20

to one seen in Figure 8. The initial screen, unlike the one seen below will have only a few

fields defined with default values, which can, at the test author’s discretion be changed.

Create a New Test:

Begin creating a new test by clicking the Create New Test button. This will position the

cursor in the test name field located in the upper left corner of the screen. Provide a

unique name for the test and in the Test Author Name field the name of the test’s

author. Save the test by clicking the Save Test button.

Figure 8

Begin defining basic parameters for the test.

Response Mode: Select a mode of responding from the pull down list. Mouse is a

standard “point and click.” If the test is to be administered on a system (e.g. PC or

tablet) which utilizes a “touch screen then Touch can be selected as an option (Note:

The normative data supplied with this test was collected with a mouse, although we

(Feldstein et. al., 1999) have found little difference between the two options.)

21

Stimulus Count: Select the number of stimulus cards (two to five cards) to be used for

this test. The stimulus card display at the top of the screen will change to reflect the

number of cards chosen.

Presentation Mode: Response cards can be presented either by having the Participant

click on the response card deck to present the next card (Dealt) or the computer can

automatically present the next card (Paced) following the Participant’s response on the

previous card.

Pace Delay: Pace delay sets the delay between a Participant response and the

presentation of the next Response Card, if Paced has been selected as the Presentation

Mode.

Match Dimension: Response Cards can match a Stimulus Cards on any one of up to

five dimensions, symbol, symbol Count, symbol color, card color and category. Select the

dimension determining a match for this “stage” from the Match Dimension pulldown list.

Correct Run Length: Select from the pull-down list the number of continuous correct

responses to be used in determining when a stage shift is to be made. (Note: The

“Failure to Maintain Set” value will be set to either half of the selected value, if the

selected Correct Run Length is an even number, or the nearest whole number less than

half of the value if it is odd.)

If the shift from the current stage to the next stage is to be signaled, check the Signal Stage Shift box and enter the required shift message in the text box located directly

above it.

Once the Match Dimension and the Correct Run Length have been selected they can be

added to the list of stages by clicking the Add button, inserted immediately prior to the

currently displayed stage by clicking the Insert button, or they can replace the currently

displayed parameters by clicking the Replace button. Clicking the Delete button will

remove the currently displayed stage from the stage list.

Messages and Instructions:

22

Messages: The Card Sort test comes with a default set of Participant messages;

however, these can be changed to meet the test authors requirements (e.g. first

language of the Participant population). Do this by typing the desired message into the

appropriate field. Checking Use Icons replaces the textual messages with the

displayed icons.

Speech: If the computer system used for testing supports “text-to-speech” then

checking the Use Audio box will cause the system to read to the participant all

messages and instructions. If the Use Icons option has been selected the system will

still read the text messages as entered in the several message fields. Windows 7TM is

supplied with one voice, while Windows 8TM supports three. Other voices can be

purchased from third party vendors. In cases in which a system supports multiple

voices, the desired voice must be selected at the system level prior to running the Card

Sort program.

Instructions: The Instruction Editor allows the test author to write test instructions

specific to their needs. Clicking the Instructions button will open a window similar to the

one seen below. This editor is relatively simple, but it does allow the test author to select

the font(s) to be used in the instruction and the font size. It also provides access to a

small collection of colors for use in coloring the text, as well as access to bolding, Italic

and underlining. If more complex text construction is required, it is suggested that an

external editor be used, and the created file be imported into the editor by clicking the

Import From File button. The only restriction on an external editor is, it must be capable

of storing the created file in “Rich Text Format,” (.rtf). Clicking the OK button will commit

the displayed text to the test definition and close the window. Clicking the Cancel button

will abandon all changes to the text and close the window. Clicking the Read button

causes the currently displayed text to be read using the currently selected voice (See the

section of Speech).

23

Figure 9

Cards:

Cards used in generating a sorting test are maintained in a single “library” of card

definitions which is shared by all developed card sort tests. This library comes with a

preconfigured set of cards. However it can be enlarged to contain additional cards or the

existing cards can be edited using the Card Editor.

Card Browser: The card browser is used to assign cards cataloged in the library to

either a particular stimulus card position or a particular location in the response card

deck.

24

Assign a card definition to a Stimulus Card by clicking on the stimulus card to be

defined. This will open the Card Browser Dialog. Navigate using the four navigation

buttons immediately under the card display to the required card and then click the OK button. The specified card definition will be assigned to the selected stimulus card.

Located between the four navigation buttons is a box displaying the “catalogue” number

for the currently displayed card. That number was assigned to that card at the time it was

created. If you know the catalogue number of the required card, you can type that

number into the box, press the <Enter> key and move directly to the required card.

Add: Assign a new card to the Response Card deck by clicking the Add button

below Card Deck. Select a card to be added from the card library and click the

OK button.

Replace: If a card at a particular location in the Response Card deck needs to

be replaced, first navigate to the card position in the deck (the ordinal position of

the currently displayed card is shown in the box labeled Card Deck located

directly below the card deck display). Click the Replace button, opening the CardBrowser. Navigate to the needed card in the card library using the Card Browser and click the OK button. The newly selected card will replace the currently

displayed card in the card deck.

Insert: A new card can also be inserted into the deck by navigating to the card in

the Response Card deck immediately following the location where the new card

is to be insert and clicking the Insert button. Navigate to the card to be inserted

from the card library using the Card Browser and click OK to insert the card.

Delete: Navigate to the card in the Card Deck to be deleted and click the Delete button. The card will be deleted.

25

Figure 10

Card Editor: Open the Card Editor by clicking the Card Editor button.

Selecting Symbols as the Card Type configures the card as nine areas each of which

can contain a single “icon.” Icons can be drawn from any font or “dingbat” currently

installed on the test computer. Select the font or dingbat to be used for the construction of

a card from the Select Font/Dingbat list. The font or dingbat set will be displayed in the

table at the bottom of the tool. Next select a color for the icon from the Select Icon Color list and the card color from the Select Card Color list. Finally, if the card is to be

assigned an arbitrary classification, select a number between one and five from the

Select Category list, otherwise leave it as zero. Select the icon to be placed on the card

by clicking on the icon displayed in the table at the bottom of the editor. The icon will

appear in the area immediately to the right of the Save Card button.

Click the New Card button and begin placing the icon(s) in the required location(s) on the

card. Do this by clicking on each of the nine squares that are to contain the icon.

26

Once the card is setup click the Save Card button to add the new card to the card list.

Continue adding or editing cards and then click the OK button to add the newly defined

cards to the card library.

Alternatively the Card Type can be defined as Word. Defining the card type as word

allows a word of up to approximately ten characters to be entered for display on the card.

Figure 8 shows word cards from our Stroop Sort. Selecting the word option opens a

small text box. Create a word card by clicking the New Card button and then select a

font, a color for the word, a card color and if appropriate a category for the word. Type the

required word in the text box followed by [Enter]. Save the new card by clicking the Save Card button.

If a card needs to be redefined, select the card by first displaying it using the four

“navigation” buttons (˂˂, ˂, ˃, ˃˃) to navigate to the card. Click the Clear Card button to clear the card. Select the replacement card characteristics (i.e. font,

icon color, card color and icon) as outlined in the procedure for creating a new

card. Finally click the Save Card button to save the card in its new configuration.

(Note: all tests using the changed card will use the new card definition.) Finally,

a card can be removed from the card library by navigating to the card and

clicking the Delete Card button.

Warning: Be careful when doing this not to delete a card currently in use by a test as that test will not continue to function properly.

27

Figure 11Normative DataParticipants were recruited from a college student population, relatives of students, or

from a local senior center. Participants under 18 years of age received five dollars per

hour, and participants over 60 received ten dollars per hour. Participants between the

ages of 18 and 59 received extra course credit for their participation. Participants were

high functioning (see Demographic Characteristics of the Normative Sample), reporting

no head injuries, neurological diseases, or psychiatric illnesses that they believe would

affect their performance.

Test data was collected for the Card Sort Tet included in the software using the “Mouse”

option for Participant responses and the “Dealt” option for response card presentation.

28

Demographic Characteristics of the Participant Sample

Shown below are the demographic characteristics of the participant sample. Some

participants did not provide “Years of Education.” The collection of data for the card sort

test was imbedded in a larger collection effort scheduled over several sessions. In some

cases participants completed the card sort test but did not return for subsequent

sessions. Those that completed WAIS or WAIS-R did so either prior to taking the card

sort test or did return for subsequent testing sessions during which the WAIS of WAIS-R

was administered. We have found no significant differences on the card sort test

between the 61.5% of the sample for which we have WAIS or WAIS-R results and the

38.5% for which we do not. Finally, this sample was drawn from a high functioning

population (i.e. college students, their families and friends and a senior citizen population

that had to meet stringent criteria in order to participate).

29

30

N Mean

Std.

Deviation Std. Error

95% Confidence Interval for

Mean

Minimum Maximum

Lower

Bound Upper Bound

Years of

Education

5 to 9 212 2.08 1.405 .096 1.89 2.28 0 6

10 to 14 243 6.59 1.640 .105 6.38 6.80 2 14

15 to 19 601 12.65 1.519 .062 12.53 12.77 3 16

20s 854 14.72 1.125 .039 14.65 14.80 5 18

30s 259 15.01 1.319 .082 14.85 15.17 7 20

40s 224 15.04 1.704 .114 14.82 15.27 1 20

50s 196 15.33 1.983 .142 15.05 15.61 10 20

60s 239 15.06 2.175 .141 14.79 15.34 9 20

70s 287 14.75 2.639 .156 14.45 15.06 3 20

80s 159 14.88 2.417 .192 14.50 15.26 9 21

Total 3274 13.04 4.016 .070 12.90 13.17 0 21

subject age 5 to 9 223 7.29 1.345 .090 7.11 7.46 5 10

10 to 14 256 12.04 1.429 .089 11.86 12.21 10 19

15 to 19 614 18.07 1.441 .058 17.96 18.19 12 36

20s 869 22.80 2.775 .094 22.61 22.98 20 29

30s 267 33.98 3.067 .188 33.61 34.35 30 39

40s 226 44.15 2.783 .185 43.79 44.52 40 49

50s 196 53.81 2.676 .191 53.43 54.18 50 59

60s 244 64.83 2.776 .178 64.48 65.18 60 70

70s 292 74.25 3.000 .176 73.90 74.59 61 79

80s 161 83.11 2.580 .203 82.70 83.51 80 90

Total 3348 34.67 22.926 .396 33.90 35.45 5 90

WASI

Matrix

Reasoning

5 to 9 136 18.92 6.780 .581 17.77 20.07 2 30

10 to 14 147 27.33 3.733 .308 26.72 27.93 16 34

15 to 19 234 29.37 3.520 .230 28.91 29.82 19 48

20s 316 29.19 3.261 .183 28.83 29.55 11 35

30s 88 29.70 3.174 .338 29.03 30.38 12 35

40s 102 28.55 2.934 .290 27.97 29.13 19 34

50s 104 27.75 2.484 .244 27.27 28.23 20 32

60s 146 25.09 5.269 .436 24.23 25.95 7 32

70s 192 23.98 6.718 .485 23.02 24.94 2 69

80s 105 19.60 6.476 .632 18.35 20.85 4 28

Total 1570 26.38 5.865 .148 26.09 26.67 2 69

WCST

categories

completed

score_G/B

5 to 9 223 3.76 1.819 .122 3.52 4.00 0 6

10 to 14 256 4.98 1.479 .092 4.80 5.17 0 6

15 to 19 614 5.54 1.172 .047 5.45 5.64 0 6

20s 869 5.61 1.084 .037 5.53 5.68 0 6

30s 267 5.40 1.396 .085 5.24 5.57 0 6

40s 226 5.48 1.357 .090 5.30 5.66 0 6

50s 196 5.16 1.539 .110 4.94 5.38 0 6

60s 244 4.29 2.075 .133 4.03 4.55 0 6

70s 292 3.96 3.748 .219 3.52 4.39 0 57

80s 161 3.54 2.156 .170 3.20 3.88 0 6

Total 3348 5.03 1.905 .033 4.97 5.10 0 57

WCST total 5 to 9 223 48.36 23.796 1.594 45.22 51.50 4 97

31

Age

Age N Mean Std. Deviation

5 to 9 223 7.29 1.345

10 to 14 256 12.04 1.429

15 to 19 614 18.07 1.41

20s 869 22.80 2.775

30s 267 33.98 3.067

40s 226 44.15 2.783

50s 196 53.81 2.676

60s 244 64.83 2.776

70s 292 74.25 3.000

80s 161 83.11 2.580

Total 3348 34.67 22.926

% Female

Age N Mean Std. Deviation

5 to 9190

55.00 5.000

10 to 14 210 50.00 5.000

15 to 19 498 70.00 5.000

20s 737 74.00 4.000

30s 235 72.00 4.000

40s 203 77.00 4.000

50s 184 77.00 4.000

60s 221 71.00 5.000

70s 256 67.00 5.000

80s 144 62.00 5.000

Total 2878 69.00 5.000

Years of Education

Age N Mean Std. Deviation

5 to 9 182 2.06 1.430

10 to 14 206 6.55 1.547

15 to 19 491 12.66 1.461

20s 725 14.73 1.113

30s 228 14.95 1.291

40s 201 14.98 1.700

50s 184 15.30 1.979

60s 217 15.00 2.206

70s 253 14.68 2.619

80s 143 14.84 2.428

Total 2830 13.06 4.004

WAIS or WAIS-R Full Scale IQAge N Mean Std. Deviation

5 to 9 111 118.95 14.550

10 to 14 129 118.11 13.459

15 to 19 236 112.83 11.708

20s 365 111.73 12.283

30s 135 116.01 11.793

40s 137 116.48 11.055

50s 139 118.69 10.796

60s 179 116.91 13.293

70s 213 121.60 14.102

80s 127 120.26 15.879

Total 1771 116.36 13.252

30

Categories Completed

Age N Mean Std. Deviation Std. Error95% Confidence Interval for Mean Minimum Maximum

Lower BoundUpper Bound

5 to 9 190 3.82 1.777 .129 3.56 4.07 0 6

10 to 14 209 5.06 1.453 .101 4.86 5.26 0 6

15 to 19 499 5.53 1.191 .053 5.42 5.63 0 6

20s 734 5.60 1.091 .040 5.52 5.68 0 6

30s 235 5.41 1.382 .090 5.23 5.59 0 6

40s 203 5.58 1.197 .084 5.41 5.74 0 6

50s 184 5.18 1.525 .112 4.96 5.41 0 6

60s 221 4.41 2.002 .135 4.14 4.67 0 6

70s 256 3.82 2.074 .130 3.57 4.08 0 6

80s 144 3.53 2.171 .181 3.17 3.89 0 6

Total 2875 5.03 1.663 .031 4.97 5.09 0 6Unique Errors

Age N Mean Std. Deviation Std. Error95% Confidence Interval for Mean Minimum Maximum

Lower BoundUpper Bound

5 to 9 190 10.90 12.195 .885 9.15 12.64 0 58

10 to 14 209 6.50 10.285 .711 5.10 7.90 0 59

15 to 19 499 3.76 9.153 .410 2.96 4.57 0 92

20s 734 2.43 7.486 .276 1.89 2.98 0 89

30s 235 3.25 9.036 .592 2.09 4.42 0 61

40s 203 2.24 6.576 .463 1.33 3.15 0 55

50s 184 2.22 5.036 .371 1.48 2.95 0 37

60s 221 3.68 7.934 .534 2.63 4.74 0 55

70s 256 4.29 8.640 .540 3.23 5.36 0 56

80s 144 5.88 11.162 .930 4.04 7.72 0 56

Total 2875 3.99 8.949 .167 3.67 4.32 0 9231

Heaton (1993) Scoring ProtocolTotal Errors (Heaton)

Age N Mean Std. Deviation Std. Error95% Confidence Interval for Mean Minimum Maximum

Lower Bound Upper Bound

5 to 9 190 51.22 22.244 1.614 48.04 54.40 7 96

10 to 14 209 34.14 20.739 1.435 31.32 36.97 0 83

15 to 19 499 24.62 18.696 .837 22.97 26.26 3 102

20s 734 22.26 17.423 .643 20.99 23.52 4 90

30s 235 24.64 21.343 1.392 21.90 27.39 1 97

40s 203 22.66 17.773 1.247 20.20 25.11 3 102

50s 184 29.04 20.362 1.501 26.08 32.01 3 91

60s 221 39.33 23.912 1.608 36.16 42.50 6 96

70s 256 44.93 24.213 1.513 41.95 47.91 3 110

80s 144 49.56 23.638 1.970 45.67 53.46 8 102

Total 2875 30.80 22.663 .423 29.97 31.63 0 110

Total Corrects (Heaton)

Age N Mean Std. Deviation Std. Error95% Confidence Interval for Mean Minimum Maximum

Lower Bound Upper Bound

5 to 9 190 70.31 16.263 1.180 67.98 72.64 32 105

10 to 14 209 75.16 13.450 .930 73.33 77.00 0 106

15 to 19 499 71.66 11.876 .532 70.61 72.70 5 102

20s 734 69.79 12.720 .470 68.87 70.72 4 104

30s 235 66.06 17.712 1.155 63.79 68.34 2 103

40s 203 69.97 12.347 .867 68.26 71.67 6 98

50s 184 71.24 13.187 .972 69.32 73.16 4 98

60s 221 69.52 14.792 .995 67.56 71.48 4 97

70s 256 69.12 13.606 .850 67.44 70.79 18 103

80s 144 68.60 15.377 1.281 66.07 71.14 26 103

Total 2875 70.20 13.871 .259 69.69 70.71 0 10632

Perseverative Errors (Heaton)

Age N Mean Std. Deviation Std. Error 95% Confidence Interval for Mean Minimum Maximum

Lower Bound Upper Bound

5 to 9 190 23.91 14.816 1.075 21.79 26.03 2 91

10 to 14 209 14.97 9.112 .630 13.72 16.21 0 49

15 to 19 499 11.36 8.108 .363 10.65 12.08 0 51

20s 734 11.04 8.486 .313 10.42 11.65 0 52

30s 235 11.80 10.064 .656 10.51 13.10 1 58

40s 203 11.56 9.893 .694 10.19 12.93 0 63

50s 184 14.97 12.788 .943 13.11 16.83 1 118

60s 221 20.55 15.527 1.044 18.49 22.61 2 92

70s 256 23.71 15.052 .941 21.86 25.57 4 86

80s 144 25.65 13.890 1.158 23.36 27.94 4 63

Total 2875 15.17 12.326 .230 14.72 15.62 0 118Non-Perseverative Errors (Heaton)

Age N Mean Std. Deviation Std. Error95% Confidence Interval for Mean Minimum Maximum

Lower Bound Upper Bound

5 to 9 190 27.34 15.912 1.154 25.06 29.62 1 76

10 to 14 209 19.27 14.043 .971 17.36 21.19 0 60

15 to 19 499 13.20 12.490 .559 12.10 14.30 1 76

20s 734 11.22 10.323 .381 10.48 11.97 0 68

30s 235 12.52 13.067 .852 10.84 14.20 0 76

40s 203 11.04 9.591 .673 9.71 12.37 0 75

50s 184 14.84 11.628 .857 13.15 16.53 0 63

60s 221 18.77 13.296 .894 17.01 20.53 0 67

70s 256 21.35 13.836 .865 19.64 23.05 0 85

80s 144 23.90 15.700 1.308 21.31 26.48 3 83

Total 2875 15.66 13.503 .252 15.16 16.15 0 8533

Grant and Berg (1948) Scoring ProtocolExcess Correct (Grant and Berg)

Age N Mean Std. Deviation Std. Error95% Confidence Interval for Mean Minimum Maximum

Lower Bound Upper Bound

5 to 9 190 31.23 13.159 .955 29.34 33.11 2 68

10 to 14 209 22.87 13.813 .955 20.98 24.75 0 61

15 to 19 499 14.86 13.066 .585 13.71 16.00 0 64

20s 734 12.96 12.868 .475 12.03 13.89 0 71

30s 235 13.17 13.549 .884 11.43 14.92 0 61

40s 203 12.72 13.219 .928 10.89 14.55 0 62

50s 184 18.24 16.958 1.250 15.78 20.71 0 65

60s 221 25.30 18.177 1.223 22.89 27.71 0 78

70s 256 30.02 18.371 1.148 27.76 32.28 0 86

80s 144 32.12 17.540 1.462 29.23 35.01 0 75

Total 2875 18.98 16.263 .303 18.39 19.58 0 86

Total Errors (Grant and Berg)

Age N Mean Std. Deviation Std. Error95% Confidence Interval for Mean Minimum Maximum

Lower Bound Upper Bound

5 to 9 190 47.67 23.492 1.704 44.31 51.03 2 96

10 to 14 209 29.84 21.499 1.487 26.91 32.77 0 81

15 to 19 499 19.94 19.442 .870 18.23 21.65 0 102

20s 734 17.59 17.967 .663 16.29 18.89 0 88

30s 235 19.90 21.799 1.422 17.10 22.70 0 97

40s 203 17.75 18.180 1.276 15.23 20.26 0 93

50s 184 24.82 21.190 1.562 21.73 27.90 1 91

60s 221 35.24 25.272 1.700 31.89 38.59 0 96

70s 256 41.20 25.320 1.583 38.09 44.32 0 110

80s 144 46.24 25.103 2.092 42.10 50.37 3 102

Total 2875 26.43 23.605 .440 25.57 27.29 0 11034

Perseverative Errors (Grant and Berg)

Age N Mean Std. Deviation Std. Error95% Confidence Interval for Mean Minimum Maximum

Lower Bound Upper Bound

5 to 9 19022.1

5 15.540 1.127 19.93 24.38 0 86

10 to 14 20914.2

4 11.120 .769 12.73 15.76 0 47

15 to 19 499 9.07 9.587 .429 8.22 9.91 0 54

20s 734 8.41 9.691 .358 7.70 9.11 0 55

30s 235 8.46 10.435 .681 7.12 9.80 0 59

40s 203 8.06 9.912 .696 6.69 9.43 0 60

50s 18411.7

4 11.544 .851 10.06 13.42 0 60

60s 22116.0

8 15.101 1.016 14.08 18.08 0 87

70s 25619.3

3 15.466 .967 17.42 21.23 0 85

80s 14419.9

4 14.292 1.191 17.58 22.29 0 65

Total 287512.1

9 12.692 .237 11.72 12.65 0 87

Non-Perseverative Errors (Grant and Berg)

Age N

Mean

Std. Deviation Std. Error95% Confidence Interval for Mean Minimum Maximum

Lower BoundUpper Bound

5 to 9 19025.29 19.344

1.403 22.52

28.06 0

96

10 to 14 209

15.59 13.655

.945 13.73

17.46 0

77

15 to 19 499

10.85 13.289

.595 9.68

12.02 0

102

20s 7349.19 10.793

.398 8.41

9.98 0

87

30s 23511.43 15.630

1.020 9.42

13.43 0

97

40s 2039.67 11.623

.816 8.06

11.27 0

93

50s 18413.09 13.259

.977 11.16

15.02 0

91

60s 22119.00 19.863

1.336 16.37

21.63 0

96

70s 25622.07 20.356

1.272 19.57

24.58 0

110

80s 14426.32 23.887

1.991 22.38

30.25 1

102

Total2875

14.23 16.364

.305 13.63

14.83 0

110

35

ReferencesBerg, E. A. (1948). A simple objective for measuring flexibility in thinking. Journal of

General Psychology, 39, 15-22.

Eling, P., Derckx, K., & Maes, R. (2008). On the historical and conceptual background of the Wisconsin Card Sorting Test. Brain and Cognition, 67, 247-253.

Flashman, L. A., Horner, M. D., Freides, D. (1991). Note on scoring perseveration on the Wisconsin Card Sorting Test. The Clinical Neurologist, 5, 2, 190-194.

Feldstein, S. N., Keller, F. R., Portman, R. E., Durham, R. L., Klebe, K. J., Davis, H. P. (1999). A comparison of computerized and standard versions of the Wisconsin Card Sorting Test. The Clinical Neurologist, 13, 3, 303-313.

Goldstein, K., & Gelb, A. (1918). Psychologische Analysen hirnpathologischer Falle auf Grund von Untersuchungen Hirnverletzer. Zeitschrift fur die gesamte Neurologie und Psychiatrie, 41, 1-142.

Goldstein, K. & Scheerer, M. (1941). Abstract and concrete behavior: An experimental study with special tests. Psychological Monographs, 53, 1-151.

Grant, D. A., & Berg, E. (1948). A behavioral analysis of degree of reinforcement and ease of shifting to new responses in a Weigl-type card-sorting problem. Journal of Experimental Psychology, 38, 404-411.

Milner, B. (1963). Effects of different brain lesions on card sorting: The role of the frontal lobes. Archives of Neurology, 12, 100-110.

Marotta, J. J., & Behrmann, M. (2004). Patient Schn: has Goldstein and Gelb’s case withstood the test of time? Neurpsychologia, 42, 633-638.

Nelson, H. E. (1976). A modified card sorting test sensitive to frontal lobe defects. Cortex, 12, 313-324.

Nyhus E., & Barceló F. (2009).The Wisconsin Card Sorting Test and the cognitive assessment of prefrontal executive functions: a critical update. Brain and Cognition, 71(3), 437-451.

Settlage, P., Zable, M., & Harlow, H. F. (1948). Problem solution by monkeys following bilaterial removal of the prefrontal areas: VI Performance on tests requiring contradictory reactions to similar and to identical stimuli. Journal of Experimental Psychology, 38, 50-65.

Weigl, E. (1942). On the psychology of so-called processes of abstraction. Journal of Abnormal and Social Psychology, 36, 319-342.

36Zable, M. & Harlow, H. F. (1945). The performance of rhesus monkeys on series of

object-quality and positional discriminations and discrimination reversals. Journal of Comparative Psychology, 39, 13-23.

37