meagan e. brock Æ andrew vert Æ vykinta kligyte...

ORI GIN AL PA PER

Mental Models: An Alternative Evaluationof a Sensemaking Approach to Ethics Instruction

Meagan E. Brock Æ Andrew Vert Æ Vykinta Kligyte ÆEthan P. Waples Æ Sydney T. Sevier ÆMichael D. Mumford

Received: 23 January 2008 / Accepted: 28 May 2008 / Published online: 21 June 2008

� Springer Science+Business Media B.V. 2008

Abstract In spite of the wide variety of approaches to ethics training it is still

debatable which approach has the highest potential to enhance professionals’

integrity. The current effort assesses a novel curriculum that focuses on metacog-

nitive reasoning strategies researchers use when making sense of day-to-day

professional practices that have ethical implications. The evaluated trainings

effectiveness was assessed by examining five key sensemaking processes, such as

framing, emotion regulation, forecasting, self-reflection, and information integration

that experts and novices apply in ethical decision-making. Mental models of trained

and untrained graduate students, as well as faculty, working in the field of physical

sciences were compared using a think-aloud protocol 6 months following the ethics

M. E. Brock (&) � A. Vert � V. Kligyte � M. D. Mumford

Department of Psychology, University of Oklahoma, 455 W. Lindsey Street, Dale Hall Tower,

Room 705, Norman, OK 73019-2007, USA

e-mail: [email protected]

A. Vert


V. Kligyte


M. D. Mumford


E. P. Waples

Department of Management and Marketing, Louisiana State University in Shreveport, One

University Place, Shreveport, LA 71115, USA


S. T. Sevier

Department of Anthropology, University of Oklahoma, 455 West Lindsey, Dale Hall Tower, Room

521, Norman, OK 73019-2007, USA


123

Sci Eng Ethics (2008) 14:449–472

DOI 10.1007/s11948-008-9076-3

training. Evaluation and comparison of the mental models of participants provided

further validation evidence for sensemaking training. Specifically, it was found that

trained students applied metacognitive reasoning strategies learned during training

in their ethical decision-making that resulted in complex mental models focused on

the objective assessment of the situation. Mental models of faculty and untrained

students were externally-driven with a heavy focus on autobiographical processes.

The study shows that sensemaking training has a potential to induce shifts in

researchers’ mental models by making them more cognitively complex via the use

of metacognitive reasoning strategies. Furthermore, field experts may benefit from

sensemaking training to improve their ethical decision-making framework in highly

complex, novel, and ambiguous situations.

Keywords Integrity � Ethics � Training � Evaluation � Sensemaking �Mental models � Think-aloud

Introduction

To improve ethical decision-making, one must be able to implement and

subsequently evaluate training, something that is the subject of much debate [55].

Currently, there are several approaches aimed at the enhancement of ethical

decision-making. Such approaches include, but are not limited to, the development

and introduction of ethical codes and policies [1, 7, 38, 64], organizational climate

interventions and assessments [11, 61, 65], and mentoring programs [66].

However, the most notable intervention has been the use of ethics education, also

known as Responsible Conduct of Research (RCR) training [8, 51, 62]. Two

frameworks are most widely utilized in the development of ethics education

training—those with a focus on moral reasoning development [46, 57] and those

that focus on cognitive processes (i.e., sensemaking) underlying ethical decision-

making [45, 53]. The present study will evaluate the sensemaking approach to

training.

Sensemaking involves the generation of a mental model in response to a complex

ethical problem [14, 52]. Prior research has shown that individuals vary in the way

they make sense of different aspects of a problem, and in particular, the

metacognitive reasoning strategies they apply in relation to ethical situations [3,

60]. Metacognition refers to the processes of thinking about one’s own thinking [6].

Researchers suggest that differences in cognition result in the generation and

application of different mental models [26, 50, 68]. To this end, a training program

targeted at enhancing ethical decision-making through engagement in sensemaking,

and metacognitive reasoning strategies in particular, was developed and tested

across the sciences [45, 53]. The assessment (i.e., learning gains) of this

sensemaking ethics training program with regards to immediate knowledge

acquisition and retention has been fruitful in terms of providing validation evidence

for the program [45, 53]. However, a more fundamental question yet to be

answered. Specifically, does sensemaking training actually work to alter the

underlying mental models of researchers?

450 M. E. Brock et al.

123

Thus, the purpose of the present study is twofold. The first goal is to assess a

novel curriculum that focuses on metacognitive reasoning strategies professionals

use when making sense of day-to-day professional issues that have ethical

implications for science. The second goal is to assess mental models of field experts,

or faculty, and graduate students trained and untrained in sensemaking. Assessment

of mental models will be accomplished through an in-depth analysis of underlying

sensemaking processes that scientists use in ethical decision-making and mental

model structure obtained using a think-aloud protocol [19].

Sensemaking Model

Research in sensemaking contends that when people are faced with a novel,

complex, and ambiguous problem having ethical implications for self and others,

individuals tend to derive solutions by actively engaging in sensemaking [15, 68].

More specifically, it is believed that people construct a mental model for

understanding an ethical problem based on personal and professional frames of

reference, when trying to make sense of an ambiguous situation [53]. Sensemaking

refers to building a schematic mental model for decision-making by integrating

different pieces of information in terms of individual emotions, projected outcomes,

individual goals, and social expectations [14]. This information integration, or

sensemaking process, gives rise to a final decision or action. Ethical decision-

making is held to depend on the underlying processes of sensemaking—framing,

emotion regulation, forecasting, self-reflection, and information integration—thus

making it an important framework to examine within ethical decision-making [52,

68]. Figure 1 illustrates the sensemaking model [53].

Framing Emotions

Forecasting

Self-reflection

Sensemaking

Decision

Prior Professional Experience

Prior Personal Experience

Professional & Personal

Goals

Perceived Requirements for Goal Attainment

Responsible Conduct of Research Principles

Perceived Causes of Situation

Fig. 1 Sensemaking model of ethical decision-making [53]

Mental Models 451

123

Sensemaking Training

Mumford and colleagues [53] developed a two-day sensemaking ethics curriculum

for scientists in the social, health, and biological sciences, with the goal of

enhancing ethical decision-making through engagement in sensemaking and

application of seven metacognitive reasoning strategies. Definitions of these

reasoning strategies are provided in Table 1.

The original RCR training curriculum was adapted for professionals specializing

in the physical sciences and engineering by emphasizing the interactive nature of

the intervention through the combination of lecture, discussion, and experiential

exercises [45]. The training consisted of ten modules primarily focused on

discussing the complexity of ethical decision-making and the limitations of a rule-

based approach in highly complex and ambiguous ethical situations; common

reasoning errors in decision-making (e.g., personal biases); and metacognitive

reasoning strategies that help address typical reasoning errors [45]. For a more

detailed description of the RCR training content see Kligyte et al. [45].

Training Assessment

The effectiveness of the training intervention was assessed using an Ethical

Decision-Making (EDM) measure, developed by Mumford and colleagues [54].

This measure assesses four broad domains of ethical behavior, namely (1) data

management, (2) study conduct, (3) professional practices, and (4) business

practices. The measure contains six broad scenarios, followed by three stems, or

subscenarios. The stems represent a wide range of day-to-day ethical events

within the specific professional field being assessed (e.g., data management in

engineering). For each item, eight potential response options are presented, and

Table 1 Metacognitive reasoning strategies

Strategy Operational definition

1 Recognizing one’s

circumstances

Thinking about origins of problems, individuals involved, and

relevant principles, goals and values

2 Seeking outside help Talking with a supervisor, peer, or institutional resource, or

learning from others’ behaviors in similar situations

3 Questioning one’s own and

others’ judgment

Considering reasoning errors that people often make when making

ethical decisions, remembering that decisions are seldom perfect

4 Dealing with emotions Assessing and regulating emotional reactions to situations

5 Anticipating consequences of

actions

Thinking about many possible outcomes such as consequences for

others, and short-term and long-term outcomes based upon possible

decision alternatives

6 Looking within by analyzing

personal motivations

Considering one’s own biases, effects of one’s values and goals,

how to explain/justify one’s actions to others, and questioning

ability to make ethical decisions

7 Considering others’ perspectives Being mindful of others’ perceptions, concerns, and the impact of

personal actions on others, socially and professionally


123

the participant chooses two most appropriate actions. The responses provided

represent a variety of potential actions ranging from low to high ethicality as well

as application of a particular metacognitive reasoning strategy. Further, the pre–

post assessment of training effectiveness includes the measurement of change in

both ethicality of decision-making and application of metacognitive reasoning

strategies.

Training Results

The sensemaking training resulted in effect sizes (Cohen’s d) ranging from .53 to

1.82 [45]. Furthermore, training also led to a preference for decisions involving the

application of the afore-mentioned metacognitive reasoning strategies, which

facilitate performance on the five major sensemaking processes. More specifically,

significant metacognitive reasoning strategy gains ranged from .60 to 1.51. Based on

the sizeable increase from pre to post-training, the following hypothesis is proposed:

Hypothesis 1 Individuals trained in application of metacognitive reasoning

strategies will be more proficient in sensemaking as compared to untrained

individuals.

While prior research has shown that evaluating the amount of learning

(declarative knowledge) is the most common means of assessing training effective-

ness [13, 41, 42], some researchers propose that the examination of changes in

knowledge structures or mental models holds considerable promise in assessment of

learning in a training context [13, 47]. A mental model is different than declarative

knowledge in that it represents the schematic organization of knowledge (i.e.,

structure), rather than the amount of acquired knowledge [35, 47, 58].

According to Stout and colleagues [63], measures of knowledge structure may be

more likely to uncover effects of learning, as they are less susceptible to attenuation

and are sensitive enough to capture differences even when all participants have

some familiarity with the training concepts [63]. Therefore, as sensemaking training

effectively enhances ethical decision-making through the use of metacognitive

reasoning strategies, it is legitimate to expect that it will also introduce a shift in

novices’ mental models [29, 63]. On the basis of the afore-mentioned assumptions,

the following hypothesis is proposed:

Hypothesis 2 Individuals trained in the application of metacognitive reasoning

strategies during sensemaking will experience a shift in mental model structure as

compared to untrained individuals.

Sensemaking and Expertise

While differences in the sensemaking processes between trained and untrained

novices are important, another area in need of investigation is the influence of

expertise on application of sensemaking processes. When people make decisions

they often refer to their prior personal and professional experiences that facilitate

decision-making in novel situations. Consequently, as individuals gain experience

Mental Models 453

123

they are also acquiring knowledge about influential factors that need to be

considered within specific situations as well as improving upon their metacognitive

reasoning strategies for ethical problem resolution [18].

As experts (i.e., faculty) are more familiar with written and unwritten ethical

guidelines and acceptable professional practices in the field, they likely become

more accurate in identifying ethical implications of the situation [9, 43]. Moreover,

prior exposure to ethical or conflict situations provides individuals with the

framework for dealing with the emotional aspects of a situation [5, 30, 40, 49]. Prior

research has demonstrated that experts rely on case-based knowledge in decision-

making. Case-based knowledge allows experts to access and interpret relevant and

salient information more effectively, primarily due to their effectiveness in

organizing and integrating schemata [16, 32, 34], which facilitates decision-making

when choosing between alternatives [5]. Based on the above discussion of

sensemaking and expertise, the following hypothesis is proposed:

Hypothesis 3 Field experts will be more proficient in the sensemaking process

than either trained or untrained novices.

Following an expertise-focused approach, in theory field experts should have a

superior mental model due to their ability to integrate and structure information

more effectively and efficiently [24, 48]. On the other hand, while experts typically

exhibit superior decision-making, the mental models used across situations tend to

be similar. More specifically, experts tend to apply heuristics in decision-making

across situations [36]. Thus, in novel situations, experts are more likely to default to

an existing mental model instead of constructing a new one or expanding on the old

one based on the demands of a new situation [56].

Due to rich prior experience experts are likely to make an appropriate final

decision. Given this rationale, experts may generate a more ethical final solution

than trained and untrained novices by focusing on the extensive knowledge of

common rules and norms in the field [9, 10, 28]. However, the mental model applied

in ethical decision-making may be similar to an untrained novices’ model in relation

to highly ambiguous and novel ethical problems due to a lack of experience with the

variety of ethical situations outside their major area of expertise. Based on the

above-provided rationale, the following hypothesis is proposed:

Hypothesis 4 Field experts will resemble untrained novices in terms of mental

model structure when solving a novel, highly complex, and ambiguous ethical

problem.

Further, it is possible to predict that individuals trained in sensemaking will

develop superior mental models to experts’ models. Experts rely on their prior

experience in a particular domain, often overlooking the current situational needs,

which limit their exposure to particular issues in an academic environment [44].

Sensemaking training, on the other hand, introduces individuals to a variety of novel

and highly realistic ethical situations that are complex and ambiguous in nature.

Training also emphasizes the application of metacognitive reasoning strategies in

relation to ethical problems that enable an individual to assess the situation from

different perspectives by considering both personal and contextual factors when


123

solving ethical problems [45, 53]. The focus on strategy application in a myriad of

novel and realistic situations may aid in reconstructing trainees’ initial novice

mental model by increasing its complexity and adaptability to changing situational

demands. Based on the afore-mentioned arguments, the following hypothesis is

proposed:

Hypothesis 5 Individuals trained in sensemaking will exhibit a superior mental

model structure as compared to untrained field experts and novices.

As the present study attempts to assess the effectiveness of the sensemaking

training by examining it from two different perspectives—learning gains and shifts

in mental model structure—a think-aloud protocol was chosen as the most suitable

method for extracting information on cognitive processes underlying ethical

decision-making.

Method

Sample

Twenty-eight faculty, research scientists, and graduate students in a top-ranked

program, from a large Southwestern university volunteered to take part in the 2-hour

study. All participants were members of a multi-disciplinary, multi-university

research center involved in the development of weather sensing technology. At the

time of this study, the branch of research center at the focal university had 17

graduate students, and 21 faculty and research scientists specializing in the fields of

electrical engineering, computer science, and meteorology. The population was

chosen because the research center members were offered the opportunity to

voluntarily participate in the RCR sensemaking training approximately 6 months

prior to the study. The total sample for the present study included 15 faculty and

research scientists (Mtenure = 16.5, SD = 2.47) and 13 graduate students (Mten-ure = 3.5, SD = 1.27). The sample included 64% males and 71% Caucasians.

Specifically, six of the 28 participants (all graduate students) had attended the

sensemaking-oriented RCR training. Thus, the final sample included 13 faculty and

research scientists (hereafter field experts), six RCR trained graduate students, and

seven untrained graduate students (hereafter trained and untrained novices). Due to

poor audio quality two field expert interviews were not included in the final

analyses.

Procedure

Participants were recruited for the study via e-mail. Upon arrival, participants were

asked to sign an informed consent form and were provided with detailed

instructions. Participants were asked to think-aloud in relation to four scenarios,

each addressing one of four areas of ethical conduct, namely (1) data management,

(2) study conduct, (3) professional practices, and (4) business practices [31] to

examine the processes that people engage in when making decisions in relation to

complex and ambiguous ethical situations.

Mental Models 455

123

Think-aloud Procedure

As mental models represent how people develop a solution and what processes are

salient in decision-making, their examination requires the use of a verbalization

technique. Therefore, a think-aloud (TA) method was selected for assessing the

implicit schematic mental models of people engaged in ethical decision-making.

Think-aloud protocol involves a participant solving a problem and making a

decision by verbalizing their thoughts aloud while performing a task [20]. This

method is particularly helpful in assessing implicit mental processes, because it (1)

enables the collection of great quantities of uncensored, spontaneous statements by

a participant that are more reflective of the actual thinking process, and (2) does not

interfere with the decision-making task [21, 23].

The TA protocol consisted of two stages, first (1) unstructured and then (2)

structured (control) verbalization. In an unstructured protocol, while discussing

each scenario the participant was instructed to TA in a stream of consciousness

while generating a solution to the scenario. The experimenter did not prompt the

participant unless the individual stopped speaking. In such cases scripted prompts

were applied (e.g., ‘‘Where did you get that idea from?’’). Of note is that the

interviewer was not blind to the level of the participants’ expertise; however only

the predetermined prompts were applied across participant groups. The structured

protocol was included for control purposes to see if participants’ mental models

would replicate the order and structure of questions asked. In the structured

protocol, the participants were provided with a series of specific questions

targeted at extracting information in relation to the specific sensemaking

processes. For example, to address situation appraisal the following question

was asked: ‘‘What do you see as the primary problem in this situation?’’, whereas

to address environmental monitoring the participants had to TA in response to the

following question: ‘‘What kinds of factors in the scenario are important to

consider in solving the problem?’’ In line with previous suggestions for enhancing

the TA process, participants were provided with an opportunity to practice

thinking-aloud with the experimenter prior to the actual session to an unrelated

issue [4].

Following the TA practice session, the same procedure, as described above, was

performed in relation to all four scenarios. Upon completion of the study

participants were debriefed and thanked for participation. Participants’ responses

were recorded via digital voice recorder and transcribed for coding purposes.

Scenario Development and Content

As previously discussed, the participants responded to four scenarios. These

scenarios pertained to one of the following four areas of ethical behavior discussed

previously. The underlying ethical problems discussed in the scenarios were

identified on the basis of ethnographic observations conducted at the research center

and information provided by two Subject Matter Experts (SMEs) in the fields of

meteorology and computer science. The scenarios were generated by a panel of

three psychologists familiar with the ethics literature and reviewed by the SMEs to


123

increase psychological fidelity of the scenarios by focusing on the relevancy and

saliency of issues to the field.1

Coding

The transcribed unstructured and structured verbalizations were coded indepen-

dently for each scenario. Consequently, eight responses were coded in relation to

each participant. To analyze the TA transcriptions a coding system was developed

following the sensemaking framework (see Fig. 1). The 15 processes underlying

five broad sensemaking constructs were identified theoretically by a panel of five I/

O psychologists using existing literature on sensemaking and ethical decision-

making [53]. The examined processes were as follows: (1) framing (problem

appraisal, goal assessment, environmental monitoring, norm-based framing, value-

based framing, perceived threat and opportunity), (2) affect (emotion regulation),

(3) forecasting (autobiographical extraction, outcome assessment, solution revision,

contingency planning), (4) self-reflection (other-based perceptions, self-based

perceptions), (5) sensemaking (information integration, solution appraisal)—and

the overall ethicality of the final decision. For each of the 15 sensemaking processes,

anywhere from 6 to 19 behavioral cues were developed by the same panel of

psychologists using existing literature in cognition and sensemaking processes,

resulting in 210 items to be rated, including the overall ethical decision. Table 2

provides examples of constructs and specific items subsumed under them.

Participants’ responses were rated by three expert judges, which consisted of two

I/O psychologists and an anthropologist familiar with sensemaking and ethical

decision-making literature. As the judgments were being made on 15 complex

sensemaking processes these judges were more appropriate for rating purposes than

scientists, who are less familiar and thus less comfortable with sensemaking

concepts [2]. Judges were blind to the group membership. Prior to rating, the judges

received 30 h of frame-of-reference training in relation to each of the sensemaking

constructs. Training consisted of (1) familiarizing with the definitions and examples

of processes; (2) individual practice of rating responses, and (3) group discussions.

Inter-rater agreements were calculated for each construct, and averaged to create a

single composite agreement index, ICC = .92.

Manipulation Check

The relevancy of scenarios was evaluated by three blind raters, I/O psychologists,

who determined if a typical individual would identify the constructs, or sensemak-

ing processes, as salient to the situation at hand and would address them in their TA

process while generating a decision. The ratings were performed using a 5-point

Likert scale (1-not at all, 5—very much so). Items were considered irrelevant when

at least two out of three raters assigned a value of 1 to a specific item. However, as

the ratings were consistently scored higher than 1, no items were excluded from the

analyses.

1 The scenarios are available from the authors upon the request.

Mental Models 457

123

Table 2 Sensemaking model processes and underlying dimensions

Sensemaking

process

Dimension Dimension definition Item example

Framing Problem

Appraisal

Assessing the primary causes of

the issue at hand

• The primary cause of the

problem is clearly identified

• The cause of the problem is

believed to be environmental

factors

Goal Assessment Discussing the goals and

expectations of all parties

involved in the situation

• The goals of the organization

are clearly stated

• The individual expectations of

the sponsor/consumer are

clearly stated

Environmental

Monitoring

Discussion of the current political,

financial and social climate

• The current political climate is

discussed

• The current economic climate

is discussed

Norm-based

Framing

Discussion of social and

professional rules, codes of

conduct

• Social values considered in

solving the problem are

discussed

• Professional conduct rules

considered in solving the

problem are discussed

Perceived Threat

and Opportunity

Individual discussion of the

threats and opportunities of a

potential solution, for all parties

• Direct threats to others are

discussed

• Indirect threats to the

individual are discussed

Value-based

Framing

Individual consideration of

beliefs, values and personal

motivations

• Decision consistency with

values are discussed

• Personally oriented motivations

are discussed

Affect Emotion

Regulation

Assessing and regulating

emotional reactions

to situations

• Emotional responses are

discussed

• Weighing emotional versus

rational factors is discussed

Forecasting Autobiographical

Extraction

Discussion of personal and

professional experiences and

outcomes

• Prior professional experiences

are described in reference to

the problem

• Professional feedback received

in the past is discussed

Solution Revision Individual discussion of

alternative solutions and

outcomes given the situation at

hand

• Alternative outcomes are

generally discussed

• Alternative outcomes for the

team are discussed

Contingency

Planning

Development of back-up plans to

address potential issues in the

future

• Alternate plans are discussed in

terms of something going

wrong in the future

• The adaptability of the idea is

discussed


123

Analysis

In order to further investigate the validity of the sensemaking training, two types of

analyses were conducted. First, a one-way analysis of variance was conducted to

evaluate differences between field experts, trained, and untrained individuals on the 15

constructs related to the overarching sensemaking processes examined in the present

study. Second, Pathfinder [59] was utilized to examine the mental model structure of

participants as represented by the 15 constructs related to sensemaking (see Table 3).

Pathfinder generates a graphical representation of the relatedness of concepts

using a network scaling procedure [59]. The linkages represent relationships

between concepts, and the length of the linkage represents the strength of the

relationship. The clustering of the concepts indicates that multiple concepts are

more related to each other than to concepts in a different cluster. For a complete

description of Pathfinder refer to Schvaneveldt [59]. Pathfinder networks have been

used in the past to evaluate training and have been reported as a significant source of

additional validation information [27].

Computing Distance Scores

In the present study, a distance metric was utilized to determine the interrelatedness

of the 15 sensemaking constructs. The first step in generating the distances was to

Table 2 continued

Sensemaking

process

Dimension Dimension definition Item example

Self-

reflection

Self-based

Perceptions

Consideration and justification of

individual ideas and opinions

• The individual justifies the

decision to himself/herself

• The individual acknowledges

self-biases

Others-based

Perceptions

Situational perceptions of friends,

colleagues, advisors and sponsors

regarding the situation

• Asking an advisor for their

input is discussed

• Directly asking for other

people’s opinion if they cannot

forecast their reactions is

discussed

Sensemaking Information

Integration

Consideration of the advantages

and disadvantages for all parties

given a certain course of action

• Personal advantage of the

situation is discussed

• The severity of consequences

of alternatives is discussed

Solution

Appraisal

Assessment of solution

effectiveness and ethicality

• The strategy for problem

resolution is logical

• The strategy for problem

resolution is ethical

Decision Final Decision Ethicality of final solution • The final idea proposed is

ethical

• Outcomes associated with the

final solution are ethical

Mental Models 459

123

determine difference scores using the absolute value of the differences for all

possible comparisons between the concepts for the mean of each group. Next,

Mahalanobis distances were computed using the differences scores. The distance

metric was computed in this manner because Mahalanobis distances are sensitive to

sample size. Thus by using difference scores sample size was consistent across all

groups. Finally, the distances were entered into Pathfinder for analysis. The

resulting representations refer to the similarities of the coded response scores of the

groups involved in ethical decision-making. Although the representation is not

directional, in that it does not demonstrate exactly the processes in order through

which the participants made their decisions, the similarities of processes demon-

strate the extent that processes are discussed in relation to each other.

Results

In the present study, each participant’s TA protocol included both an unstructured

and structured component. The outcomes of the structured protocol were used for

control purposes as a check to see how participants would respond if they were

guided through the think-aloud process with structured questions. Results of the

analyses on the structured protocol revealed that participants’ mental model

structure adhered to the order in which the questions were asked and differed from

Table 3 Results of one-way ANOVA with post-hoc comparisons for underlying sensemaking processesa

One-way ANOVA Post-hoc comparisons

Untrained

novice (1)

Trained

novice (2)

Field

expert (3)

1 vs. 2 1 vs. 3 2 vs. 3

M SD M SD M SD F p p p p

Problem Appraisal 1.63 0.16 1.83 0.11 1.94 0.18 11.81 0.00 0.01 0.00 0.09

Emotion Regulation 1.01 0.01 1.03 0.03 1.23 0.18 18.67 0.00 0.64 0.00 0.00

Goal Assessment 1.54 0.18 1.57 0.12 1.93 0.18 29.48 0.00 0.59 0.00 0.00

Environmental Monitoring 1.72 0.25 1.70 0.23 2.11 0.19 21.18 0.00 0.80 0.00 0.00

Autobiographical Extraction 1.07 0.07 1.03 0.02 1.39 0.27 18.52 0.00 0.56 0.00 0.00

Norm-based Framing 1.54 0.26 1.69 0.19 1.82 0.20 7.85 0.00 0.08 0.00 0.08

Others-based Perceptions 1.25 0.16 1.27 0.08 1.32 0.12 0.61 0.55 0.77 0.30 0.50

Information Integration 1.59 0.16 1.68 0.19 1.83 0.17 7.02 0.00 0.24 0.00 0.04

Solution Appraisal 2.57 0.29 2.71 0.27 3.20 0.37 17.47 0.00 0.32 0.00 0.00

Threat and Opportunity 1.47 0.22 1.56 0.21 1.83 0.23 12.54 0.00 0.33 0.00 0.00

Value-based Framing 1.28 0.17 1.38 0.13 1.60 0.22 14.35 0.00 0.17 0.00 0.00

Outcome Assessment 1.33 0.15 1.39 0.15 1.53 0.19 5.53 0.01 0.41 0.00 0.04

Solution Revision 1.23 0.14 1.32 0.18 1.38 0.21 1.65 0.20 0.40 0.07 0.45

Self-based Perceptions 1.40 0.16 1.43 0.08 1.70 0.16 30.03 0.00 0.54 0.00 0.00

Contingency Planning 1.22 0.11 1.26 0.05 1.28 0.18 0.51 0.60 0.62 0.32 0.69

Ethical Decision 2.63 0.28 2.91 0.23 3.07 0.43 4.64 0.01 0.10 0.00 0.30

a F(2,101); p \ .05


123

the findings of the unstructured protocol.2 Prior to discussing mental model results

(e.g., Hypotheses 2, 4, & 5), ANOVA results will be presented for the unstructured

protocol.

Assessment of Individuals’ Proficiency in Sensemaking

A one-way ANOVA was conducted to identify whether untrained novices, trained

novices, and field experts differed on the 15 sensemaking processes, and overall

ethical decisions, that were coded by the expert judges. Results indicated significant

group differences on 12 of the 15 sensemaking processes, namely problem

appraisal, emotion regulation, goal assessment, environmental monitoring, auto-

biographical extraction, norm-based framing, information integration, solution

appraisal, perceptions of threat and opportunity, value-based framing, outcome

assessment and self-based perceptions, as well as overall ethical decision made. To

further assess the differences between groups, a series of post-hoc pairwise

comparisons, using Tukey’s Honestly Significant Difference (HSD) test, were

conducted. Full means, standard deviations, ANOVA and post-hoc test results for

the unstructured condition may be found in Table 3.

The first comparison was done to examine Hypothesis 1, which stated that those

individuals who had been trained in the application of metacognitive reasoning

strategies would be more proficient in sensemaking than untrained novices. Trained

novices scored significantly higher one of framing processes, problem appraisal. No

other significant differences emerged. The results of the mean differences in

sensemaking processes between the trained novices and untrained novices yield

weak support for Hypothesis 1.

The second and third comparisons examined Hypothesis 3, which stated that field

experts would be more proficient than both untrained and trained novices in

sensemaking. Experts were found to be more proficient in sensemaking processes

than untrained novices. More specifically, experts’ performance was superior in

problem appraisal, goal assessment, environmental monitoring, norm-based fram-

ing, value-based framing, and perceived threat and opportunity—all components of

framing. Further, experts were more proficient in emotion regulation, and also in the

markers of forecasting, namely autobiographical extraction, and outcome assess-

ment. Interestingly, there were no differences between experts and untrained

novices for the contingency planning. Similarly, in terms of self-reflection there

were no differences observed in relation to other-based perceptions; however

experts significantly focused on self-based perceptions. Marked by differences in

information integration and solution appraisal, experts outperformed novices in the

integrative sensemaking process. Further, there was a significant difference between

experts and untrained novices on overall ethical decision, with experts outperform-

ing untrained novices. Thus, for the comparison of experts against untrained

novices, this hypothesis was supported.

Regarding the comparison of field experts to trained novices different findings

emerged. Here, experts were more proficient in the framing processes of goal

2 The results of the structured protocol are available from the authors upon the request.

Mental Models 461

123

assessment, environmental monitoring, value-based framing, and perceived threat or

opportunity. No differences were found for problem appraisal or norm-based

framing. Experts also surpassed trained novices in emotion regulation. However,

experts were superior to trained novices in only two of four forecasting components,

autobiographical extraction and outcome assessment, and only one self-reflectionmarker, self-based perceptions. Finally, experts outperformed trained novices in

both sensemaking components, information integrations and solution appraisal.

Despite outperforming trained novices on several of the sensemaking processes,

experts and trained novices did not differ in their overall ethical decisions made.

Thus, for the comparison of experts against trained novices, this hypothesis was

partially supported.

Assessment of Mental Models

To help determine the extent of mental model shifts, both the structured and

unstructured TA protocols were assessed. This was done because implicit mental

models obtained via the structured, or prompted protocol (control) should be more

similar to each other than those extracted via the unstructured protocol (see footnote

2). As the ultimate goal of this study was to examine the unprompted cognitive

processes of individuals, only the unstructured protocol analyses are presented. The

mental models represent the interrelationships between constructs, or patterns of

constructs. Thus, schematic mental model processes should be used to interpret the

structure of the sensemaking process depicted.

Hypothesis 2 predicted that the mental models of individuals trained in the

application of metacognitive reasoning strategies would evidence a shift from those

not trained in the application of strategies. In the untrained novice’s mental model

the core component is environmental monitoring. This initial monitoring is

influenced by others’ perceptions and value-based framing. Ultimately, the appraisal

and revision of the solution is also centered on environmental monitoring. The

second core component is autobiographical extraction. Thus, past experience

essentially drives norm-based framing, goal assessment, and information integration

for untrained novices. Of note is that perceived threat or opportunity is directly

linked to emotion regulation, which is related most strongly to goal assessment and

self-based perceptions.

The mental model of the trained novices is considerably different conceptually.

First, the core component in the trained novices’ mental model is the appraisal of the

problem. This is linked to both forecasting (i.e., outcome assessment and solution

revision), self-reflection (other-based perceptions), and sensemaking (solution

appraisal). Thus, for the trained novices, framing, in the form of a problem-based

assessment, is essential. This assessment is also linked to contingency planning,

which incorporates additional elements of framing (i.e., norm-based framing and

environmental monitoring). Information integration is then directly linked to

contingency planning. Of note is that the most distal influences on information

integration, and ultimately the problem solution, are autobiographical extraction,

goal assessment, and emotion regulation, respectively. From the examination of the

mental models of trained and untrained novices is apparent that considerable


123

conceptual differences exist between how these two groups sensemake during

ethical decision-making, providing support for Hypothesis 2. These differences will

be more clearly delineated in the discussion. Figures 2 and 3 illustrate the mental

models for untrained and trained novices in the unstructured condition. Of note is

that these mental models are schematic in nature. They outline the relationships

between 15 sensemaking processes and illustrate their significance in researchers’

ethical decision-making examined via unstructured think-aloud protocols.

Hypothesis 4 stated that experts would resemble untrained novices in their mental

model structure. The key similarity for the unstructured models is that both groups

rely on environmental monitoring. Both groups utilize autobiographical extraction

to inform the initial appraisal of the problem and the overall framing. Similarly,

both groups utilize emotion regulation in conjunction with self-based perceptions

and perceived threat or opportunity. This evidence provides strong support for

Hypothesis 4. The schematic mental model for field experts in the unstructured

condition can be found in Fig. 4.

Finally, Hypothesis 5 stated that individuals trained in the application of

metacognitive reasoning strategies would exhibit a superior mental model structure

when compared to either experts or untrained novices. A central piece of trained

Emotion Regulation

Problem Appraisal

Goal Assessment

Env. Monitoring

Autobiographical

Norm-based Framing

Ethical Decision

Contingency Planning

Other-based Perceptions

Self-based Perceptions

Info Integration

Solution Appraisal

Perceived Tht/Opp

Value-based Framing

Outcome Assessment Solution Revision

Fig. 2 Mental models for untrained novices (N = 7) based on the 15 sensemaking processes asdiscussed in the unstructured protocol condition

Mental Models 463

123

individuals’ mental model structure is problem appraisal, a key framing construct.

Thus, the entire mental model of sensemaking, for trained individuals, is built around

active assessment of the ethical problem. On the contrary, untrained novices and

experts rely more centrally on environmental monitoring, which may be defined as a

more passive approach to information gathering—a decision is based on what is

superficially recognized and observed in the environment. Thus, for field experts, and

particularly for untrained novices, problem appraisal appears to be a much less central

process in ethical decision-making. That being said, the mental models produced by

trained novices are, in fact, superior in nature to those produced by either field experts

or untrained novices, providing support for Hypothesis 5. The reason for this is that

sensemaking involves processes that are meant to gather a variety of information

about a situation. Within sensemaking training the major focus is the application of

metacognitive reasoning strategies. Thus, we would expect that trained individuals

would take a more active information seeking approach to situational assessment, or

framing and forecasting within the situation than untrained individuals.

Discussion

Prior to discussing the findings and implications of the present study, certain

limitations should be noted. In particular, this study is limited by sample

Emotion Regulation

Problem Appraisal

Goal Assessment

Env. Monitoring

Autobiographical

Norm-based Framing

Ethical Decision




Info Integration

Solution Appraisal

Perceived Tht/Opp

Value-based Framing

Outcome Assessment Solution Revision

Fig. 3 Mental models for trained novices (N = 6) based on the 15 sensemaking processes as discussedin the unstructured protocol condition


123

characteristics. First, the small sample size in the present study limits the strength of

inferences drawn as well as the generalizability of the findings. That being said, the

sample size for the present study is fairly typical for think-aloud protocols, which

normally utilize 10–15 participants, as such methods are relatively time consuming

and costly [20].

Second, the samples sizes for individual groups, particularly the trained and

untrained groups, are small. However, since the numbers of trained individuals were

limited and all participants were from one institution, it is difficult to obtain more

participants, as departments are limited in size and this was a voluntary effort.

Further, based on the population of interest, over 70% of that population

participated in the study.

Third, individual differences were not assessed. Prior research has shown that

qualities such as verbal intelligence [67] and personality traits such as extraversion

[33] may impact the nature of think-aloud protocol results. More importantly,

previous research has shown that individual differences such as emotional

intelligence [22, 25, 69] and neuroticism and openness [3] impact receptivity and

application of training efforts [45]. However, due to concerns for anonymity by the

sample population data collection on individual differences was restricted.

Fourth, the study was limited by the application of a non-crossed design. A

crossed design would have provided additional insight into differences between the

Emotion Regulation

Problem Appraisal

Goal Assessment

Env. Monitoring

Autobiographical

Norm-based Framing

Ethical Decision




Info Integration

Solution Appraisal

Perceived Tht/Opp

Value-based Framing

Outcome Assessment

Solution Revision

Fig. 4 Mental model for field experts (N = 13) based on the 15 sensemaking processes as discussed inthe unstructured protocol condition

Mental Models 465

123

groups; however this project was a follow-up study meant to evaluate the

effectiveness of ethics training using an alternative assessment approach 6 months

following the training. As the training was conducted in an actual organization—a

multi-disciplinary and multi-university research center—and the participation was

voluntary, we did not have control over who received training. As no faculty or

research scientists took part in the training it was unfeasible to cross faculty versus

graduate students with trained versus untrained conditions. Future research should

investigate the differences between field experts trained and untrained in research

ethics to validate the currently obtained findings.

Finally, the present study is limited by the fact that the differences between

groups examined in the present study were only examined within a sensemaking

training framework. Thus, we were not able to examine differences in sensemaking

processes from individuals who had received a different form of ethics training (i.e.,

professional associations or codes of ethics). The evaluation methodology proposed

in the present study could be potentially applied to examine alternative methods for

ethics training.

Bearing these limitations in mind, the present study provides several compelling

and thought provoking findings. First, the study provides additional validation

evidence for sensemaking training by identifying the differences in key sensemak-

ing processes applied by field experts, untrained novices, and trained novices in

complex and ambiguous ethical situations. This is notable as interviews were

conducted 6 months following the training providing evidence for knowledge

transfer and retention. Second, considerable structural differences in mental models

of field experts, untrained, and trained novices were found, both in terms of the

number and types of relationships between key sensemaking processes. More

specifically, trained novices exhibited a shift in mental model, generating a more

cognitively complex model than untrained novices and experts. Further, field

experts demonstrated mental model structures that were similar to untrained

novices’, in that they were routine in nature. Consistent with prior research on

expertise [32, 43], it is not surprising that field experts were more proficient than

untrained and trained novices in sensemaking.

It is also not surprising that trained novices were more proficient than untrained

novices in application of sensemaking processes, as trainees were exposed to a

number of metacognitive reasoning strategies practiced in a variety of novel and

complex situations during ethics training. However, interestingly, this did not result

in decisions higher in ethicality than of trained novices’. Such results further

confirm that training should not only be examined in terms of group mean

differences in the application of sensemaking processes but also in terms of

differences in mental model structure [13].

When comparing the mental model structures of trained novices versus untrained

novices and field experts it is evident that those who have been trained exhibit a

model that is more cognitively complex. While experts and untrained novices

develop solutions to ethical problems by focusing on environmental influences,

trained novices tend to take a more in-depth analysis of the ethical situation by

assessing specific situation needs and considering themselves and their role, or

position, in the context of the problem. Trained individuals look within by assessing


123

personal as well as other people’s motivations in the situation and by questioning

potential biases in their assessment. Greater awareness of the self in context of

ethical problem represents a more internal focus in ethical decision-making as

compared to untrained novices and experts. Therefore, the trained individuals are

more likely to identify potential ethical implications in relations to complex and

ambiguous situations, which may not always be the case with untrained novices and

experts. A clear understanding of the problem and consideration of ideas and

opinions in context results in a decision that reflects one’s perception of the correct

course of action and leads to better-informed decisions.

Conversely, experts take a more pragmatic world view. On the basis of

organizational experience, experts have learned to assess problems in terms of

potential risks and benefits within a given situation, especially when the situation

has implications for the self. In highly ambiguous situations experts may become

defensive in situations of perceived threat; resulting in less ethical decisions if the

perceived threats or benefits of an ethical decision are outweighed by the need for

self-protection.

When developing solutions to problems, experts, like trained novices, search

their memory for prior personal and professional experiences that facilitate the

interpretation of current environmental cues. Doing so provides a framework for

forecasting potential outcomes. Further, examination of their mental models

indicate that untrained novices and experts are solving ethical problems by

monitoring the environment and collecting additional information necessary to

adjust their schemata to a new situation. However, they are not accurately

appraising the problem via the assessment of specific situational and personal needs,

or unique circumstances, of the situation. Overall, it has been shown that untrained

novices and experts were quite similar when generating solutions to highly complex

and ambiguous ethical problems—despite the experts’ greater skill in applying this

model.

While environmental monitoring is salient in ethical decision-making, one has to

assess both external and internal (i.e., personal) factors in the situation to obtain a

broad picture of a problem. Furthermore, given that untrained novices do not have a

considerable amount of prior experience with ethical issues, they are developing

solutions to highly complex and ambiguous situations on the basis of their limited

and poorly integrated schemata as well as unidimensional assessment of environ-

mental cues, which naturally result in poorly-informed decisions [37].

Poor decisions often stem from people’s tendencies to over-rely on their previous

experiences, which do not always generalize to new situations. Prior research has

shown that experts tend to apply heuristics, which result in fast and often highly

effective solutions due to rich, flexible, and well-integrated schemata [9]. However,

this strategy may not be effective in novel and highly ambiguous situations, in

which experts may have no prior exposure. For example, field experts are highly

familiar with ethical violations inherent in research practices (e.g., data hoarding,

falsification, etc.). However, they may not be familiar with ethical considerations

within the domain of professional practices (e.g., moral leadership, effective

collaboration and communication). Thus applying heuristics from one ethical

situation to the next may not always lead to an optimal ethical action. Overall, the

Mental Models 467

123

differences between experts and untrained novices are consistent with a more

personalized outcome focus versus a strategic focus—a difference that often

emerges in novice and expert comparisons [39]. Nonetheless, both models represent

externally driven decision-making systems, based in perceived treat versus

opportunity (experts) or environmental demands (novices).

Potential Explanations

There are several potential explanations for the significant differences in mental

model of trained and untrained individuals. Sensemaking training results in an

alteration of how an individual appraises and understands the problem, providing a

framework for working through the decision-making process. By introducing

trainees to a myriad of real world problems followed by a set of metacognitive

reasoning strategies necessary to make sense of these problems and extensive

practice in application of the strategies to actual and highly realistic cases provided

trainees with the necessary tools to appropriately respond to ethical issues in novel

situations. While this is a plausible explanation for the shift in mental models, the

underlying processes and actual mechanisms driving the mental model shifts, are

not known. We can only assume, based on the results, that sensemaking training

facilitates changes in mental models by increasing their structural complexity and

flexibility. Therefore, extensive research is needed in trying to understand the

relationship between sensemaking processes and mental models, which may result

in more effective and efficient future ethics training efforts.

Implications

Bearing limitations in mind, there are several theoretical and practical implications

are of note based on the findings of this investigation. First, the examination of

training effectiveness is enhanced via investigation of differences of groups’ mental

model structure. Consistent with Day et al.’s [13] research, the evaluation of the

mental models, and more specifically the shift in mental models of trainees,

provided greater insight into the sensemaking process, as wells as the effects of

sensemaking training on ethical decision-making.

Second, the use of multiple methods in evaluating the training provides

additional training validation evidence. The examination of implicit mental models

and significant group differences found between untrained novices and experts and

trained individuals provide further validation evidence that training in sensemaking

can improve one’s ability to make ethical decisions across novel situations. The use

of the think-aloud protocol approximately 6 months after training demonstrated that

the training is effective and that the effects are retained over time. Thus by training

individuals on a variety of metacognitive reasoning strategies and introducing them

to a variety of novel real world cases may enhance professionals’ ethical decision-

making process, which will result in better-informed decisions.

Third, training benefits novices significantly as it improves their sensemaking

process, and, consequently, results in the generation of a more complex and


123

effective mental model. Furthermore, since the structural changes in mental models

for ethical decision-making are not determined by expertise per se, exposing experts

to sensemaking training may enhance their mental model as well. Otherwise,

untrained experts and novices will maintain and apply an outcome-based model

rather than an analytic model, which can result in poor decision-making, especially

in novel and highly complex and ambiguous situations, which are characteristic of

the situations that those in the sciences and engineering face [12, 17]. Therefore,

professionals, in spite of their expertise in any given field, should consider

improving their mental models and ethical decision-making via education in

sensemaking.

Conclusion

Overall, the results of the present study indicate that sensemaking training can lead

to a shift in mental model structure. Specifically, models become more complex and

information becomes highly integrated. Although the sensemaking training was not

specifically designed to alter mental models of trainees, the results are not

necessarily surprising as sensemaking training introduces a complex way of

thinking about novel, highly complex and ambiguous ethical situations that

researchers face in their professional lives. The study demonstrates that training can

significantly benefit both field experts and novices by improving their sensemaking

process as well as introducing complexity in their mental models. The shift in

mental models is facilitated by the introduction of trainees to a diversity of novel

and realistic ethical situations that can be resolved by applying a combination of

metacognitive reasoning strategies. Such strategies are helpful in dealing with the

complexity and ambiguity posed by ethical problems. Consequently, field experts

may also significantly benefit from sensemaking training to improve their ethical

decision-making framework in novel and highly complex and ambiguous ethical

situations.

Acknowledgements We thank Elaine S. Godfrey and Richard T. Marcy for their assistance in

developing theoretical framework and data collection materials for the project. We also thank Dr. Dean F.

Hougen for sharing his expertise in physical sciences which was beneficial in contextualizing the obtained

information. We would also like to acknowledge the National Science Foundation (NSF), contract No.

SES 0529910. The resarch was funded by the Council of Graduate Students Grant, contract No. LTR

090506.

References

1. Adams, J. S., Tashchian, A., & Shore, T. (2001). Codes of ethics as signals for ethical behavior.

Journal of Business Ethics, 29, 199–211.

2. Anderson, M. S. (2007). Collective openness and other recommendations for the promotion of

research integrity. Science and Engineering Ethics, 13, 387–394.

3. Antes, A. L., Brown, R. P., Murphy, S. T., Hill, J. H., Waples, E. P., Mumford, M. D., et al. (2007).

Personality and ethical decision-making in research: The role of perceptions of self and others.

Journal of Empirical Research on Human Research Ethics, 2, 15–34.

Mental Models 469

123

4. Austin, J., & Delaney, P. F. (1998). Protocol analysis as a tool for behavior analysis. Analysis ofVerbal Behavior, 15, 42–56.

5. Baumeister, R. F., Vohs, K. D., DeWall, C. N., & Zhang, L. (2007). How emotion shapes behavior:

Feedback, anticipation, and reflection, rather than direct causation. Personality and Social Psy-chology Review, 11(2), 167–203.

6. Berardi-Coletta, B., Buyer, L. S., Dominowski, R. L., & Rellinger, E. R. (1995). Metacognition and

problem solving: A process-oriented approach. Journal of Experimental Psychology, 21, 205–223.

7. Bullock, M., & Panicker, S. (2003). Ethics for all: Differences across scientific society codes. Scienceand Engineering Ethics, 9, 159–170.

8. Chen, F. W. (2003). A study of the adjustment of ethical recognition and ethical decision-making of

managers-to-be across the Taiwan Strait before and after receiving a business ethics education.

Journal of Business Ethics, 45, 291–307.

9. Chermack, T. J. (2003). Mental models in decision making and implications for human resource

development. Advances in Developing Human Resources, 5, 408–422.

10. Chi, M. T. H., Feltovich, P., & Glaser, R. (1981). Categorization and representation of physics

problems by experts and novices. Cognitive Science, 5, 121–152.

11. Davis, M. (2003). The role of culture in research misconduct. Accountability in Research, 10, 189–201.

12. Davis, M. (2006). Engineering ethics, individuals, and organizations. Science and EngineeringEthics, 12, 223–231.

13. Day, E. A., Arthur, W., & Gettman, D. (2001). Knowledge structures and the acquisition of a

complex skill. Journal of Applied Psychology, 86(5), 1022–1033.

14. Dorner, D., & Schaub, H. (1994). Errors in planning and decision-making and the nature of human

information processing. Applied Psychology: An International Review, 43, 433–453.

15. Drazin, R., Glynn, M. A., & Kazarjain, R. K. (1999). Multi-level theorizing about creativity in

organizations. Academy of Management Review, 24, 286–329.

16. Eisenhardt, K. M. (1989). Making fast strategic decision in high-velocity environments. Academy ofManagement, 32(3), 543–576.

17. Emison, G. A. (2006). The complex challenges of ethical choices by engineers in public service.

Science and Engineering Ethics, 12, 233–244.

18. Ericsson, K. A., & Charness, N. (1994). Expert performance: Its structure and acquisition. AmericanPsychologist, 49(8), 725–747.

19. Ericsson, K. A., & Simon, H. A. (1980). Verbal reports as data. Psychological Review, 87(3), 215–

251.

20. Ericsson, K. A., & Simon, H. A. (1984). Protocol analysis: Verbal reports as data. Cambridge, MA:

The MIT press.

21. Fidler, E. J. (1983). The reliability and validity of concurrent, retrospective, and interpretative verbal

reports: An experimental study. In P. H. O. Svenson & A. Vari (Eds.), Analysing and aiding decisionprocesses (pp. 429–440). Amsterdam: North-Holland.

22. Fisher, C. D., & Ashkanasy, N. M. (2000). The emerging role of emotions in work life: An intro-

duction. Journal of Organizational Behavior, 21, 123–129.

23. Fonteyn, M. E., Kuipers, B., & Grobe, S. J. (1993). A description of Think Aloud method and

protocol analysis. Qualitative Health Research, 3(4), 430–441.

24. Gary, M. S., & Wood, R. E. (2007). Mental models, decision making and performance in complex,

dynamic environments. Social Science Research Network (SSRN). Working Paper Series.

25. George, J. M. (2000). Emotions and leadership: The role of emotional intelligence. Human Relations,53, 1027–1041.

26. Glaser, R., & Bassok, M. (1989). Learning theory and the study of instruction. Annual Review ofPsychology, 40, 631–666.

27. Goldsmith, T. E., & Davenport, D. M. (1990). Assessing structural similarity of graphs. In R. W.

Schvaneveldt (Ed.), Pathfinder associative networks: Studies in knowledge organization (pp. 75–87).

Westport, CT: Ablex Publishing.

28. Goldstone, R. L., & Sakamoto, Y. (2003). The transfer of abstract principles governing complex

adaptive systems. Cognitive Psychology, 46, 414–466.

29. Gott, S. P. (1989). Apprenticeship instruction for real-world tasks: The coordination of procedures,

mental models, and strategies. Review of Research in Education, 15, 97–169.

30. Gross, J. J. (1998). Antecedent- and response-focused emotion regulation: Divergent consequences

for experience, expression, and physiology. Journal of Personality and Social Psychology, 74(1),

224–237.


123

31. Helton-Fauth, W. B., Gaddis, B., Scott, G., Mumford, M. D., Devenport, L. D., Connelly, S., et al.

(2003). A new approach to assessing ethical conduct in scientific work. Accountability in Research,10, 205–228.

32. Hmelo-Silver, C. E., & Pfeffer, M. G. (2004). Comparing expert and novice understanding of a

complex system from the perspective of structures, behaviors and functions. Cognitive Science, 28,

127–138.

33. Howard, P. J., & Howard, J. M. (2001). The owner’s manual for personality at work. Austin, TX:

Bard Press.

34. Jacobson, M. J. (2001). Problem solving, cognition, and complex systems: Differences between

experts and novices. Complexity, 6(2), 1–9.

35. Johnson-Laird, P. N. (1983). Mental models: Toward a cognitive science of language, inference, andconsciousness. Cambridge, MA: Harvard University Press.

36. Johnson-Laird, P. N. (2004). The history of mental models. In K. Manktelow & M. C. Chung (Eds.),

Psychology of reasoning: Theoretical and historical perspectives (pp. 179–212). Hove, Sussex:

Psychology Press.

37. Jonassen, D. H. (1997). Instructional design model for well-structured and ill-structured problem-

solving learning outcomes. Educational Technology: Research and Development, 45(1), 65–95.

38. Jones, N. L. (2007). A code of ethics for the life sciences. Science and Engineering Ethics, 13, 25–43.

39. Kazier, C., & Shore, B. M. (1995). Strategy flexibility in more and less competent students on

mathematical work problems. Creativity Research Journal, 8, 77–82.

40. Kennedy-Moore, E., & Watson, J. C. (2001). How and when does emotional expression help? Reviewof General Psychology, 5(3), 187–212.

41. Kirkpatrick, D. L. (1976). Evaluation of training. In R. L. Craig (Ed.), Training and developmenthandbook: A guide to human resource development (2nd ed., pp. 18–27). New York: McGraw-Hill.

42. Kirkpatrick, D. L. (1996). Great ideas revisited. Training and Development Journal, 50, 54–59.

43. Klein, G., Moon, B., & Hoffman, R. R. (2006). Making sense of sensemaking: Alternative per-

spectives. Available at http://doi.ieeecomputersociety.org/10.1109/MIS.2006.75.

44. Kligyte, V., Marcy, R. T., Sevier, S. T., Godfrey, E. S., & Mumford, M. D. (2008). A qualitative

approach to Responsible Conduct of Research (RCR) training development: Identification of meta-

cognitive strategies. Science and Engineering Ethics, 14(1), 3–31.

45. Kligyte, V., Marcy, R. T., Waples, E., Sevier, S. T., Godfrey, E. S., & Mumford, M. D. (2007).

Application of a sensemaking approach to ethics training in the physical sciences and engineering.

Science and Engineering Ethics. doi: 10.1007/s11948-007-9048-z.

46. Kohlberg, L. (1984). Essays on moral development: The psychology of moral development: Thenature and validity of moral stages. San Francisco: Harper & Row.

47. Kraiger, K., Ford, J. K., & Salas, E. (1993). Application of cognitive, skill-based, and affective

theories of learning outcomes to new methods of training evaluation. Journal of Applied Psychology,78, 311–328.

48. Larkin, J. H., McDermott, J., Simon, D. P., & Simon, H. A. (1980). Expert and novice performance in

solving physics problems. Cognitive Science, 12, 101–138.

49. Larsen, R. J. (2000). Toward a science of mood regulation. Psychological Inquiry, 11, 129–141.

50. Mason-Mason, S. D., & Tessmer, M. A. (2000). Expert systems as a mind tool to facilitate mental

model learning. Educational Technology Research and Development, 48(4), 43–62.

51. Minkes, A. L., Small, M. W., & Chatterjee, S. R. (1999). Leadership and business ethics: Does it

matter? Implications for management. Journal of Business Ethics, 20(4), 327–335.

52. Mumford, M. D. (2006). Pathways to outstanding leadership: A comparative analysis of charismatic,ideological, and pragmatic leaders. Mahweh, New Jersey: Erlbaum.

53. Mumford, M. D., Connelly, S., Brown, R. P., Murphy, S. T., Hill, J. H., Antes, A. A., Waples, E. P.,

& Devenport, L. D. (2008). A sensemaking approach to ethics training for scientists: Preliminary

evidence of training effectiveness. Ethics and Behavior, 18(2).

54. Mumford, M. D., Devenport, L. D., Brown, R. P., Connelly, M. S., Murphy, S. T., Hill, J. H., et al.

(2006). Validation of ethical decision-making measures: Internal and external validity. Ethics andBehavior, 16(4), 319–345.

55. Plemmons, D. K., Brody, S. A., & Kalichman, M. W. (2006). Student perceptions of the effectiveness

of education in the responsible conduct of research. Science and Engineering Ethics, 12, 571–582.

56. Pinkus, R. L. B., Shuman, L. J., Hummon, N. P., & Wolf, H. (1997). Engineering ethics, balancingcost, schedule and risk—lessons learned from the space shuttle. Cambridge: Cambridge University

Press.

Mental Models 471

123

http://doi.ieeecomputersociety.org/10.1109/MIS.2006.75

http://dx.doi.org/10.1007/s11948-007-9048-z

57. Rest, J. R. (1986). An overview of the psychology of morality. In J. R. Rest (Ed.), Moral developmentand behavior: Theory, research, and social issues (pp. 133–175). New York: Praeger.

58. Rouse, W. B., & Morris, N. M. (1986). On looking into the black box: Prospects and limits in the

search for mental models. Psychological Bulletin, 100(3), 349–363.

59. Schvaneveldt, R. W. (1990). Pathfinder associative networks: Studies in knowledge organization.

Westport, CT: Ablex Publishing.

60. Scott, G. M., Leritz, L. E., & Mumford, M. D. (2004). The effectiveness of creativity training: A

meta-analysis. Creativity Research Journal, 16, 361–388.

61. Sims, R. R. (2000). Changing an organization’s culture under new leadership. Journal of BusinessEthics, 19, 393–403.

62. Sims, R. L., & Keon, T. L. (1999). Determinants of ethical decision-making: The relationship of

perceived organizational environment. Journal of Business Ethics, 19, 393–404.

63. Stout, R. J., Salas, E., & Kraiger, K. (1997). Role of trainee knowledge structures in aviation team

environments. International Journal of Aviation Psychology, 7(3), 235–250.

64. Tereskerz, P. M. (2003). Research accountability and financial conflicts of interest in industry:

Sponsored clinical research. Accountability in Research, 10, 137–158.

65. The National Academies Press. (2002). Integrity in scientific research: Creating an environment thatpromotes responsible conduct. Available at http://books.nap.edu/catalog.php?record_id=10430.

66. Titus, S. (2004). Descriptive study: Research integrity measures used in biomedical laboratories.

Paper presented at the 3rd Research on Research Integrity Conference, San Diego, CA.

67. Walczyk, J. J., Roper, K. S., Seemann, E., & Humphrey, A. M. (2003). Cognitive mechanisms

underlying lying to questions: Response time as a cue to deception. Applied Cognitive Psychology,17, 755–774.

68. Weick, K. (1995). Sensemaking in organizations. Beverly Hills, CA: Sage.

69. Wilson, T. D., & Gilbert, D. T. (2005). Affective forecasting: Knowing what to want. CurrentDirections in Psychological Science, 14, 131–134.


123

http://books.nap.edu/catalog.php?record_id=10430

meagan e. brock Æ andrew vert Æ vykinta kligyte...

Documents