Maurice A. Deane School of Law at Hofstra UniversityScholarly Commons at Hofstra Law

Hofstra Law Faculty Scholarship

Fall 2012

Performance Isn't Everything: The Importance ofConceptual Competence in Outcome Assessmentof Experimental LearningStefan H. KriegerMaurice A. Deane School of Law at Hofstra University

Serge A. Martinez

Follow this and additional works at: https://scholarlycommons.law.hofstra.edu/faculty_scholarship

This Article is brought to you for free and open access by Scholarly Commons at Hofstra Law. It has been accepted for inclusion in Hofstra Law FacultyScholarship by an authorized administrator of Scholarly Commons at Hofstra Law. For more information, please contact lawcls@hofstra.edu.

Recommended CitationStefan H. Krieger and Serge A. Martinez, Performance Isn't Everything: The Importance of Conceptual Competence in Outcome Assessmentof Experimental Learning, 19 Clinical L. Rev. 251 (2012-2013)Available at: https://scholarlycommons.law.hofstra.edu/faculty_scholarship/286

PERFORMANCE ISN'T EVERYTHING:THE IMPORTANCE OF CONCEPTUAL

COMPETENCE IN OUTCOME ASSESSMENTOF EXPERIENTIAL LEARNING

STEFAN H. KRIEGER*

SERGE A. MARTINEZ**

This article scrutinizes-and ultimately rejects-the recommen-dations of the Carnegie Report for outcome assessments in experien-tial legal education. The Carnegie Report argues that practicaleducation should focus on teaching students to mimic the actions ofexpert lawyers by encouraging students to follow expert protocols,procedures, rules, and checklists that can enable them to deal effec-tively with lawyering situations. To the contrary, however, an exten-sive body of cognitive science and neuroscience research on thedevelopment of expertise questions the theoretical underpinnings ofthe Carnegie Report and suggests that educators should focus not onwhat experts are doing but on what they are thinking as they deal witha lawyering problem. Therefore, this article proposes that outcomeassessments in experiential legal education should focus primarily onstudents learning to reason in practice, and presents an experimentalassessment technique designed to evaluate student reasoning in prac-tice. Ultimately, the Carnegie critique developed in this article has sig-nificant implications for the drafting of the final ABA standards onoutcome assessments which are likely to reach the final stage of theadoption process in late 2012 or early 2013.

INTRODUCTION

On January 15, 2009, US Airways Flight 1549 flew through a flock

* Professor of Law and Director Emeritus of Clinical Programs, Maurice A. Deane

School of Law at Hofstra.** Clinical Professor of Law, Maurice A. Deane School of Law at Hofstra. A version of

this paper was presented in May 2010 at a meeting of the Section on Clinical Legal Educa-tion of the Association of American Law Schools in Baltimore, Maryland, and we wish tothank the participants at that conference for their comments and suggestions on this re-search. We also wish to thank the clinical faculty at the Maurice A. Deane School of Law atHofstra for their comments on this manuscript, Theo Liebmann for his assistance on thestudent interviews, all of the students who agreed to participate in the interviews, IanWeinstein and Mary Lynch for their assistance in understanding the procedures of theAmerican Bar Association, Dr. Vimla Patel for her comments and insights on the cognitivescience of expertise, and our research assistants Joseph Losos and Jessica Chiavara. Finally,we would like to thank the Maurice A. Deane School of Law at Hofstra for providing theresearch support that made this Article possible.

CLINICAL LAW REVIEW

of geese on takeoff from LaGuardia Airport, sucking birds into bothof its engines. The engines immediately went dead, turning the planeinto a low-flying, extremely heavy glider.'

In an emergency situation, US Airways procedures call for thefirst officer to take the controls, so the captain's decision making isunencumbered by the distraction of actually trying to fly the plane.2

On Flight 1549, however, Captain Chesley "Sully" Sullenberger as-sessed the situation and broke with protocol, taking the controls him-self. Captain Sullenberger knew that he was the more experiencedpilot, he was looking out the side of the plane that faced all the impor-tant local landmarks, and furthermore he was not up-to-date withemergency protocols for that particular aircraft. Sully also knew thatthe first officer, Jeff Skiles, had recently been through training for theaircraft and would therefore more easily find the appropriate checkl-ists and protocols in the bulky emergency handbook.3

Unaware that the engines had been damaged beyond repair bythe goose strikes, First Officer Skiles quickly initiated the three-pageprotocol for the loss of both engines. 4 While Skiles was struggling torestart the engines per standard procedure, Sully was considering theoptions for bringing the plane down safely. He saw that TeterboroAirport, in New Jersey, was too far away to reach at their airspeed andrate of descent. He also calculated that they did not have the ability toturn around and return safely to LaGuardia.5 Outside the aircraft, itwas a calm, clear day. The Hudson River, while dangerously cold, wasalso home to a great deal of river traffic, which meant that rescuerswould be able to reach the plane quickly. Concluding that the safestcourse of action would be to land in the river, Sully guided the planeto a smooth landing, preserving the lives of everyone aboard. Thewhole event unfolded in just a few minutes, so quickly that Skiles didnot make it past the first page of the three-page checklist for enginefailure by the time they hit the water.6

Sully obviously demonstrated consummate flying skills by pilot-ing the powerless plane through a gentle descent and then a controlled

1 Robert D. McFadden, Pilot is Hailed after Jetliner's Icy Plunge, N.Y. Times Al (Jan-uary 16, 2009).

2 Sully's Tale, AIR & SPACE MAG. (Feb. 18, 2009), http://www.airspacemag.com/flight-

today/Sullys-Tale.html.3 Id.4 National Transportation Safety Board, Loss of Thrust in Both Engines After Encoun-

tering a Flock of Birds and Subsequent Ditching on the Hudson River, US Airways Flight1549, Airbus A320-214, N106US, Weehawken, New Jersey, January 15, 2009, 82.

5 Sully's Tale, supra note 2.6 The protocol was designed for high-altitude engine failure, not low-altitude failure.

The solution proposed by the NTSB was to create additional checklists for low-altitudeemergencies. National Transportation Safety Board, supra note 4, at 52.

[Vol. 19:251

Performance Isn't Everything

landing in the middle of the Hudson River. But focusing on Sully'sflying performance can obscure an important element of the incident:the reasoning underlying his performance. To successfully land theplane, Sully ignored the US Airways protocols for command and didnot rely on the procedure for restarting the engines. If Sully and Skileshad simply performed the procedures they had been trained to follow,perhaps things would have ended much less successfully. Instead,Sully quickly created a new plan to account for the specifics of thesituation. His quick thinking led to the actions that saved the lives ofthe crew and passengers.

Captain Sullenberger's decision making on Flight 1549 highlightsthe importance of the cognitive process in expertise. Experts do notsimply perform well. They must also reason well.

The lessons about expertise that this story reveals should concernthe American Bar Association ("ABA"). The ABA has recently sig-naled its intention to make a major change in its accreditation stan-dards for law schools by moving away from the current focus on inputmeasures-what students are taught in law school 7-to a focus onlearning outcomes-what students learn in law school.8 In 2007,amidst a growing chorus calling for the incorporation of outcomemeasures into American legal education,9 the ABA's Council of the

7 The current Standard 302(a) mandates a law school to "require that each studentreceive substantial instruction in:

1. the substantive law generally regarded as necessary to effective and responsibleparticipation in the legal profession;

2. legal analysis and reasoning, legal research, problem solving, and oralcommunication;

3. writing in a legal context, including at least one rigorous writing experience inthe first year and at least one additional rigorous writing experience after thefirst year;

4. other professional skills generally regarded as necessary for effective and respon-sible participation in the legal profession; and

5. the history, goals, structure, values, rules and responsibilities of the legal profes-sion and its members."

ABA Sec. on Legal Educ. & Admission to the Bar, 2012-2013 Standards and Rules ofProcedure for Approval of Law Schools (2012) [hereinafter Standards].

8 ABA Sec. on Legal Educ. & Admission to the Bar, Report of the Outcome MeasuresCommittee 1 (July 27, 2008) [hereinafter Outcomes Report].

9 See, e.g., GREGORY S. MUNRO, OUTCOMES ASSESSMENT FOR LAW SCHOOLS 50

(2000) ("If a law school is to be effective as an educational institution, it needs to be guidedby student outcomes-a statement of the knowledge, abilities, and attributes its studentsshould derive from their legal education."); Roy STUCKEY & OTHERS, BEST PRACICESFOR LEGAL EDUCATION: A VISION AND A RoADMAP 42 (2007) (urging law schools todescribe "what the school's students will be able to do after graduating and how they willdo it in addition to what they will know"); WILLIAM M. SULLIVAN, ANNE COLBY, JUDITHWELCH WEGNER, LLOYD BOND & LEE S. SHULMAN, EDUCATING LAWYERS: PREPARA-

TION FOR THE PROFESSION OF LAW 180-2 (2007) (endorsing and commending Munro'sexhortations for "greater institutional intentionality"-including the use of outcome mea-sures-in student learning); John 0. Sonsteng, Donna Ward, Colleen Bruce and Michael

Fall 20121

CLINICAL LAW REVIEW

Section of Legal Education and Admissions to the Bar (the "Coun-cil") created a "Special Committee on Output Measures" (the "Out-come Subcommittee") and charged it with determining "whether andhow [the Council] can use output measures, other than bar passageand job placement, in the accreditation process." 10 After conducting athorough review of legal education around the world, professionsother than law, and American legal scholarship about outcomes," theOutcome Subcommittee's report recommended a re-examination andreframing of the Standards for Approval of Law Schools (the "Stan-dards") "to reduce their reliance on input measures and instead adopta greater and more overt reliance on outcome measures.' 2

Responding to the Outcome Subcommittee's report, the ABAStandards Review Committee has proposed new approval criteria (the"Proposed Standards") that heavily incorporate outcome measures.The latest version of the Proposed Standards 13 would require each lawschool to articulate the outcomes it will seek to give its students.1 4 Inconjunction with selecting outcomes, law schools would be required todesign a curriculum to create those outcomes, 15 assess student learn-ing through a variety of methods, give students meaningful feed-back, 16 and regularly and continuously assess their success atachieving their student learning outcome goals.17

Although the proposed changes to the accreditation requirements

Peterson, A Legal Education Renaissance: A Practical Approach for the Twenty-First Cen-tury, 34 WM. MITCHELL L. REV. 303, 456 (2007) (calling on law schools to incorporate intotheir curricula "general and specific learning objectives that provide students an opportu-nity to demonstrate a predetermined competency level.").

10 ABA Sec. on Legal Educ. & Admission to the Bar, Interim Report of the Outcome

Measures Committee 1 (May 12, 2008).11 Outcomes Report, supra note 8, at 1.12 Id.13 See ABA Sec. on Legal Educ. & Admission to the Bar, Draft Report for the April

2012 Meeting of the ABA Standards Review Committee (2012) [hereinafter Draft Report].As of this writing in September 2012, the Draft Report is in the final stages of considera-tion by the Standards Review Committee. At its July 2012 meeting, the Standards ReviewCommittee conducted its scheduled second consideration of the Proposed Standards with-out making changes to the relevant sections. The Proposed Standards are scheduled for athird and tentatively final consideration at its November 2012 meeting. After the final re-view, the Committee will send the Draft Report to the Council. The Council will thenreview the Draft Report and conduct a public comment process, including holding publichearings. If the Council decides to revise or amend the Standards, the ABA House ofDelegates must review the changes. The House of Delegates will then either concur in therevisions or send them back to the Council for additional consideration, but the Councilhas the final authority with respect to adoption of revisions. Standards, supra note 7, at vi-vii, Standard 803, Internal Operating Practices 9, 11.

14 Draft Report, supra note 13, at Standard 302(a).15 Id. at Standard 304(a).16 Id. at Standard 305.17 Id. at Standard 306(a).

[Vol. 19:251

Performance Isn't Everything

would generally afford law schools flexibility in determining their ownlearning outcomes, they also include several required outcome goals.One mandatory outcome measure that schools would be required toadopt is "competency as an entry-level practitioner" in "a depth inand breadth of" professional skills.18 This requirement would re-present a significant change for experiential legal education: at pre-sent, law schools are required only to provide "substantial instruction"in professional skills, with no competency requirement. 19

Requiring law schools to focus on instruction for competency inskills is a laudable goal. However, neither the proposed standards northe proposed interpretations attempt to define the nature of compe-tency in skills. To effectively apply Proposed Standard 302(b), theABA will first need to articulate a definition of competency in prac-tice: is it simply the ability to perform certain skills, or, as in Sully'scase, the ability to effectively reason through problems in practice?

Much of the recent focus by the ABA-and legal scholarship-on defining learning outcomes has been influenced by the CarnegieFoundation for the Advancement of Teaching's 2007 monograph, Ed-ucating Lawyers: Preparation for the Practice of Law20 (the "CarnegieReport" or the "Report" or "Carnegie"), which argues that lawschools should make a greater effort to prepare students for the pro-fessional practice of law.21 Carnegie draws heavily on the work of Hu-bert Dreyfus and Stuart Dreyfus2 2 to articulate a theory of expertlegal practice based on performance. This theory leads to a model fortraining law students in lawyering skills that stresses the importance oflearning the concepts and procedures of the legal profession in a waythat is structured for performance. By focusing on performance as themeasure of competence, Carnegie emphasizes the importance to cur-riculum design and outcome assessment of protocols and techniquesfor performing particular lawyering tasks.

In reaching its conclusions, Carnegie had the benefit of anotherreport on legal education reform, Best Practices for Legal Education2 3

("Best Practices"). Best Practices was published in 2007 by the Clinical

18 Id. at Standard 302(b). Although the Proposed Standards are silent about which pro-fessional skills are required to give sufficient depth and breadth, a proposed Interpretationindicates that "interviewing, counseling, negotiation, fact development and analysis, con-flict resolution, organization and management of legal work, collaboration, cultural compe-tency, and self-evaluation are among the professional skills that could fulfill" this ProposedStandard. Id. at Interpretation 302-2.

19 Standards, supra note 7, at Standard 302.20 SULLIVAN ET AL., supra note 9.21 See infra notes 28-40 and accompanying text.22 SULLIVAN ET AL., supra note 9, at 116-19. See infra notes 66-79 and accompanying

text.23 STUCKEY ET AL., supra note 9.

Fall 20121

CLINICAL LAW REVIEW

Legal Education Association, a national organization of clinical lawprofessors, with the purpose of providing "a vision of what legal edu-cation might become if legal educators step back and consider howthey can most effectively prepare students for practice. ' 24 Like Car-negie, Best Practices focuses on performance as a measure of compe-tency and suggests varied techniques for teaching law students thetools of the trade.25

However, current research into expert practice indicates that theability to follow protocols is not a fully accurate reflection of compe-tence.26 As is apparent from the landing of Flight 1549, experts do notsimply rely on standardized protocols. Sully quickly reasoned throughthe problem and crafted an appropriate solution. His quick reasoningin the problem-solving process highlights the centrality of cognition tohis expertise as a pilot.

Current research into expert practice indicates that reasoning isan important element of expertise not only in the work of piloting anairplane, but also in the practice of professional skills such as law-yering. Expertise in practice involves thinking, reasoning, and deci-sion-making processes. 27 Therefore, this research indicates that toteach law students to handle their clients' problems expertly, learningobjectives should include the development of effective reasoningstrategies to handle different and varied problems, not just the abilityto perform in prescribed ways.

Given the influence of Carnegie, a strong possibility exists thatthe ABA-or a law school following the lead of Carnegie-will un-reflectively define "competency of an entry-level practitioner" prima-rily in terms of the ability to perform certain lawyering tasks ratherthan the capacity to reason in practice. The ramifications of such adefinition for legal education would be quite significant. If Carnegie'sapproach to learning outcomes prevails, all aspects of experiential le-gal education-from curriculum development to individual course de-sign to outcome assessment-will be focused on training students forstandard performance, not for reasoning in practice.

In this Article, we respond to the ABA's call for increased focuson learning outcomes by critically examining the nature of "compe-tency in practice." Relying on cognitive science research on the quali-ties of expertise, we, unlike Carnegie, propose that outcomes inexperiential legal education should be focused primarily on studentslearning to reason in practice. In Part I we describe the approach to

24 Id. at 1.25 See infra notes 57-61 and accompanying text.26 See, e.g., infra notes 95-131 and accompanying text.27 See infra notes 80-131 and accompanying text.

[Vol. 19:251

Performance Isn't Everything

legal education, especially the theories of experiential learning and ex-pertise, underpinning the Carnegie Report. In Part II we present find-ings on expert practice from current cognitive science research andcontrast them with the performance-based focus of Carnegie. In PartIII we address the different measures of outcome assessment that fol-low from Carnegie's approach and from our approach. In Part IV, wedescribe and consider the applications of our own experiment in as-sessing reasoning in practice, which asks students to think aloud asthey reason through problems. Finally, in Part V, we provide recom-mendations to the ABA and to law schools in defining learning out-comes such that assessment methods can be tailored to measuring thedevelopment of professional expertise. We hope that this article willinfluence the outcome assessment conversation in a constructive way.

I. THE CARNEGIE REPORT

To describe its vision of professional competence, Carnegie iden-tifies what it calls "the three apprenticeships" of professional educa-tion.28 The first of these focuses on student knowledge and "the wayof thinking" of the legal profession. 29 Carnegie calls this the cognitiveapprenticeship. 30 The second is the practical apprenticeship, which isconcerned with "the forms of expert practice shared by competentpractitioners."' 31 The third is the ethical and social apprenticeship,which incorporates "the purposes and attitudes that are guided by thevalues for which the professional community is responsible. '32 Car-negie argues that law schools have focused overwhelmingly on thefirst apprenticeship, to the neglect of the second and third.

A. The Signature Pedagogy

Carnegie asserts that traditional legal education has been nar-rowly focused on the cognitive apprenticeship, through the almostuniversal use of the "case-dialogue" method of education, in whichprofessors lead students through discussions of appellate cases totease out legal analysis. Carnegie calls the case-dialogue method the"signature pedagogy" of legal education, its distinct method for in-ducting new members into the profession.33 Assessing the success of

28 SULLIVAN ET AL., supra note 9, at 28.29 Id. at 28.30 Id. at 60.31 Id. at 28.32 Id.33 Id. at 23. The signature pedagogies of other professions described by Carnegie in-

clude "the varieties of design and performance studio in engineering and architecture, bed-side teaching and clinical rounds in medicine and nursing, the interpretation of texts andinstruction in preaching in seminaries." Id.

Fall 2012]

CLINICAL LAW REVIEW

law schools' signature pedagogy in "preparing students for the com-plex demands of professional work-to think, to perform, and to con-duct themselves like professionals,' 34 Carnegie finds it wanting. Whileendorsing the case-dialogue method in the first year of law school as amethod of initiating students into the study of law, Carnegie criticizeslaw schools' overemphasis on legal analysis through continued use ofthe case-dialogue method with diminishing returns in the second andthird years.35

Carnegie argues that, although the signature pedagogy rapidly so-cializes students into an academic approach to studying law,36 and ad-equately teaches students to engage in legal analysis and to "think likea lawyer, '37 it has several consequences. Rather than preparing stu-dents to become lawyers, the signature pedagogy works to "prolongand reinforce the habits of thinking like a student rather than an ap-prentice practitioner. '38 Carnegie argues that a narrow focus on legalanalysis has a "corrosive effect" 39 on professional development be-cause it marginalizes practical and ethical aspects of lawyering. Toremedy the current imbalance in legal education, the Carnegie Reportargues that legal education should establish a new signature pedagogy,one that incorporates training in cognitive, practical, and ethical as-pects of law through "educational experiences oriented toward prepa-ration for practice. '40

B. A Model for Learning from Practice

An important aspect of Carnegie's model is its proposal for in-creased emphasis on the practical apprenticeship in legal education. Inthe traditional legal apprenticeship, as Carnegie describes it, a signifi-cant amount of learning came informally from observation and imita-tion of an expert.41 The expert modeled performance in such a waythat the learner could eventually imitate the performance, while theexpert provided feedback to guide the learner in making the activityhis or her own. 42

Carnegie acknowledges that the traditional apprenticeship, con-sisting of an extended, close, individual relationship designed to offerpractical training, is not realistic in legal education today. Nonethe-

34 Id. at 27.35 Id. at 77.36 Id. at 186.37 Id.38 Id. at 188.39 Id. at 77.40 Id. at 88.41 Id. at 26-27, 98.42 Id. at 26.

[Vol. 19:251

Performance Isn't Everything

less, Carnegie purports to reformulate the apprenticeship model to in-corporate the benefits of such a relationship into a moderneducational setting.43 In its reformulation, Carnegie urges that thecentral elements of expert performance should be passed on in a moreformal fashion. They should be studied, distilled, simplified, andtaught to novices "in the form of rules, procedures, protocols and or-ganizing metaphors for approaching situations or problems."'44 Car-negie refers to these devices as "scaffolds. ' 45 A scaffold could be, forexample, a particular interviewing procedure, a protocol for problemsolving, a technique for negotiating a deal, or a method for drafting acontract. With its focus on techniques and procedures, Carnegie'smodel is based on studying the performance of experts and teachingstudents to imitate that performance.

In the Carnegie model, increased competence comes as a studentgradually accumulates a "toolkit of well-founded procedures" 46 in dif-ferent areas of legal practice. Within this performance framework,"the prime learning task of the novice in law is to achieve a basicacquaintance with the common techniques of the lawyer's craft."'47 Inthis approach, student reasoning takes a backseat to learning experttechniques. In fact, Carnegie argues that reasoning and attention tocontext by novice learners is unhelpful; instead, it posits that studentsshould be taught to "recognize certain well-defined elements of thesituation and apply precise and formal rules to these elements, regard-less of what else is happening. '' 48 To Carnegie, the goal of the practicalapprenticeship should be that students accumulate experience bypracticing in accordance with the rules.49 With this learning goal, thefocus of educators tasked with teaching novice practitioners shouldtherefore be on particular "scaffolds," the techniques, rules, and pro-cedures of expert practice.

C. Using Scaffolds in Legal Education

The two examples below illustrate the types and uses of scaffoldsin a performance-based model of practical legal education:

1. The T-Funnel Interview

As an example of the concept of a scaffold for law students, Car-

43 Id. at 100.44 Id. at 99.45 Id. at 98.46 Id. at 103.47 Id. at 117.48 Id.49 Id.

Fall 20121

CLINICAL LAW REVIEW

negie points to the textbook Lawyers as Counselors," which providesa framework for legal interviewing and counseling developed by sev-eral law professors. The text offers step-by-step methods, identifying aset of "practical guides and techniques"'5 for novice interviewers andcounselors.

One of Lawyers as Counselors' techniques that is touted by Car-negie is "T-funnel" interviewing. Such an interview "is a pattern ofinformation-seeking" 52 with two elements: open questions-the upperpart of the T-induce clients to think freely about the problem andelicit information from the clients' perspective. Closed questions-thelower part of the T-are used to focus narrowly on gathering informa-tion related to specific legal aspects of the client's problem. The pre-mise underlying this technique is that "thorough information-gathering rests on a combination of open and closed questions" 53 in aparticular sequence: start with an open question and gradually narrowthe questioning.

Expanding on the basic T-funnel concept, Lawyers as Counselorsgives techniques for using the T-funnel in specific interview situations,such as gathering information about a specific event or a particulartopic. Lawyers as Counselors also provides techniques for resolvingproblems frequently encountered by beginners, such as avoiding get-ting sidetracked, "cycling" through multiple T-funnels to capture asmuch information as possible, and facilitating recollection in forgetfulclients.54 The scaffold gives a novice a variety of techniques to use indifferent interviewing situations.

For a teacher, using a scaffold in Carnegie's model takes on adistinct form: the scaffold is introduced to students, who are givenmultiple opportunities to practice using it, and to receive feedbackfrom the instructor on how they use it. In this way, scaffolds can helpguide students' "assimilation of more skillful practice. ' 55 By practicingthe application of the T-funnel technique to interviewing, a studentcan accumulate enough experience with this scaffold to allow her tomaster its application. The T-funnel technique will then become partof the "toolkit" that, according to Carnegie, the student must assem-ble to achieve "skilled performance"-the goal of the practicalapprenticeship.5 6

50 DAVID A. BINDER, PAUL BRUCE BERGMAN, SUSAN C. PRICE & PAUL R. TREM-

BLAY, LAWYERS AS COUNSELORS: A CLIENT-CENTERED APPROACH (2d ed. 2004).51 SULLIVAN ET AL., supra note 9, at 102.52 DAVID A. BINDER ET AL., supra note 50, at 167.53 Id.54 Id. at 167-78.55 SULLIVAN ET AL., supra note 9, at 101-2.56 Id. at 124.

[Vol. 19:251

Performance Isn't Everything

2. The PrOACT Methodology for Problem Solving

A second example of a scaffold for novices comes from Best Prac-tices.57 As an example of an effective methodology for helping novicesdevelop skills for problem solving,58 Best Practices presents thePrOACT method of decision making, with its broad applicability andease of recall. 59 The acronym PrOACT is created from these fivesteps: Problem, Objectives, Alternatives, Consequences, Tradeoffs. 60

In this methodology, the lawyer 1) defines the problem to be solved,2) determines the desired objective(s), 3) identifies alternative coursesof action, 4) evaluates the consequences of each alternative, and 5)structures how to make tradeoffs among objectives and alternatives asa prelude to making the decision.61 The scaffold is a straightforwardtechnique for structuring the task of problem solving.

D. The Impact of Carnegie

It would be difficult to overstate the influence of the CarnegieReport on recent scholarship about legal education. The Report hasbeen cited as an authority on legal education in countless scholarlyarticles across a broad spectrum of pedagogical issues.62 Several con-

57 See STUCKEY ET AL., supra note 9, at 64.

58 Id.59 Id.6 Id.61 Id.62 See, e.g., Charlotte S. Alexander, Learning to Be Lawyers: Professional Identity and

the Law School Curriculum, MD. L. REv. 465 (2011) (using Carnegie as a "starting point"for integrating professionalism into curriculum); Kathleen M. Burch and Chara FisherJackson, Creating the Perfect Storm: How Partnering with the ACLU Integrates the CarnegieReport's Three Apprenticeships, 3 J. MARSHALL L.J. 51 (2009) (describing a course de-signed explicitly to integrate Carnegie's recommendations); Kate Nace Day & Russell G.Murphy, "Just Trying to Be Human in This Place": Storytelling and Film in the First YearLaw School Classroom, 39 STETSON L. REV. 247 (2009) (describing use of storytelling interms of Carnegie's critiques of legal education); Mark L. Jones, Fundamental Dimensionsof Law and Legal Education: Perspectives on Curriculum Reform, Mercer Law School'sWoodruff Curriculum and..."Perspectives," 63 MERCER L. REv. 975 (2012) (relying onCarnegie to support argument for requiring "perspectives" courses); Madeline June Kass,Educating the Next Generation of Environmental Lawyers, 25 NAT. RESOURCES & ENV'T52 (2010) (describing efforts of environmental law teachers to respond to Carnegie); Har-riet N. Katz, Evaluating the Skills Curriculum: Challenges and Opportunities for LawSchools, 59 MERCER L. REV. 909 (2008) (suggesting ways to change skills curriculum inlight of Carnegie's goals for skills education); Patricia Grande Montana, Lessons from theCarnegie and Best Practices Reports: A Look at St. John's University School of Law's StreetLaw Program as a Model for Teaching Professional Skills, 11 T.M. COOLEY J. PRAC. &CLINICAL L. 97 (2008) (citing conformity with Carnegie's suggestions as support for effec-tiveness of Street Law course); Ira Steven Nathenson, Best Practices for the Law of theHorse: Teaching Cyberlaw and Illuminating Law through Online Simulations, 28 SANTA

CLARA COMPUTER & HIGH TECH. L.J. 657 (2012) (proposing online simulations as a re-sponse to Carnegie's critiques of case-analysis pedagogy); Brent E. Newton, PreachingWhat They Don't Practice: Why Law Faculties' Preoccupation with Impractical Scholarship

Fall 20121

CLINICAL LAW REVIEW

ferences have focused on Carnegie. 63 Law schools have cited the Re-port as inspiration for curricular design.64 Indeed, the ABA OutcomesSubcommittee acknowledged Carnegie as a source of "significant gui-dance ' 65 in its study of outcome measures in legal education.

In light of the potential impact of the Carnegie Report on selec-tion and definition of outcomes in the wake of the ABA's proposedreformulation and application of its Standards, it is important to scru-tinize Carnegie's recommendations rigorously rather than acceptingthem uncritically. This scrutiny benefits from an understanding of thetheoretical origins of the expert performance model adopted byCarnegie.

II. ROLE OF COGNITION IN EXPERT PERFORMANCE

A. Dreyfus Theory of Expertise

The crucial problem underlying Carnegie's focus on performanceassessment is that it does not rest on a sound theoretical or empiricalfoundation. Cognitive science findings challenge the expertise theorythat Carnegie employs and suggest a different approach to the issue.This alternative approach highlights the cognitive attributes of expertperformance rather than the performance itself. Such a difference in

and Devaluation of Practical Competencies Obstruct Reform in the Legal Academy, 62 S.C.L. REV. 105 (2010) (citing Carnegie's recommendations in support of radical restructuringof law faculty); Sean M. O'Connor, Teaching IP From an Entrepreneurial Counseling andTransactional Perspective, 52 ST. Louis U. L.J. 877 (2008) (citing Carnegie's critique ofcase method in support of experiential teaching of intellectual property law); John 0. Son-steng, Donna Ward, Colleen Bruce and Michael Peterson, supra note 9 (citing Carnegie assupport for the need for reform of entire legal education system); Emily Zimmerman, AnInterdisciplinary Framework for Understanding and Cultivating Law Student Enthusiasm,58 DEPAUL L. REV. 851 (2009) (citing Carnegie for evidence of "lack of enthusiasm" oflaw students).

63 See International Conference on the Future of Legal Education http://law.gsu.edu/Fu-tureOfLegalEducationConference/ (indicating that the 2008 conference would take theCarnegie Report as its "point of departure" for exploring the future of legal education);Legal Education Reform after Carnegie: Bringing Law-in-Action into the Law School Class-room http://www.law.wisc.edu/ilsl20l0legaleducationconf/homepage.html (describing howthe 2010 conference took its "impetus" partially from the publication of Carnegie); SharonL. Beckman and Paul R. Tremblay, The Way to Carnegie: Practice, Practice, Practice-Pedagogy, Social Justice and Cost in Legal Education: Foreword, 32 B.C. J. L. & Soc. JUST.215 (2012) (describing Fall 2011 conference focused on Carnegie's emphasis on preparinglaw students for practice).

64 Erwin Chemerinsky, the Founding Dean of the University of California IrvineSchool of Law, designed the skills-intensive curriculum of the new law school "with theCarnegie Report's recommendations in mind." Drew Coursin, Acting Like Lawyers, 2010Wis. L. REV. 1461, 1473 (2010). See also The Third Year in Detail, http://law.wlu.edu/thirdyear/page.asp?pageid=651 (describing Carnegie as one of the sources of insight thatwas particularly influential in Washington & Lee University Law School's development ofan entirely experiential third-year curriculum beginning in 2008).

65 Outcomes Report, supra note 8, at 5-6.

[Vol. 19:251

Performance Isn't Everything

emphasis is critical to the issue of law student assessment, because themethods for assessing performance diverge widely from those used toassess cognition.

Carnegie's theory of expertise is based entirely on the work oftwo brothers, Hubert and Stuart Dreyfus, trained respectively as aphilosopher and an engineer.66 The Dreyfus brothers posit that exper-tise is simply a matter of pattern recognition. 67 They argue, for exam-ple, that we are able to ride bikes because of prior experiencesoperating such vehicles, not because we are engaging in some kind ofcognitive process.68 As they observe, "No detached choice or delibera-tion occurs. It just happens, apparently because the proficient per-former has experienced similar situations in the past and memories ofthem trigger plans similar to those that worked in the past and antici-pation of events similar to those that occurred. '69 They call this abilityto use patterns without cognitive rules "holistic similarity recognition"and argue, "Normally, experts don't solve problems and don't makedecisions; they do what normally works. '70

With this theoretical outlook, the Dreyfuses assert that acquisi-tion of this kind of expert intuition requires the novice to learn proto-cols and strategies for identifying the facts and features of a particularsituation and performing in response to these facts.71 They also con-tend that people pass through at least five stages of qualitatively dif-ferent perceptions of their tasks (novice, advanced beginner,competence, proficiency, expertise) as they acquire a skill through in-

66 SULLIVAN ET AL., supra note 9, at 116-19. The only other source cited by the Report

on the issue of expertise is a book on nursing education which adopts the Dreyfus theory.PATRICIA BENNER, CHRISTINE A. TANNER & CATHERINE A. CHESLA, EXPERTISE IN

NURSING PRACTICE: CARING, CLINICAL JUDGMENT, AND ETHICS (1996). Like the Dreyfustheory, Benner's work has also been subject to serious criticism. See, e.g., Fernand Gobet& Philipe Chassy, Towards an Alternative to Benner's Theory of Expert Intuition in Nurs-ing: A Discussion Paper, 45 INr'L J. NURSING STUD. 129 (2008); Ian English, Intuition as aFunction of the Expert Nurse: A Critique of Benner's Novice to Expert Model, 18 J. AD-VANCED NURSING 387 (1993). The Carnegie Report is not alone in its wholesale adoptionof the Dreyfus model. In the medical profession, this model has also been accepted bymany physicians without any serious examination. Adolfo Pefia, The Dreyfus Model ofClinical Problem-Solving Skills Acquisition: A Critical Perspective, 15 MED. EDUC. ONLINE1, 2 (2010), http://journals.sfu.ca/coaction/index.php/meo/article/view/4846/pdf_3.

67 HUBERT L. DREYFUS & STUART E. DREYFUS, MIND OVER MACHINE: THE POWER

OF HUMAN INTUITION AND EXPERTISE IN THE ERA OF THE COMPUTER 28 (1986).68 Id. at 16. The Dreyfus model was derived from direct observations of experts such as

jet pilots and dancers. These types of experts are used to tackling well-structured problems,in contrast to practitioners in other domains, such as law and medicine, who often confrontill-structured problems. Pefia, supra note 66, at 6. For the differences between these twoproblem categories, see infra notes 108-09 and accompanying text.

69 DREYFUS & DREYFUS, supra note 67, at 28.70 Id. at 28, 30-31.71 See id. at 20-21.

Fall 20121

CLINICAL LAW REVIEW

struction and experience. 72 After progressing through these five stagesof accumulated experience, a learner develops the unconscious capac-ity to recognize new situations as similar to remembered ones and,through this process, ultimately becomes an expert. 73

These stages of development, the Dreyfuses claim, reflect anevolution from the abstract toward the concrete, "from the analyticbehavior of the detached subject, consciously decomposing his envi-ronment into recognizable elements, and following abstract rules, toinvolved skilled behavior based on accumulation of concrete exper-iences and the unconscious recognition of new situations similar towhole remembered ones. ' '74 In the Dreyfuses' own words, as studentsbecome experts, they act "arationally. ' '75 In other words, expert per-formance is essentially mindless. 76 Accordingly, under the Dreyfus ap-proach, expertise is not reflected as much in cognitive competencies asin mindless performances responding to perceived situations.

It should now be apparent why the Carnegie apprenticeshipmodel focuses on performance rather than on cognition. Consistentwith Dreyfus, Carnegie envisions that students should first learn rules,strategies, methods, and protocols to enable them to recognize pat-terns and perform in particular situations.77 Following Dreyfus, thereport contends that after numerous experiences, students progressthrough stages and acquire expertise. As they develop expertise, theystop relying on abstract rules and instead respond unconsciously tonew situations by perceiving similarities to whole remembered pastexperiences. 78

In sum, Carnegie contends that the focus of experiential educa-tion should be primarily on performance. The desired outcome of theeducational process is the student's "ability to judge that when a situa-tion shows a certain pattern of elements, it is time to draw a particularconclusion. . .[and to] act in a certain way to achieve the selectedgoal."'79 From this perspective, the student's action, rather than her

72 Id. at 19-35.73 Id. at 35.74 Id.75 Id. at 36.76 See Tim Thornton, Clinical Judgment, Expertise and Skilled Coping, 16 J. EVALUA-

TION IN CLINICAL PRAc. 284, 290 (2010). See also Pefia, supra note 66, at 2 (noting thatunder the Dreyfus model, "the brain is a secondary or spurious referent"). It is surprisingthat the clinical community, with its long tradition of encouraging the teaching of "reflec-tive practice," see, e.g., Richard K. Neumann, Donald Schon, the Reflective Practitioner,and the Comparative Failures of Legal Education, 6 CLIN. L. REV. 401 (2000), has been atthe forefront of urging the adoption of a report that is based on a theory which envisionsmindlessness as the goal of legal education.

77 SULLIVAN ET AL., supra note 9, at 99, 117.78 Id. at 116.79 Id. at 117 (emphasis added).

[Vol. 19:251

Performance Isn't Everything

reasoning process, has paramount importance.

B. Cognitive Science Critique of the Dreyfus Theory

Most cognitive scientists recognize the role that pattern recogni-tion plays in expert performance.80 Nonetheless, they reject the notionthat intuitive pattern recognition alone is determinative of expert per-formance. In fact, the Dreyfus theory conflicts with a number of em-pirical findings on expert decision making.

First, contrary to the Dreyfus theory, studies show that in manydomains requiring complex problem solving, expertise does not pro-duce a decrease in abstract thought and a concurrent increase in con-crete thinking. Indeed, in these domains, experts have been found toanalyze problems at a deeper, more abstract level than nonexperts. 81

Second, the existence of progressive stages in expert development isnot supported by the evidence.8 2 The Dreyfus theory suggests that themore experience individuals have in a particular area, the more intui-tion they acquire, and the more expertise they gain. Studies haveshown, however, that those individuals with extensive experience in afield do not necessarily perform better than people with less train-ing.83 In fact, the number of years of experience in a field is a poorpredictor of attained performance. 84 It is not unusual, for example, toobserve lawyers with 30 to 40 years of experience in a particular court-

80 See, e.g., Vimla L. Patel, David R. Kaufman & Josd F. Arocha, Steering through theMurky Waters of a Scientific Conflict: Situated and Symbolic Models of Clinical Cognition,7 ARTIFICIAL INTELLIGENCE IN MED. 413, 421 (1995) (in examining cognition in medicaltreatment, authors acknowledge that physicians' performance in routine situations oftennecessitates immediate nonanalytic responses and that "there are diagnostic tasks inperceptual domains such as dermatology and radiology in which a significant degree ofskilled performance... rel[ies] more on pattern recognition than deliberative reasoning.").

81 Fernand Gobet & Phillipe Chassy, Expertise and Intuition: A Tale of Three Theories,19 MIND & MACHINE 151, 154 (2009) (observing that in physics, experts solve problems at"a deep, abstract level, while novices perform at a superficial, concrete level.").

82 Id.83 K. Anders Ericsson, Deliberate Practice and Acquisition of Expert Performance: A

General Overview, 15 ACAD. EMERGENCY MED. 988, 989 (2008) [hereinafter Ericsson, De-liberate Practice]; K. Anders Ericsson, James Whyte & Paul Ward, Expert Performance inNursing: Reviewing Research on Expertise in Nursing within the Framework of the Expert-Performance Approach, 30 ADVANCES IN NURSING SCI. E58, E59 (2007); see generallyDennis J. Devine & Steve W.J. Kozlowski, Domain-Specific Knowledge and Task Charac-teristics in Decision Making, 64 ORGANIZATIONAL BEHAV. & HUM. DECISION PROCESSES

294, 295 (1995).84 Indeed, some studies even suggest that professional performance decreases in accu-

racy and consistency with length of experience. K. Anders Ericsson, The Influence of Expe-rience and Deliberate Practice on the Development of Superior Expert Performance, in THECAMBRIDGE HANDBOOK OF EXPERTISE AND EXPERT PERFORMANCE 685, 688 (K. AndersEricsson et al. 2006); Jean B6dard & Michelene T. H. Chi, Expertise in Auditing, 12 AUDIT-ING: J. PRAC. & THEORY 21, 29 (Supp. 1993) (suggesting performance of auditors de-creased in accuracy and consistency with length of experience).

Fall 20121

CLINICAL LAW REVIEW

house who are far from exhibiting expertise in their field. Finally,neuroscience evidence does not support the notion of holistic similar-ity recognition. This research demonstrates that complex decisionmaking entails a rich connection of different neural subsystems (ex-plicit and implicit) and an interplay between them.85

Apparently, then, some other factors must be involved in expertperformance beyond mere pattern recognition acquired through accu-mulated experiences. Contrary to the Dreyfus theory, most cognitivescientists contend that, in fact, experts do use particular cognitiveprocesses in their decision making.8 6 These processes are not alwaysconscious and deliberate. Rather, they reflect the interaction betweenimplicit knowledge-unconscious abstract representations that ex-perts have acquired through experience and their knowledge of thedomain-and explicit knowledge-explicit representations which areconscious and can be verbalized.8 7

Especially in domains like law and medicine, in which complexknowledge systems and symbolic representations play an integral role,more is involved in making decisions than mere pattern recognition ofprevious similar situations.8 8 For example, although she may not beaware of all the cognitive processes involved, when a physician evalu-ates a patient, she is conscious of the patient's characterization of hissymptoms, her own diagnosis of the problem, and her requests fortests.89 By overlooking the complex and rich interaction between im-plicit and explicit knowledge, the Dreyfus model fails to explain skillsthat are not just routines but instead involve complex tasks, such asfinding solutions to problems.90

Unlike driving a car or riding a bike, handling a legal problem inpractice requires more than intuition based on pattern recognition.Lawyers must juggle, for example, the substantive legal doctrine, theprocedural context, the particular facts of the situation, the client'sneeds, and the cultural and social context. The Dreyfus theory may

85 Pefia, supra note 66, at 5.86 See, e.g., Gobet & Chassy, supra note 66, at 132; John R. Anderson, Lynn Reder &

Herbert A. Simon, Situated Learning and Education, EDUC. RESEARCHER, May 1996, at 5;Patel et al., supra note 80, at 413.

87 See, e.g., Pefia, supra note 66, at 5-6; Patel et al., supra note 80, at 421-22.88 See, e.g., Geoff Norman, The Basic Role of Basic Science, 17 ADVANCES HEALTH

Sci. EDuc. 453, 454 (2012) (observing that "[r]ecent studies of clinical reasoning show thatexpert diagnosis is characterized by... some form of pattern-recognition.., followed by amore systematic confirmation"); Patel et al., supra note 80, at 417 (examining medical pro-fession, researchers observe, "[t]he complex and varied nature of medicine demands that aphysician acquire certain abstract biomedical models that have a certain degree of general-izability across classes of problems, and medical tasks (e.g., diagnostic and therapeutic de-cision making). This necessitates the development of rich symbolic representations.").

89 Pefia, supra note 66, at 6.90 Id.

[Vol. 19:251

Performance Isn't Everything

account for simple procedurally-oriented skills like asking for an ad-journment or conducting an inquest to obtain a default judgment, butit does not address the kinds of complex decision making required inmost lawyering. Lawyers make decisions at a much more complexconceptual level than just recognizing patterns, and real expertise isassociated with this higher level.

These insights from cognitive science suggest that expert lawyersneed more than simple rules, protocols, and strategies to facilitate pat-tern recognition. They need to acquire cognitive processes that helpthem organize the abundance of information pertinent to a case.91 Yetthese essential cognitive processes are largely ignored by theDreyfuses and the Carnegie Report. Just as researchers have con-cluded that the Dreyfus theory "is too simple to account for the com-plex pattern of phenomena linked to expert intuition" in nursing, sotoo should the same conclusion apply to the practice of law. 92

C. Cognitive Processes Used by Experts

Relying on the Dreyfus model of expertise, the Carnegie Reporteschews the importance of "formal modes of thinking" and insteademphasizes "skilled human performance" as the focus of lawyeringskills education.93 To move from novice to expert, the Report asserts,students must acquire "the common techniques of the lawyer's craft"for such performance. 94 In contrast to focusing simply on expert tech-niques and protocols, cognitive scientists have looked to the differentreasoning processes that are used by experts when making decisions inpractice. It is the students' acquisition of these cognitive processes,they assert, that is at least as significant, if not more significant, in thedevelopment of expertise.

1. Expert Mental Schemas

One attribute of expertise identified by cognitive scientists is theuse of schemas, frameworks of representations that help experts en-gage quickly in complex reasoning. These researchers distinguish be-

91 See generally Anderson et al., supra note 86, at 9 (criticizing apprenticeship models

of instruction such as those touted by Carnegie, and arguing that "the real goal should beto get students motivated and engaged in the cognitive processes that will transfer. What isimportant is what cognitive processes a problem evokes and not what real-world trappingsit might have. Often real-world problems involve a great deal of busy work and offer littleopportunity to learn the target competencies.").

92 Gobet & Chassy, supra note 66, at 132. See also Pefia, supra note 66, at 6 (arguingthat "[a]nyone who wants to propose a model to develop clinical problem-solving skills,must recognize that the skills used to solve [ill-defined] problems are of a different naturethan the skills used to solve [well-defined] problems.").

93 SULLIVAN ET AL., supra note 9, at 116.94 Id. at 116-17.

Fall 20121

CLINICAL LAW REVIEW

tween working memory and long-term memory. Working memory isequivalent to conscious thought. Our working memory is only capableof holding seven elements of information at a time and can only dealsimultaneously with two or three items when processing information.Therefore, anything beyond the simplest cognitive activities over-whelms working memory.95 In contrast, long-term memory can storelarge amounts of information and engage in complex cognitiveprocesses. We are not directly conscious of long-term memory, but"human intellectual prowess comes from this stored knowledge, notfrom an ability to engage in long, complex chains of reasoning inworking memory. '96

Knowledge is stored in long-term memory in the form ofschemas. Schemas are "ordered patterns of mental representationsthat encapsulate all our knowledge regarding specific objects, con-cepts, or events. '97 They are sophisticated, unconscious problem-solv-ing rules that allow us to organize information efficiently. Forinstance:

[w]hen faced with a problem such as (a+b)/c = d, solve for a, peoplemay immediately and automatically know that this problem issolved by multiplying out the denominator as the first move. Theyhave an automated schema for this problem that tells them immedi-ately, without conscious processing, how the problem should besolved.98

We develop schemas from repeated encounters with similar situa-tions. From those recurring experiences, we are able to categorizewhich characteristics of a given event are relevant, which should bestored for the future, and which should be rejected as irrelevant.99 Forcomplex problem solving such as that required in law or medicine, theframework around which these categorizations are developed is thebasic doctrine of the profession. 100

In regard to the acquisition of expertise in a profession, research-ers theorize that as a result of greater experience in a particular do-main, experts reflexively use their well-developed schemas to filterout irrelevant data and focus on relevant information to derive solu-tions to problems. 10 1 Experts automatically use their schemas to delve

95 John Sweller, Jeroen J. G. van Merrienboer & Fred G. W. C. Paas, Cognitive Archi-tecture and Instructional Design, 10 EDUC. PSYCHOL. 251, 252-53 (1998).

96 Id. at 253-54.97 Mark P. Higgins & Mary P. Tully, Hospital Doctors and Their Schemas about Appro-

priate Prescribing, 39 MED. EDUC. 184, 185 (2005) (citations omitted).98 Sweller et al., supra note 95, at 257.99 Id.

100 See Gobet & Chassy, supra note 66, at 133; Stefan H. Krieger, Domain Knowledgeand the Teaching of Creative Legal Problem Solving, 11 CLIN. L. REv. 149, 267-69 (2004).

101 Patel et al., Reasoning and Instruction in Medical Curricula, 10 COGNITION & IN-

[Vol. 19:251

Performance Isn't Everything

into the deep structure of a situation (its systematic properties), andthey seek to reformulate it in a way that enables them to use domainknowledge and previous experience to determine a course of action.

In the field of law, the anchors for schemas are basic legal doc-trine (e.g., contract, tort, property, evidence, and agency law; the rulesof procedure; professional responsibility). As novice lawyers delveinto practice, the doctrinal framework helps them to organize theirexperiences, and they begin to construct schemas. As they accumulateexperiences, these schemas assist them in handling new and unfamiliarcases more effectively and efficiently.

From a cognitive perspective, expert lawyers conducting a clientinterview in a personal injury case involving an automobile accidentare not always consciously considering, for example, each element of anegligence claim and deductively crafting a theory. Rather, theirschemas developed from past experiences, and structured around ba-sic legal doctrines, help them to focus their inquiry semi-automaticallyon those elements. In this instance, their inquiry will probably concen-trate on issues relating to the actions of the two drivers before theaccident rather than the prior relationships between the parties, theirfinancial situations, or their family difficulties. While this process maynot be deliberate or conscious, it is as much a cognitive process as theconscious use of an algorithm to solve a problem.10 2

Schema theory differs in important respects from the Dreyfuses'model of holistic similarity recognition. While perception plays an im-portant role in both theories, for the Dreyfuses, it is the basis for allexpert decision making. In their model, experts recognize patternsbased on their perceptions of the surface characteristics of a particularsituation and their abilities to relate those characteristics to the spe-cific contexts of prior experiences. 10 3 Under schema theory, however,the expert practitioner unconsciously delves beneath these perceivedcharacteristics. Her mental representations are organized around con-

STRUCInON 335, 339 (1993).102 The Dreyfuses are incorrect when they assert that the process is mindless. A central

basis for the Dreyfuses' contention that cognition generally plays no part in expert per-formance is that artificial intelligence ("Al") researchers have been unable to developrules that replicate the conceptual process used by experts in decision making. Hubert L.Dreyfus & Stuart E. Dreyfus, Making a Mind v. Modeling the Brain: Artificial IntelligenceBack at a Branchpoint, DAEDALUS, Winter 1988, at 15. The fact that Al research has notyet reproduced the expert reasoning process does not mean, however, that such replicationis impossible. Nor does it address the insight of schema theory that the use of representa-tional frameworks works unconsciously in expert decision making. See generally Thornton,supra note 76, at 290; Patel et al., supra note 80, at 424.

103 DREYFUS & DREYFUS, supra note 67, at 28 ("When we speak of intuition or know-how, we are referring to the understanding that effortlessly occurs upon seeing similaritieswith previous experiences.").

Fall 2012]

CLINICAL LAW REVIEW

cepts from her domain knowledge, which she uses to discover all therelevant facts she encounters in the situation. Although the Dreyfuses'theory rejects the idea that internal representations guide attention,schema theorists assert that unconscious, internal frameworks are keyfeatures of expert practice. Such practice has a cognitive as well as aperceptual basis, requiring the filtering out of irrelevant informationand the focusing in on relevant facts.104

The distinction between schema theory and the Dreyfuses' modelof "holistic similarity recognition" is not merely semantic. For theDreyfuses, effective pedagogy in a field requires the use of protocolsand techniques to assist students in recognizing the surface patterns ofproblems. Schema theory, on the other hand, suggests that the designof effective pedagogies for experiential courses should focus on help-ing students acquire representational frameworks for decision makingin practice.10 5 With these schemas, students can delve, both con-sciously and unconsciously, beneath the surface of problems to deeplyanalyze them. To acquire such schemas, students must have a soliddoctrinal foundation for the problems they are handling, they must begiven sufficient opportunities to engage in practice in similar cases,and instructors must provide them with effective feedback so that theycan begin to develop their own schemas. 10 6 This process requires morethan repeat performances of different protocols and the learning ofcommon techniques and procedures. Rather, it involves the develop-ment of representational frameworks for applying doctrine in practice.

2. Cognitive Flexibility

Cognitive scientists also assert that the application of knowledgeby an expert in handling ill-structured problems requires the simulta-neous consideration of multiple concepts that are individually com-plex, a process called cognitive flexibility.10 7 An ill-structured problemis characterized by some of the following conditions:

(1) [t]he place to begin to define the problem is usually not clear;(2) there often are many contingencies to take into account; (3) howto weigh and assess the various interdependent variables is uncer-tain; (4) one has to continuously reframe and reconsider what one is

104 See Gobet & Chassy, supra note 66, at 135; Patel et al., supra note 101, at 339.105 Gobet & Chassy, supra note 66, at 136-37 (recommending that instructional methods

be developed to foster the acquisition of schemas).106 Stefan H. Krieger & Serge A. Martinez, A Tale of Election Day 2008: Teaching

Storytelling Through Repeated Experiences, 16 J. LEGAL WRITING INST. 117 (2010).107 Vimla L. Patel, Nicole A. Yoskowitz, Josd F. Arocha & Edward H. Shortliffe, Cogni-

tive and Learning Sciences in Biomedical and Health Instructional Design: A Review withLessons for Biomedical Informatics Education, 42 J. BIOMEDICAL INFORMATICS 176, 180(2009).

[Vol. 19:251

Performance Isn't Everything

doing in light of new information and shifting calculations; and (5)the goals to be sought are frequently subject to debate and refine-ment and are not usually susceptible to clear measurement. 10 8

Legal problems are often ill-structured because they arise underuncertain conditions with regard to the client's interests, the otherparty's intentions, the controlling legal doctrine, and the proceduralconstraints of the legal system. For example, in an initial interview of aclient about a landlord/tenant problem, lawyers may encounterproblems with different legal doctrines-tort, contract, and propertytheories; evidentiary issues; diverse procedural obstacles; the relation-ship between the parties; and tricky ethical quandaries. Expert attor-neys must consider all these issues at the same time and often cannoteasily compartmentalize them. In such situations, "there is not likelyto be a set, technical approach to follow to reach a solution nor neces-sarily a single determinant answer to resolve the matter."'10 9

Given the uncertain nature of ill-structured problems, cognitiveflexibility theorists contend experts do not passively relate situationsto prior experiences or retrieve well-developed schemas. Rather,when faced with a problem, they construct meaning about the situa-tion using the given information in conjunction with their prior knowl-edge and schemas. 110 The prior knowledge that is brought to bear isitself reconstructed on a case-by-case basis rather than merely re-trieved from memory. This process requires the flexible use of pre-existing knowledge, the ability to use multiple schemas, and the skillto view a problem from different conceptual perspectives." 1

Therefore, in training novices to handle ill-structured problems,oversimplification of concepts is not helpful and may actually impedeacquiring expertise."12 Following the Dreyfuses, one might think thatinstructors should adopt methods that simplify concepts, compartmen-talize knowledge, and teach rules and strategies for performing partic-ular skills.113 In fact, that is the recommendation of the CarnegieReport.114 But cognitive flexibility theory shows that reductivism is

108 Mark Neal Aaronson, We Ask You to Consider: Learning About Practical Judgment

in Lawyering, 4 CLIN. L. REV. 247, 257 (1998).109 See id.110 Rand J. Spiro, Paul J. Feltovich, Michael J. Jacobson & Richard L. Coulson, Cogni-

tive Flexibility, Constructivism, and Hypertext: Random Access Instruction for AdvancedKnowledge Acquisition in Ill-Structured Domains, in CONSTRUCTIVISM AND THE TECHNOL-OGY OF INSTRUCTION: A CONVERSATION 64 (Thomas M. Duffy & David H. Jonassen, eds.1992).

111 Id.112 Id. at 63. See also Patel et al., supra note 107, at 180.113 See DREYFUS & DREYFUS, supra note 67, at 21-22.114 Drawing from the Dreyfuses' theory, the Carnegie Report asserts:

The prime learning task of the novice in the law is to achieve a basic acquaintance

Fall 20121

CLINICAL LAW REVIEW

only helpful in training novices in well-structured domains in whichthe solution to problems is simple and straightforward. 115

Oversimplifying issues does not help learners construct meaningin ill-structured domains because it can result in different types of "re-ductive biases."'1 6 One such bias is the generally incorrect assumptionthat parts of a complex process retain their characteristics when inte-grated into the entire process. 1 7 In the lawyering process, this biasmight result in minimizing the significance of obtaining evidence fromthird parties or adversaries. Novice attorneys, who are trained in thetechnique of client-centered interviewing, might be prone to overem-phasize client-reported information in the fact investigation process.Another bias resulting from overcompartmentalizing instruction is theassumption that highly-interrelated elements of a process are indepen-dent of each other." 8 For example, a law student instructed about thenecessity of using a leading form of questioning in cross examination,in isolation from an understanding of the overall theories of the case,might be prone to use that form even when it is detrimental to theproceeding." 9 These errors of oversimplification, cognitive scientistswarn, "can compound each other, building larger scale networks ofdurable and consequential misconception.' 20 Quite simply, they givestudents an artificial sense of the expert practice of law and can haveenduring negative effects.12'

Rather than compartmentalizing concepts, cognitive flexibility

with the common techniques of the lawyer's craft. The novice should not be asked toexercise judgment or interpret a situation as a whole. Instead, the novice must learnto recognize certain well-defined elements, regardless of what else is happening.

See SULLIVAN ET AL., supra note 9, at 117.115 Patel et al., supra note 107, at 180; Spiro et al., supra note 110, at 60-61 (comparing

the fairly well-structured problems of basic physics with the application of those conceptsin "messy" real-world engineering problems).

116 Spiro et al., supra note 110, at 62.117 See id.118 Id.119 While we are raising questions about the use of these techniques, we do not advocate

discarding instruction in lawyering methods, such as client-centered interviewing or cross-examination drills. Rather, we are asserting only that overemphasis on the use of thesetechniques, as suggested by the Carnegie Report, may be detrimental to the acquisition ofcognitive flexibility by law students. These techniques do have some use in introducingstudents to certain discrete skills. For overall instruction in lawyering skills, however, it ismore important to adopt methods by which students can acquire the cognitive flexibilitythat will enable them to handle ill-structured problems effectively.

120 Spiro et al., supra note 110, at 62.121 As a medical educator observes:

believing that students should only memorize rules has a dark side and can causedeleterious consequences. When rules are available for everything, novices can sparethe effort of imagining a different way to solve an [ill-structured] problem. Hence,they would tend to proceed to solve problems in a rather mindless way.

Pefia, supra note 66, at 7.

[Vol. 19:251

Performance Isn't Everything

theory focuses on the connection between different concepts and theirinteraction and variations across contexts.122 The theory suggests thatlearners in ill-structured domains can acquire the durable ability toconstruct understanding in complex situations only by "revisiting thesame material, at different times, in rearranged contexts, for differentpurposes, and from different conceptual perspectives.' 1 23 In an arenalike legal practice, any single explanation of a complex concept or casewill miss important facets that would have greater or lesser signifi-cance in a different context.12 4

Accordingly, as researchers suggest for medical education, casesin law school experiential courses should cover a range of situationsand problems that use different pieces of knowledge or the sameknowledge in different ways. Also, emphasis should be put on the re-lations among case problems and between cases and concepts. Thefocus on these relations reveals how knowledge can be reconstructedfor novel situations, highlighted by examining the same cases frommultiple perspectives and with different goals.125

3. Adaptive Expertise

Finally, in examining the nature of expertise, cognitive scientistsposit that experts possess "adaptive expertise." Adaptive expertise en-tails the ability to use standard strategies to efficiently solve problemsin routine situations, and the ability to develop innovative strategies tosolve problems in novel situations. 126 Without efficient use ofschemas, a practitioner's attention is overwhelmed with details. With-out some innovation, however, one cannot handle the new problemsthat often arise in ill-structured domains. 127 The most effective expertshave the metacognition to distinguish between these two types of situ-ations and to judge when to be efficient and when to be innovative. 128

Research has indicated that this metacognitive ability reflects adeep, theory-based understanding of the domain. 129 Drawing fromthat understanding, adaptive experts are able to distinguish semi-auto-matically between those situations which require only the use of stan-

122 Patel et al., supra note 107, at 180.123 Spiro et al., supra note 110, at 65.124 See id. at 65.125 Patel et al., supra note 107, at 180.126 Id. at 188.127 Valerie M. Crawford & Susan Brophy, Adaptive Expertise: Theory, Methods, Find-

ings, and Emerging Issues, http://ctl.sri.com/publications/downloads/AESymposiumReportOct06.pdf at 12 (2006).

128 Id. at 13.129 Susan M. Barnett & Barbara Koslowski, Adaptive Expertise: Effects of Experience

and the Level of the Theoretical Understanding It Generates, 8 THINKING & REASONING

237, 252 (2002).

Fall 2012]

CLINICAL LAW REVIEW

dard schemas and those which demand more deliberate andinnovative reasoning. They are also able to recognize situations inwhich their knowledge is limited, requiring consultation with thosewho have more expertise in the domain. Similarly, this deep under-standing helps experts identify those problems that require collabora-tion with other practitioners, whether within or outside of theirdomain.

To help students acquire adaptive expertise, more is required inthe instructional process than the Carnegie Report's recommendationthat novices "achieve a basic acquaintance with the common tech-niques of the lawyer's craft."' 30 To foster adaptive expertise, research-ers recommend that novices be exposed to a variety of problems withdiffering complexity but sharing a similar theoretical base. With ap-propriate feedback during this process, learners can begin to distin-guish between routine problems and those that require moredeliberate consideration, consultation with an expert, or collaborationwith others.' 3' In the law school context, this recommendation sug-gests that clinical and other experiential courses should be designed toprovide students with different experiences in cases of varying com-plexity in a particular subject matter area. These experiences may en-able novices to achieve an understanding that not all cases are majorfederal actions, nor are they simply run-of-the-mill proceedings. Sometasks require close consultation with supervisors, while others can behandled independently. And in some situations, nonlawyers and ex-perts in other fields might be helpful in addressing the client'sproblem.

130 SULLIVAN ET AL., supra note 9, at 117. The Carnegie Report does recommend thatstudents become "metacognitive" about their learning, but, in the context of the entirestudy, it is unclear what the authors mean by the use of that term. The Report asserts,"[T]he essential goal of professional schools must be to form practitioners who are awareof what it takes to be competent in their chosen domain and to equip them with the reflec-tive capacity and motivation to pursue genuine expertise. They must become 'metacogni-tive' about their own learning." Id. at 173. Without any further explanation about thenature of the metacognition process, this recommendation is opaque and unhelpful. It sug-gests simply and generally that students should become reflective about their learning butfails to flesh out a specific description of what this reflective capacity entails.

131 See Crawford & Brophy, supra note 127, at 17-18. As these authors observe, "Muchlearning in the professions occurs by tacit acquisition. It is important to consider how toengineer learning experiences in the context of practice to encourage people to becomeadaptive through their learning." Id. at 17. Accordingly, adaptive expertise is not necessa-rily taught by pedantic lessons in the virtue of this ability, but rather through experiencesthat help novices tacitly acquire it.

[Vol. 19:251

Performance Isn't Everything

III. OUTCOME ASSESSMENT IN LAW SCHOOL

EXPERIENTIAL COURSES

A. Carnegie Report Recommendations on AssessingLawyering Skills

To assess student learning in lawyering skills courses, the Car-negie Report urges the use of methods that evaluate the learner's de-velopment of expertise. 132 While we do not disagree with thisapproach to outcome assessment, our differences with Carnegie arisefrom its conception of expertise. Since cognitive processes play only asecondary role in expert practice, when viewed through the Carnegie-Dreyfus lens, performance becomes the primary focus of outcome as-sessment in experiential courses. Alternatively, following the researchdescribed in the prior section, we suggest that cognitive competence isan essential element of expertise. In our view, then, assessment mustconsider not only a learner's performance, but also the cognitiveprocesses revealed during that performance.

Following Dreyfus, Carnegie envisions law schools where stu-dents "acquire[ ] mature skill by moving from a distanced manipula-tion of clearly delineated elements of a situation according to formalrules toward involved behavior based on an accumulation of concreteexperience. ' 133 Focusing on the students' development of "involvedbehavior," the Report asserts that, "[s]ound assessment [of lawyeringskills] must include an evaluation of students actually performing. 134

When outcomes are framed in terms of a student's ability to util-ize particular protocols or techniques, assessment is likely to focus onhow well the student adhered to the protocol or applied the tech-nique: For example, did the student start with open questions andgradually narrow her questioning? Did she use the T-funnel techniquedesigned for this particular situation? Did the student effectively fol-low the PrOACT protocol by identifying problems, setting objectives,developing alternatives, evaluating consequences, and structuringtradeoffs?

One example of this type of performance-focused assessmentcomes from Best Practices, which describes a rubric for assessing stu-

132 SULLIVAN ET AL., supra note 9, at 171-73. The authors of the Report obviously do

not claim that law schools can prepare students in three years to become expert practition-ers, but they do suggest that, by graduation, students should have begun to acquire some of

the attributes of expertise. We concur with this proposition. Most recent cognitive sciencestudies show that it takes people at least ten years of intense involvement with a skill orprofession to acquire expertise. Ericsson, Deliberate Practice, supra note 83, at S72; K.

Anders Ericsson, Ralf T. Krampe & Clemens Tesch-R6mer, The Role of Deliberate Prac-

tice in the Acquisition of Expert Performance, 100 PSYCHOL. REV. 363, 366 (1993).133 SULLIVAN ET AL., supra note 9, at 172.134 Id. at 174.

Fall 20121

CLINICAL LAW REVIEW

dents' progress toward expertise in lawyer-client communicationskills.135 In this model, client interviewing has been broken down intodiscrete segments, and evaluators use eight questions to measure stu-dent proficiency in lawyer-client communication:

1. Were the greeting and introduction appropriate?

2. Did the lawyer listen to the client?

3. Did the lawyer use a helpful approach to questioning?

4. Did the lawyer accurately summarize the client's situation?

5. Did the client understand what the lawyer was saying?

6. Did the client feel comfortable with the lawyer?

7. Would the client feel comfortable having the lawyer dealwith her situation?

8. Would the client come back to this lawyer if she had a newlegal problem?136

For each of these eight elements, proficiency can be assessed on ahighly specific scale and given a score between 1 and 5. For example,the sixth criterion, client comfort with the lawyer, is scaled as follows:

1 point: Lawyer was bored, uninterested, rude, unpleasant, coldor obviously insincere.

2 points: Lawyer was mechanical, distracted, nervous, insincere,or used inappropriate remarks.

3 points: Lawyer was courteous to the client and encouraged theclient to confide in her.

4 points: Lawyer was generally attentive to and interested in theclient. The client felt confident to confide in her.

5 points: Lawyer showed a genuine and sincere interest in theclient. There was a sense of connection between the clientand the lawyer.137

Using this set of clear and observable criteria, a student interviewcan be given a score from 8 to 40 to indicate the student's level ofproficiency in professional expertise in communication with clients.This assessment scale is based entirely on elements of student per-formance that do not require any evaluation of the student'sreasoning.

135 STUCKEY ET AL., supra note 9, at 246-7.136 Id.

137 Id. at 247.

[Vol. 19:251

Performance Isn't Everything

The appeal to teachers of a checklist approach to assessment isnot insignificant. This kind of method is relatively straightforward andunambiguous in its application, with clear goals and clear criteria forevaluation. 138 Problematically, however, by focusing on whether thestudent adequately follows a checklist of performance elements, thistype of assessment diminishes the importance of the student's reason-ing process as she engages with the lawyering problem.

Pointing to assessment methods in medical education, Carnegieseems to concur with this checklist approach. The Report notes ap-provingly that in the early years of medical school, students are evalu-ated on their ability to take medical histories and perform physicalexaminations on actors playing the role of "standardized patients."Later, they are observed by supervisors in their interactions with ac-tual patients, and still later, they are assessed in their residencies on awide range of other "technical and interpersonal skills."'1 39 Carnegierecommends the use of a similar approach of ongoing performanceassessment in law school skills programs.' 40

Besides its reference to assessment in medical school, the Reportcites no supporting evidence of a correlation between performanceassessments and accurate evaluation of developing expertise in law-yering skills. Nor does it cite to any empirical evidence supportingperformance-based assessment in medical training. Relying on theDreyfus model, the authors of the Report simply argue that since ex-pertise is exhibited by particular types of behavior, performanceshould be the focus of the assessment of lawyering skills.

A review of the empirical research on this subject raises seriousdoubt about Carnegie's unsupported claims regarding the benefits ofperformance assessment of students' learning of lawyering skills.While Carnegie is correct that performance assessment has becomequite the rage in medical education, little empirical research exists onwhether such methods are valid measures of clinical competence.' 41

Indeed, one major study in the medical field suggests that in assessingclinical ability, cognitive competence may be at least as important as,if not more important than, performance. 142

138 Students are also likely to embrace performance-based assessment. They will be as-

sessed favorably if they simply select and apply the proper tool from their toolkit of law-yering techniques.

139 SULLIVAN ET AL., supra note 9, at 175.140 Id.141 Geoff Norman, So What Does Guessing the Right Answer Out of Four Have to Do

with Competence Anyway?, 77 BAR EXAMINER, Nov. 2008, at 18.142 Robyn Tamblyn, Physician Scores on a National Clinical Skills Examination as

Predictors of Complaints to Medical Regulatory Authorities, 298 J. AM. MED. Ass'N 993(2007). Interestingly, in the Carnegie Foundation's recently published study of medical ed-ucation, the authors (a different set than those who studied legal education) acknowledge

Fall 2012]

CLINICAL LAW REVIEW

In this study, researchers examined the relationship between pa-tient complaints to medical regulatory authorities about the nature oftheir physician's care and the physician's previous performance on theCanadian medical licensing exam. The research sample included all3,424 physicians who took the licensing exam between 1993 and 1996and were licensed to practice in Ontario and/or Quebec. Researchersthen compiled data on all complaints filed with provincial regulatoryauthorities between 1993 and 2005 which were investigated and foundto be valid. For each physician, they determined complaint rates, de-rived by dividing the number of valid complaints by years of practicetime, for two different types of complaints, those concerning commu-nication issues and those concerning quality of care. Finally, the re-searchers compared the two different complaint rates with eachphysician's performance on the various components of the licensingexam.143

One part of the licensing exam assessed medical knowledge usingapproximately 450 multiple-choice questions about medicine, surgery,obstetrics-gynecology, pediatrics, psychiatry, and preventivemedicine. 144 A second component assessed clinical decision-makingskills using write-in or menu-selection response formats on 36 to 40clinical problems concerning critical aspects of diagnosis or manage-ment. Grades on these problems were not based on a single correctanswer but on the relative quality of the responses regarding criticaldecisions in situations in which errors could affect the patient out-come.145 Essentially, this part tested the candidate's cognitive abilityto solve problems in practice. The" final part was a performance-basedstandardized patient examination which asked candidates to interactwith simulated patients for five to ten minutes. Trained physician-ob-servers assessed candidates in a number of areas, including data col-lection (e.g. medical history and physical examination) andcommunication skills (e.g., whether the test-taker used condescending,offensive, or judgmental behaviors or ignored patient responses). Inthis final section, problem-solving skills were assessed by post-encoun-ter questions on diagnosis, investigation, interpretation of test results,and management. Physician-examiners used an answer key to score

that, "[m]uch work remains to be done to establish the reliability and validity ... of per-formance assessment" of complex clinical skills. MOLLY COOKE, DAVID M. IRBY &BRIDGET C. O'BRIEN, EDUCATING PHYSICIANS: A CALL FOR REFORM OF MEDICAL

SCHOOL AND RESIDENCY 107 (2010). The Carnegie Report on educating lawyers containsnone of this cautionary language about performance assessment.

143 Robyn Tamblyn, supra note 142, at 994-95.144 Id. at 995.145 Id.

[Vol. 19:251

Performance Isn't Everything

these responses. 146

After examining the data, researchers found that the best predic-tor of quality-of-care complaints was the licensing exam's clinical deci-sion-making component, which focused on the cognitive ability ofcandidates to solve problems. The better the test-taker's score on thatpart of the exam, the lower the complaint rate for that physician. Al-though high scores on the communications component of the perform-ance exam were not as good a predictor of low quality-of-carecomplaint rates, researchers also found a statistically significant in-verse correlation between that measure and such rates.147 Addition-ally, researchers found that scores on both the communication part ofthe performance exam and on the clinical decision-making examserved inversely, and at nearly the same level, as predictors of commu-nication complaint rates.148 Finally, researchers found a statisticallysignificant inverse relationship between overall complaint rates andscores on the multiple-choice and clinical decision-making sections, aswell as on the communication component of the performance exam.149

In sum, scores on the clinical decision-making component of thelicensing exam were better inverse predictors of complaint rates thanany other component of the exam. Clinical decision-making scoresfared just as well as inverse predictors of overall complaints as thecommunications component of the exam. Those scores were better in-verse predictors of quality-of-care complaints than any other compo-nent of the exam. Moreover, even in terms of complaints from actualpatients about communication problems, the scores on the clinical de-cision-making exam were comparable to those on the communicationsexam as an inverse predictor of complaints. And surprisingly, even thescore on the multiple-choice medical knowledge component of theexam was a statistically significant inverse predictor of the overallcomplaint rate.150

146 Id. By measuring problem-solving skills using an answer key, the licensing body wasnot assessing the cognitive processes used by the test-takers as much as their "correct"performance on a post-examination quiz.

147 Id. at 999. Researchers found no statistically significant relationship between quality-of-care complaint rates and scores on the multiple-choice part of the exam or on the dataacquisition and problem-solving components of the performance exam.

148 Id. Researchers found no statistically significant relationship between communica-tion complaint rates and scores on the multiple-choice part of the exam or on the dataacquisition and problem-solving components of the performance exam.

149 Id. The best inverse predictors of any complaint were scores on the clinical decision-making and communications exams. No statistically significant relationship was found be-tween overall complaint rates and the data acquisition and problem-solving components ofthe licensing exam.

150 Indeed, a major researcher in the area of medical education asserts that multiple-choice tests appear to consistently outperform performance tests in terms of measuredvalidity. Norman, supra note 141, at 19-20.

Fall 2012]

CLINICAL LAW REVIEW

Of course, there are limitations to the conclusions that can bedrawn from this study. It is open to question whether or not the num-ber of complaints to regulatory authorities reflects a practitioner's ac-tual expertise. Moreover, the different experiences of the subjectsafter graduation were not considered in the analysis of the data. Buteven with these limitations, given the large sample size and thetwelve-year period over which researchers tracked licensing examscores and complaints, the study certainly provides stronger supportfor its conclusions than the Carnegie Report does for relying uncriti-cally on analogies between legal education and standardized patientperformance assessment in medical education.

Admittedly, we cannot be certain that research findings regardingthe medical profession are transferable to the legal profession. Never-theless, this major medical study raises serious questions about thefocus of Carnegie and the Dreyfuses on performance as a primarymeasure of ability in practice. Contrary to the assumption implicit inthe Carnegie Report's recommendations, performance assessmentmay not be the best measure of a student's long-term lawyering abil-ity. Instead, scores on tests of clinical reasoning and cognitiveprocesses appear to be better predictors of acquisition of expertiseand ability in practice. At the very least, this research should call intoquestion a headlong rush to adopt Carnegie's recommendations to useperformance as the primary assessment tool for measuring lawyeringskills.

B. Assessment of Cognitive Competence

If the Canadian study is correct that assessment of clinical reason-ing ability is a significant factor in measuring a learner's acquisition ofexpertise, an issue arises as to the best methods for evaluating thedevelopment of such competence. As described in Part II, cognitivescience research has demonstrated that expert reasoning has a numberof attributes: the use of schemas, cognitive flexibility, and adaptiveexpertise. While it is easy to assess a test-taker's performance with astandardized patient, it is much more difficult to determine whetherthe reasoning of a test-taker demonstrates any of these attributes.Simply stated, it is impossible to peer into someone's head, as "think-ing cannot be observed by other people. '151

To address this problem, one of the experimental methods nowemployed by medical educators to assess cognitive competence is theuse of the "think-aloud" interviewing methods employed in cognitive

151 K. Anders Ericsson, Protocol Analysis, in A COMPANION TO COGNITIVE SCIENCE

425 (William Bechtel & George Graham, eds., 1998).

[Vol. 19:251

Performance Isn't Everything

science studies of the reasoning process. 152 The purpose of using thismethod is to replicate as closely as possible the actual cognitive pro-cess of the subjects. Under the think-aloud protocol, researchers asksubjects during the interview to verbalize their thoughts spontane-ously as they emerge into their attention.153 Even though use of thismethod does not provide a perfect match between subjects' thoughtsand reports, researchers have found consistently strong evidence thatthis method results in a strong correlation between the two. 154

In medical education studies of this method, interviewers providemedical residents with problem scenarios describing patient histories,known symptoms, and the proposed case management in each in-stance. The residents are asked to think aloud as they read these sce-narios, to summarize the case history, and to evaluate the proposedcase management. The interviews are recorded and transcribed. Theresearchers then review the transcripts to assess the reasoning used bythe residents in practice.1 55

IV. USE OF THINK-ALOUD INTERVIEWS WITH LAW STUDENTS

Using the medical field's "think aloud" technique as inspiration,we have designed an experimental assessment method intended toidentify the different kinds of cognitive processes used by students asthey solve problems in practice. Specifically, we give students in aclinical program a hypothetical problem that is representative of workthey have experienced in the program, and record them as we askthem to talk it through. Our hypothesis is that by prodding students totalk about a problem without a filter, we will understand, as much aswe possibly can, what they are thinking "in practice."

A. Methodology for Think-Aloud Assessments

1. Creating Hypothetical Scenarios for Think-Aloud Assessments

The goal of our experiment is to give us insight into the develop-ment of specific cognitive processes used by our students in practice.Specifically, as a means of assessing their progress toward expert prac-

152 Telephone Interview with Vimla Patel, Professor of Biomedical Informatics, School

of Health Information Sciences, University of Texas Health Sciences Center, in Houston,TX (Mar. 28, 2010).

153 Researchers have also found that if subjects are asked to recall their reasoning pro-cess after a long delay, the completeness and accuracy of recall is impaired. Subjects areprone to infer their thoughts as opposed to correctly recalling them from memory. Erics-son, supra note 151, at 429, 430.

154 Id. at 430.155 E-mail from Vimla Patel, Professor of Biomedical Informatics, School of Health In-

formation Sciences, University of Texas Health Sciences Center, to Stefan Krieger, Profes-sor of Law, Hofstra University (Mar. 31, 2010, 8:23 p.m. EDT) (on file with authors).

Fall 20121

CLINICAL LAW REVIEW

tice, we are trying to observe their schemas for representation ofproblems, their cognitive flexibility, and their development of adap-tive expertise. With these objectives in mind, we designed our hypo-thetical scenarios with the following goals:

a. Lead students to access the schemas that they have created foraddressing particular issues that commonly arise in representation.We developed the hypotheticals within the context of a clinic-spe-cific problem, to allow our students to draw on doctrinal knowledgeand mental models that they should have developed through theirclinical work. 156 To identify and evaluate the mental models thatthey had already constructed, we examined how they approached areasonably familiar problem. Students in completely unfamiliar en-vironments, we reasoned, were not likely to rely on schemas thathad been developed through their clinic work.

b. Require the students to attend to multiple issues simultane-ously. We created hypotheticals with multiple, complex, and interre-lated elements to assess our students' ability to handle severalelements at the same time, balance them, and understand how theirconsideration of these elements changed when students perceivedtheir interaction.

c. Present novel problems. To test our students' adaptive exper-tise, which required observing their ability to recognize and respondto novel situations, we presented them with problems containing el-ements that did not fit neatly within the routine schemas that weexpected them to have developed in the clinic. In addition, to testtheir ability to recognize the limits of their own expertise and theneed to consult with those who had more expertise, we includedelements that we considered too advanced for clinic students.

2. Conducting the Think-Aloud Interviews

We invited several clinic students to participate in the project,emphasizing that they would not be graded. We then asked a col-league-one with whom the students did not have a relationship-toconduct on-camera interviews with the students. We used an indepen-dent interviewer to prevent any undue influence from our own pre-existing relationships with the students. We feared that the studentsmight be intimidated by our presence or feel compelled to give re-sponses we expected. We also wanted to avoid the possibility that wemight influence the students' responses with our questions or reac-tions to their comments.

In the actual interviews, the interviewer simply gave a written hy-pothetical to the students and asked them to think aloud through their

156 See Appendix.

[Vol. 19:251

Performance Isn't Everything

responses as they reviewed the problem. The interviewer followed upas needed with open-ended questions to help the students expresstheir thoughts, to make sure they continued to think aloud, and toencourage them to explore as fully as possible their reasoningprocesses as they were happening. The entire interview lasted approx-imately 15 minutes for each student. Subsequently, we viewed andtranscribed each recorded interview to identify the cognitive processesused by students in responding to the problem.

3. Using the Interviews to Assess Student Reasoning in Practice

To assess the interviews, we considered the goals of the clinic inwhich the student was enrolled, then analyzed the transcript to iden-tify how well the students had met the course goals for developmentof cognitive processes. In the next section, we will demonstrate howthe assessment process worked for two students from a single clinic bydescribing the clinic's goals, presenting annotated transcripts of stu-dent interviews, and describing possibilities for practical application ofthe information developed from these interviews.

B. The Think-Aloud Interviews

1. The Hofstra Community and Economic Development Clinic

The Hofstra Community and Economic Development Clinic (the"CED" or the "Clinic") is a one semester, non-litigation clinic thatmainly represents small community-based organizations andmicrobusinesses owned by aspiring entrepreneurs. While there is widevariety in the work of the Clinic, a core element is helping inchoatenot-for-profit and for-profit organizations choose and create the mostappropriate corporate form. During the semester, students have theopportunity to counsel several clients about the particular set of issuesthat arise at the early stages in the development of a small entity.

The CED has pedagogical goals, both general and domain-spe-cific, that drive course design, teaching, and client selection. The CEDis designed to cultivate in students the following cognitive processes:

a. Schemas: Students should develop mental models that serve as aframework for understanding and dealing with lawyering problems.In the CED, these schemas should include:

i. A schema for helping a client choose the form of a corporateentity, and then creating that entity;

ii. Because all of the Clinic's clients are groups or entities, aschema for asking and answering the question, "Who is the cli-ent?," and identifying and addressing other ethical issues relat-ing to non-individual client representation;

Fall 2012]

CLINICAL LAW REVIEW

iii. A schema for understanding the attorney-client relationshipand the role of the attorney in representation, including apply-ing ethical rules to representation; and

iv. A schema for identifying and understanding client goals andpriorities.

b. Cognitive Flexibility: Students should be able to identify andkeep track of several different factors simultaneously. In the CED,these factors frequently include:

i. Activities of the organization (especially as they relate toqualification as a New York not-for-profit corporation and/or a501(c)(3) exempt organization);

ii. Involvement in lobbying and political campaigns;

iii. Sources of finance;

iv. Taxes;

v. Corporate governance and control;

vi. Liability; and

vii. Other client-specific goals.

c. Adaptive Expertise: Students should be able to distinguish be-tween "routine" and novel situations, as well as to understand whena situation is beyond their ability to handle and requires consulta-tion with those who have more expertise.Obviously, this list is ambitious. In practice, students are unlikely

to master all of these cognitive processes over the course of a four-month semester. The goal of the CED is to help students start to de-velop their ability to reason in practice through their work withclients.

2. Think-Aloud Interviews

In this section, we share the transcripts from two of our studentinterviews, coded for the students' reasoning processes. 157 Each inter-view was conducted at the end of the subject's semester in the CED.

157 Videos can be viewed at: http://studentlegalreasoning.info.

[Vol. 19:251

Performance Isn't Everything

David CommentaryLuscious Landscaping Scenario

1 5 8

In terms of immigration status, I'd - We Cognitive Flexibility: David takes note of theneed to look further into that. immigration issue as a potential factor inWell they feel that they don't want to be in client decision. However, this is the last timethe position of asking for or providing he mentions it in this exercise.immigration about their immigration -information about their immigration status. Adaptive Expertise: David is aware that theSo I would obviously look-if they were my immigration issue is beyond his expertise,clients, I would represent their interests and and that he needs to conduct research todo what's possible to limit the type of learn more.information that they may need to give whilecomplying with our regulations as imposedby the law. I would do further research intoseeing if that in fact creates a conflict bywithholding their immigration status or not.If it's not a problem then I don't assume weneed to go further into it but if there aresome policy issues, then definitely. So that'swhere I would start - that's pretty muchwhere I would start.

I would certainly ask the clients - or Schema: David has a framework forprospective clients - what their immediate understanding the lawyer's role at this stage.needs are and where these needs rank, He also shows that he has a model forwhether they're interested in creating a identifying client goals and helping the clientsbusiness or in working to develop the prioritize among them.community and stand up for unskilled Latinolaborers or Latino laborers in general...

And in doing so, I would - obviously I would Schema: The framework that David is usinguse that information to decide whether I'd for identifying client goals suggests that justlike to - I would suggest using - creating a because clients might say they wantnot-for-profit corporation or a general something doesn't mean that that is what willcorporation. They say they want to form a best meet their needs. He wants to dig acorporation but I don't necessarily know if little deeper.they know the difference and the intricaciesof corporations and not-for-profits or even Schema: David's model of the lawyer's rolecreating LLCs while still limiting their includes developing and presenting options toliability but not having to pay different a client.taxes.. .So I would, you know, create - Iwould make a long list of options for these Cognitive Flexibility: There are many optionsclients, choice-of-entity decisions for them with lots of permutations that areinitially, interconnected. David is aware of this, even

though at this point he does not expand onhis thinking.

But I would also - I would like to, you Schema: David comes back to theknow, list exactly where their needs stand importance of the lawyer knowing howwhich, you know.. .What are their primary important each client goal is relative to theneeds versus what are their secondary. others.

158 See Appendix.

Fall 20121

CLINICAL LAW REVIEW

I would.. First of all, I'd like to knowexactly who's involved. If, you know, it saysthey're representing a group of 5 men so -but they have ideas of representing all ofLatino laborers so I would have to informthis person that I'm speaking to andfollowing up with that I would berepresenting their corporation and not justthem and that my representation extends toeverybody that's involved with thecorporation - all the directors and officersthat represent the corporation and that mylegal representation would be towards thecorporation that they created, not just them.So I would find out what their needs are interms of the corporation as a whole. Andalso other information as to who else I canspeak to because I wouldn't be representingthis one person, I'd be representing thecorporation. Um ... I would - again, Iwould do some research into currentorganizations in the area to see if -

I mean, obviously I would check the nameavailability. They sound like they have aname - "Luscious Landscaping" - that they'dlike to use so if that's not available, I'd haveto inform them that that's not possible.But also I would, you know, I would seewhat other corporations are - ororganizations are around in the communityand are they succeeding in what their styleand structure is.

Schema: David has developed a schema thatemphasizes the importance of understandingwho the individuals are and their relationshipto "the client."

Adaptive Expertise: David recognizes thatthis is not an area that he has a frameworkfor understanding, so he comes up with aplan to get more information and seek outguidance from others to help him explorethis situation.

Schema: David is demonstrating a model forthe formative stages of any corporate entityin New York.

I would certainly call up the - I know certain Schema: David shows that he has a robustregulations have changed in terms of the schema for nonprofit formation.purposes that the Department of State islooking for in order to create not-for-profit Adaptive Expertise: David seems very lockedcorporations so I would look for some into the nonprofit schema-he appears to beguidance by the Department of State as to if having difficulty identifying this as a case thatI were trying to create this type of does not comfortably fit within his existingorganization, exactly how to word the type of schemas and that will require somepurposes or exempt purposes that the not- innovation.for-profit corporation would be looking for inorder to, you know, speed up theincorporation process.

And I would also, you know, I would ask Adaptive Expertise: David realizes that thisaround, I would find out - ask my colleagues is not an area that he has enough expertiseand ask other people in the clinic who are in to handle alone, and that he will need somethe process of incorporating their clients and help from his colleagues. However, he is stillfind out exactly how they've complied with thinking about procedural issues that arethe State's requirements and use all resources specific to the formation of nonprofitthat the clinic has that I could find. organizations. He seems to be having trouble

accepting that this problem may not fitwithin his non-profit framework.

I would investigate further who else is Schemas: David has returned to his earlierinvolved in the project. If it's just this one concern about who is the client, who is theperson who I'm speaking to, I would like person he will be interacting with, etc. Thissome other contact information for who else gives a good picture of how robust thiswould be involved or who else would be a schema is with this student.director of the corporation. It sounds like aninteresting client. I'd like to help them.

i

[Vol. 19:251

Performance Isn't Everything

No, I would certainly - I'd like to go and do Adaptive Expertise: David realizes that thissome of my own personal research on the is a novel problem and that he will probablyLatino laborers and the Latino community in be able to represent the client better bythe area and determine some of the, you understanding the community that they wantknow, needs of those laborers. You know, to help.determine myself and be able to bring someinformation to the table that might either bedifferent from what my client may bring upor, you know, create conversation.

I know that they are looking to reduce their Cognitive Flexibility: He has identifiedtax liability, so I would certainly suggest that reducing taxes as a factor in helping theif they were interested in creating a clients with their goals. He does not,corporation that they might create an LLC so however, resolve-or even identify-thethat they're not doubly taxed like a tension between this and other client goals.corporation.

Now if they're a not-for-profit corporation Cognitive Flexibility: David addresses theand they're interested in, you know, helping for-profit/not-for-profit dichotomy and thethe Latino community versus making a tension inherent in the client's stated goals.profit, then, you know, I would have to He does not seem to make much progressaddress all the IRS regulations of taking toward understanding the tension or how itrecognition of tax exempt status and such. impacts his client's goals.

I don't know if I would necessarily go along Cognitive Flexibility: David is struggling withthe lines of, you know, if I would go with a the tension between the group's charitablenot-for-profit corporation. It does seem like aims and its business aims. He never resolvesthis is a business venture for them and one this issue and ignores other elements, such asof the side points is that they, you know, in outside sources of funding.turn would like to help the Latinocommunity. But if it is in fact a business Schema: David's use of "I" indicates that heventure, then I think we're focusing on has a framework for his role in the decision-corporations versus LLCs or C-Corps and making process that puts him at thedifferent entities - profit-making entities. So, forefront.but in reducing tax liability then obviouslythe LLC would be more favorable than Adaptive Expertise: David has escaped fromcreating a C-Corp. Let's see if I missed any - his purely nonprofit schema. Now, however,

he is locked into an either-or scenario: anonprofit or a for-profit. He does not seemto think that this situation requires moreinnovation, such as exploring the idea of twoside-by-side organizations, a joint venture, noorganization at all, or other possibleinnovative solutions.

MaryRivertown Redevelopment

15 9

Okay - So I see that it's not a good Schema: Mary has developed aneighborhood and they want to do economic representational framework for nonprofitrevitalization so I'm thinking maybe non- organizations, and she is seeing how well herprofit. client fits that schema.

Training - So I'm thinking we're going to Schema: This is a routine step inhave to get consent from the Department of incorporation of New York nonprofitEducation. organizations-Mary has a schema for how

this process goes.

Okay, so then... since the 3 of them want to Cognitive Flexibility: Mary sees that there iswork together, I would talk to them about some tension here: partnerships would entailthe different types of corporations that they personal liability but are cheap; LLCs limitwant to set up. If they want to, they could liability but are expensive to set up. Afterdo, like, a partnership but then they would some mental calculus, she realizes that theyhave personal liabilities... So then, otherwise, probably have funds to do the LLC and thatthey could do a corporation or a limited it would probably be an appealing option.liability company - it depends how muchthey have for start-up costs - which it seems Adaptive Expertise: Mary does not seem tolike they do have their own funds so they consider that there might be other optionsmight want to do that. than the very few she lays out here.

159 See Appendix.

Fall 20121

CLINICAL LAW REVIEW

3. Using Think-aloud InterviewsDesign

for Student Feedback and Course

The think-aloud interview, like other types of assessment, "offerstwo insights: one into the students, and one into the instruction.' 160Neither of these insights is an end in itself; rather they are means tothe end of helping students achieve competence in practice. Insightsabout students can be used to give meaningful feedback. Insights intothe effectiveness of instruction can inform course design. In this sec-tion, we describe how think-aloud interviews can be used to guide stu-dent feedback and assist in course design in ways that help studentsdevelop their reasoning abilities and make progress toward compe-

160 Michael Hunter Schwartz, Teaching Law by Design: How Learning Theory and In-structional Design Can Inform and Reform Law Teaching, 38 SAN DIEGo L. REV. 347, 404(2001).

And then I would also think about doing a Schema: Mary has developed a model fornon-profit - especially because they want how to form an organization in New York.donations and government grants. So a She also shows that she has created a mental501(c)(3). So then, how to proceed for that. model of the nonprofit that includesWell, first they'd have to incorporate under organizations looking for grants andNew York and make a certificate of donations.incorporation and then send it to theDepartment of State and say that they wantto use - they're going to be building up thetown - so they'd have to meet non-profitgoals.

And I think that the grocery - well, the Cognitive Flexibility: She identifies and isgrocery store would make a profit, so. . . trying to deal with the tension between somemake a profit.. .so then, I think they would of their stated goals-in this case, the tensionstill be fine - maybe if they added something between making a profit and havingto help the people? Well, that's just the charitable purposes.initial phase, also.

So then, umm, maybe they'd want to do two Adaptive Expertise: Mary has concluded thatdifferent corporations then? Maybe one to - this situation does not fit well within thefor the first part of the stuff they wanted to schemas she has developed. She stops tryingdo like provide employment to local youth to fit the client into the non-profit or the for-and decrease crimes and then maybe they'd profit schemas and instead begins towant to do a separate corporation that would innovate. Her solution of multiplejust run the grocery store because it can organizations working in tandem is creativemake a profit. And then they could work and consistent with client goals.together?

And so, for the first one, I would tell them Schema: "I would tell them" suggests ato do a non-profit 501(c)(3) and that would schema relating to the lawyer's role indecrease crime and increase local incomes representation that gives importance to whatand provide training for the local work force, the lawyer, rather than the client, wants. It isSo then they could think of a name and not clear from this exercise whether Mary isreserve that or - and then send in the wedded to this particular representationcertificate of incorporation and apply - fill schema.out a [IRS Form] 1023 to get 501(c)(3) statusso they can get donations and investments ingovernment grants.

And then maybe another corporation so that Adaptive Expertise: Mary returns to herthey could set up the grocery store. And so innovation of multiple organizations workingthat way they could make a profit with that together to achieve the client's stated goalsand help the town in that way also. Umm, I of helping the community by succeeding.think that's it.

[Vol. 19:251

Performance Isn't Everything

tency in skills.

a. Student Feedback

Think-aloud interviews can help instructors monitor studentlearning in order to provide useful feedback to students about theirdevelopment of reasoning processes. By watching the videos with stu-dents and relating them to the work that students are doing in thecourse, the instructor can help students develop their cognitiveframeworks. The purpose of these discussions should not be to tellstudents "you did this right" or "you did that wrong," but rather tohelp them identify and reflect on their own reasoning processes.

One element of a post-interview discussion with David, for exam-ple, might focus on his continued efforts to make the client's needs fitneatly into the schema he had developed for forming and representingnonprofit corporations. Although much of David's work in the Clinicwas with nonprofit organizations, the clients in the Luscious Land-scaping hypothetical are not likely to benefit from such a structure.However, rather than address whether David's was the "correct" ap-proach, a discussion of his reasoning process would focus on under-standing why David was reluctant to abandon his preconceivednotions, and ways that he might become more aware of his use ofthese notions in the future.

A post-interview discussion with Mary would include discussionof her suggestion that her clients might be served best by creatingmultiple entities. Like David's, Mary's Clinic work largely consisted ofrepresenting small nonprofits, and none of her clients required multi-ple entities to achieve their goals, so the Rivertown Redevelopment &Revitalization scenario was in some respects a novel situation forwhich she had not developed a schema. Discussion of Mary's reason-ing processes, however, would not focus on whether she came up withthe "right" answer, but would address aspects of adaptive expertise,such as what she was thinking as she identified the novelty of the situ-ation, how she came up with her proposed solution on the fly, andwhat consideration she gave to consulting someone with greater ex-pertise. Such a discussion would help Mary and her instructor betterunderstand the cognitive processes involved in her representation ofclients.

b. Course Design

Think-aloud interviews can also be used to assess the ability ofcourse design to meet goals for student learning outcomes. We believethat development of cognitive processes is largely a function of coursedesign. Students unconsciously develop their own schemas and cogni-

Fall 20121

CLINICAL LAW REVIEW

tive strategies in response to their opportunities to practice with ex-pert feedback. 161

Contrary to Carnegie's suggestion, there is no standard "toolkit"of protocols and procedures possessed by all experts that can simplybe taught to novices. Rather, through their experiences in practice andthe effective feedback they receive, novices construct their ownschemas for solving problems. The goal of course design should be tocreate the environment and feedback mechanisms to nurture the con-struction of these schemas. The assessment of think-aloud interviewscan help instructors identify areas in which they have assisted studentsin developing cognitive frameworks and those in which they have notbeen effective.

The interviews with Mary and David suggest that the Clinic couldbe more effective in helping students develop cognitive flexibility.David and Mary each seemed to struggle with considering and balanc-ing multiple factors. To help them develop a framework for cognitiveflexibility, the Clinic might be redesigned to provide more opportuni-ties for students to practice in situations that require keeping track of,and evaluating, multiple complex factors simultaneously. This changemight be incorporated into in-class exercises and simulations. In addi-tion, as an aid to developing cognitive flexibility, the Clinic might se-lect more clients with problems that require students to understandthe interactions of multiple factors and contingencies.

V. ASSESSMENT OF COMPETENCY AND THE ABA'sPROPOSED STANDARDS

The ABA's Proposed Standards would require law schools to reg-ularly and continuously assess student competency in learning out-comes. 162 We support such a requirement. As we discussed in theprevious section, assessment can be a valuable tool for guiding studentattainment of competency when it is used to give feedback to studentsor to shape course design. Assessment is only useful for guiding stu-dents to competency, however, if it accurately measures their abilities.There is a marked difference in the nature-and value-of what canbe learned about competency in skills between assessment of perform-ance and evaluation of reasoning in practice.

For example, in experiential legal education, a typical subject forassessment is the initial interview with a client, in which the lawyermeets the client for the first time to begin the representation. An as-sessment based on performance would evaluate elements such as the

161 See Krieger & Martinez, supra note 106, at 127-34.162 Draft Report, supra note 13, at Standard 306.

[Vol. 19:251

Performance Isn't Everything

student's greeting of the client, whether the student made the clientfeel comfortable, how effectively the student applied the T-funneltechnique of asking open-ended questions and then narrowing themdown, how the interview was organized, the appropriateness of the"small talk" phase of the meeting, the cultural awareness and sensitiv-ity of the student, the discovery of any important deadlines, the de-gree to which client and lawyer know what happens next, and settingthe time for the next meeting.163

In contrast, an assessment of the same interview focused on thestudent's reasoning would look at very different elements, such as:What were the student's goals in the interview? Why did the studentask a particular question at a particular time? What was the studentthinking when the client said, "I just want justice"? How did the stu-dent attend to multiple, competing, interdependent factors such as theclient's goals, the relevant law, and the interests and resources of theother party in the case? What legal theories was the student consider-ing while talking to the client? How did the student conceive of herrole in the attorney-client relationship that was developing?

From the performance-based evaluation, we can learn a greatdeal about the student's ability to conduct a coherent interview or tobehave appropriately with a client. Indeed, skills such as these are notinsignificant elements of practice. For example, the lawyer who is notable to respond to cultural differences in a client interview is bound tohave problems with clients. The lawyer who cannot interview a clientin an organized manner will inevitably miss important information.We have no problem with teaching students to use techniques such asthe T-funnel interview. In fact, we both teach the T-funnel to ourstudents.

Nevertheless, an evaluation of a student's ability to conduct a T-

funnel interview is of limited value in assessing whether the student isprogressing toward expertise, and it should not be used as the sole orprimary measure of competency in interviewing. The fact that a stu-dent follows perfect T-funnel protocol does not indicate that she hasacquired the underlying schemas to understand the legal claims of theclient, that she has started to consider the multiple variables at play inthe case, or that she has any insights into novel issues that are raised.Performance-based assessments capture only the surface elements ofpractice, rather than the deeper reasoning processes that are central toexpert practice.

From an assessment of student reasoning, we can understand the

student's thinking as she works with the lawyering problem. By focus-

163 See supra notes 134-40 and accompanying text.

Fall 2012]

CLINICAL LAW REVIEW

ing on the reasoning process, instructors can see whether the studentis developing and applying schemas, demonstrating cognitive flexibil-ity, or exhibiting the ability to identify novel situations. These aremore significant elements of competency in practice than simple per-formance skills. Therefore, to assess competency in practical skills, lawschools must implement assessment methods that target students'ability to reason in practice, not just their ability to perform.

We fear, however, that the ABA and schools moving toward afocus on learning outcomes will rely too heavily on performance-based assessment, given its relative ease of administration and thegrowing number of voices that support this form of assessment. Asdiscussed previously, Carnegie suggests that assessment of compe-tency in lawyering skills should focus on performance elements.164

Moreover, the small amount of post-Carnegie scholarship that ad-dresses assessment of student competency in practice has tended toframe student learning in terms of performance skill and to advocatethe use of performance-based rubrics for assessment. 165 Although theamount of this scholarship is limited, a performance-based approachto assessment of competency in skills seems to be prevailing amonglegal scholars.

There is no doubt that performance-focused assessment methodssuch as the Best Practices communication skills checklist, 166 or thestandardized patient method endorsed by Carnegie, 167 have appeal foreducators. Checklist assessments can be easily administered and

164 SULLIVAN ET AL., supra note 9, at 174.165 See, e.g., Ann Marie Cavazos, The Journey Toward Excellence in Clinical Legal Edu-

cation: Developing, Utilizing and Evaluating Methodologies for Determining and Assessingthe Effectiveness of Student Learning Outcomes, 40 Sw. L. REV. 1, 37 (2010) (proposing acomprehensive written evaluation of student performance as well as an oral examinationthat primarily focuses on student performance in terms of communications skills, bothscored with a predetermined rubric); Jerry R. Foxhoven, Beyond Grading: Assessing Stu-dent Readiness to Practice Law, 16 CLIN. L. REV. 335, 344-8 (2010) (describing assessmentof student competency for practice based on assessment of student work product and stu-dent ability to perform various lawyering skills and practices); Lori A. Roberts, AssessingOurselves: Confirming Assumptions and Improving Student Learning by Efficiently andFearlessly Assessing Student Learning Outcomes, 3 DREXEL L. REV. 457, 479-83 (2010)(proposing assessment methods for appellate oral advocacy based exclusively on observa-tion of student performance and application of a scoring rubric). But see AnthonyNiedwiecki, Teaching for Lifelong Learning: Improving the Metacognitive Skills of LawStudents through More Effective Formative Assessment Techniques, 40 CAP. U. L. REV. 149(2012) (arguing that assessment should move away from a focus on the end product such asan oral argument or a negotiation toward a focus on the underlying reasoning processinvolved in the production as a means of improving students' ability to learn from theirexperiences).

166 STUCKEY ET AL., supra note 9, at 246-7. See supra notes 135-37 and accompanyingtext.

167 SULLIVAN ET AL., supra note 9, at 175. See supra notes 139-40 and accompanyingtext.

[Vol. 19:251

Performance Isn't Everything

scored in a uniform, straightforward fashion that minimizes subjectivejudgment. By contrast, assessing student reasoning is likely to be morechallenging and time-consuming than a checklist-type assessment.

In the think-aloud interviews, student thinking tends to be non-linear, looping back on itself repeatedly, and it is peppered with fillersand non-sequiturs. Questions are raised and then revisited again onlymuch later, if at all. That, of course, is the nature of cognition. Studentreasoning, like all human thinking, is messy. It resists easy decomposi-tion into discrete component parts. Instead, it must be analyzed as awhole to identify various reasoning processes glimpsed piecemealthroughout each interview. This analysis requires a deep understand-ing of the domain, as well as a firm grasp of expert reasoning in prac-tice in that domain.

The challenges of using assessment methods that focus on studentreasoning, however, should not deter the ABA or law schools fromusing this type of method. Only by focusing on essential elements ofexpertise, such as reasoning in practice, can assessments reveal stu-dents' progress toward becoming experts themselves, and not merelytheir ability to imitate experts. Because they target important indica-tors of competency in practice, educators and the ABA alike shouldembrace assessments focused on student reasoning. They should re-ject the notion that skilled performance is the sole acceptable hall-mark of competency in skills. Instead, they should understand thatmeaningful assessment of student learning in experiential educationmust focus on students' ability to reason in practice.

CONCLUSION

This is a seminal moment for experiential legal education. Thechorus calling for reform is swelling, and the ABA stands on the vergeof requiring law schools to prepare every student to be competent inskills when they graduate. These circumstances create an opportunityfor law schools to give careful consideration to the sort of practicaleducation that they will provide their students.

With the loud voice of the Carnegie Report describing expertiseprimarily in terms of performance and the ability to follow protocolsor imitate expert techniques, it might be easy simply to follow Car-negie's recommendations for teaching and assessing students. Currentscientific research, however, disputes Carnegie's view of expertise, in-stead offering important, empirically validated insights into how bestto prepare law students to become expert practitioners.

Cognitive science research into the nature of expertise demon-strates that reasoning, not arational application of "toolkit" tech-niques to familiar situations, is the hallmark of expert practice.

Fall 20121

CLINICAL LAW REVIEW

Focusing on performance misses the mark. The implications for expe-riential education are significant. Course design must give studentsopportunities to develop the ability to reason in practice, and not sim-ply to learn different expert techniques.

Moreover, this research demonstrates that providing useful feed-back to students requires assessment methods that similarly focus onstudent reasoning. To properly assess student reasoning, evaluationmust try to get into the heads of students in practice. We do not pre-tend that the think-aloud interview method we have proposed is thelast word in assessment of student reasoning, but the information wehave been able to extract from our interviews thus far suggests thatthis type of assessment can be valuable in evaluating students' acquisi-tion of expertise. The results so far invite further research to refine thethink-aloud methodology and develop additional methods for assess-ing reasoning.

Our law students are unlikely to be called upon to land a fullplane on a river in mid-winter, but they will be called upon to serveclients in complex, challenging situations without precedent. While weall hope that our students will respond to difficult client problems withthe same expertise that Captain Sullenberger showed, such a result isonly possible if we accurately understand the nature of expertise andgear our teaching to developing that expertise in our students.

[Vol. 19:251

Performance Isn't Everything

APPENDIX

Community & Economic Development Hypothetical Scenarios

1. Luscious Landscaping

You represent a group of five men who do landscaping work lo-cally. You are helping them to start a business venture together. Youjust met with three of the five men at your office where one of them,Esteban, started off by telling you "We want to form a Corporation-what do we need to do?"

Although each of them has been successfully doing small-scalework for some time now, they want to work together to allow them todo larger-scale projects, develop a recognizable brand name, and re-duce the cost of supplies by buying in bulk. Another reason for enter-ing into this venture together is to try to fight the exploitation thateach of them, as Latino men, has seen within the landscaping world.They will make sure that Latino laborers are paid decent wages, pro-vide decent employment opportunities for unskilled laborers, and seta good example of the value of hard work for the increasingly disaf-fected youth of their Latino community.

They all want to run the business jointly, and they intend to usetheir own funds as an initial investment. In the future, they want torecruit more members (as long as they can assure that the quality ofthe work will not suffer) and also to look for outside financing. Be-cause of the low profit margins in this market, they want to keep theirinitial costs down as much as possible while still reducing tax liability.In addition, they feel that they do not want to be in the position ofasking for (or providing) information about immigration status. Theyhave told you that they want to use the name "Luscious Landscap-ing," and they are looking to get this done immediately.

What will you do now?

2. Rivertown Redevelopment & Revitalization

You represent a group of three successful entrepreneurs whogrew up in the community of Rivertown, one of the poorest and mostrun down areas of the county, with a high unemployment rate andterrible housing options. You are helping them to start a project torevitalize the business and economic climate of Rivertown. You justmet with members of the group at your office.

Each of them has been successful with individual and joint busi-ness ventures around the area, and now they want to take their talentsback to the community of Rivertown. Decades ago, Rivertown was athriving working-class neighborhood, with a bustling and modern

Fall 2012]

CLINICAL LAW REVIEW

Main Street that was home to locally-owned small businesses, employ-ing hundreds of local residents and bringing lots of capital into thearea. Now there's just an abandoned main street with a couple liquorstores and a barely surviving small hardware shop, surrounded by di-lapidated housing and a lot of unemployed residents.

The group wants to create a project that will improve the eco-nomic climate in Rivertown and attract more new businesses. The ulti-mate goal is to provide employment for local residents, increase localincomes, set a positive example for local youth, decrease crime, im-prove the area's political power, and help restore Rivertown to itsglory days. To begin, they envision doing things like providing trainingfor the local workforce, developing a unified faqade for thestorefronts, installing streetlights, creating a pedestrian zone with side-walk cafes, and improving security.

As part of the initial phase of this initiative, the group wants tolease one of the empty storefronts to open a grocery store that will sellhealthy and affordable food. Currently, Rivertown has no grocerystores, just a few delis and a bunch of fast food chains. They believethat a successful grocery will show other retailers that Rivertown is agood place to do business and will help speed the project along.

They all want to control the project jointly. They intend to usetheir own funds as an initial investment, but they think they will needdonations, outside investors, and/or government grants to make theirvision a reality. They have big dreams for the place they came from,but no experience in this area, so they came to you for counsel on howto proceed.

What will you do now?

[Vol. 19:251