a comparison of responses from attorneys and from … · from law librarians on legal research...

79
A Comparison of Responses from Attorneys and from Law Librarians on Legal Research Practices and Expectations for New Associates June 2014 Conducted and Prepared by the ALL-SIS Task Force on Identifying Skills and Knowledge for Legal Practice SUSAN NEVELOW MART (CHAIR) Colorado Law, University of Colorado at Boulder SHAWN NEVERS (VICE-CHAIR) Brigham Young University, J. Reuben Clark Law School TONI AIELLO Hofstra University, Maurice A. Dean School of Law NANCY TALLEY Rutgers University School of Law, Camden NOLAN WRIGHT Southern Illinois University School of Law JASON ZARIN Georgetown University Law Center

Upload: dolien

Post on 19-Aug-2018

214 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: A Comparison of Responses from Attorneys and from … · from Law Librarians on Legal Research Practices and Expectations for New Associates . ... F. Case Digests ... legal service

A Comparison of Responses from Attorneys and from Law Librarians on Legal Research

Practices and Expectations for New Associates

June 2014

Conducted and Prepared by the ALL-SIS Task Force on Identifying Skills and Knowledge for Legal Practice

SUSAN NEVELOW MART (CHAIR) Colorado Law, University of Colorado at Boulder

SHAWN NEVERS (VICE-CHAIR)

Brigham Young University, J. Reuben Clark Law School

TONI AIELLO Hofstra University, Maurice A. Dean School of Law

NANCY TALLEY

Rutgers University School of Law, Camden

NOLAN WRIGHT Southern Illinois University School of Law

JASON ZARIN

Georgetown University Law Center

Page 2: A Comparison of Responses from Attorneys and from … · from Law Librarians on Legal Research Practices and Expectations for New Associates . ... F. Case Digests ... legal service

TOC

- 2 -

CONTENTS

INTRODUCTION........................................................................................................................... 4

I. QUESTIONS POSED: BEGINNING THE RESEARCH PROCESS ........................................... 6 A. Start by Asking a Fellow Attorney for Research Advice ........................................ 6 B. Start with a Google Search .................................................................................... 7 C. Start by Looking in a Statutory Database .............................................................. 8 D. Start by Consulting In-House Work Product.......................................................... 9 E. Start by Looking in a Secondary Source .............................................................. 10 F. Start in a Case Law Database .............................................................................. 10 G. Start by Consulting a Subject-specific Practice Guide ........................................ 10

II. QUESTIONS POSED: THE RESEARCH PROCESS ............................................................ 11 A. Case Digest or Online Equivalent to Find Topics and Key Numbers .................. 11 B. Headnotes and Key Numbers in Cases to Find Other Relevant Cases ................ 12 C. Follow Citations in a Case ................................................................................... 13 D. Index or Table of Contents When Researching Statutes Online ........................... 14 E. Follow Citations in a Secondary Source .............................................................. 14 F. Terms and Connectors Searching ........................................................................ 15 G. Follow Citations in an Annotated Code ............................................................... 16

III. QUESTIONS POSED: REPORTING RESOURCE USAGE ................................................... 17 A. Print Materials ..................................................................................................... 17 B. Free Internet Resources ....................................................................................... 18 C. Fee-Based Databases ........................................................................................... 19

IV. QUESTION POSED: USING RESEARCH TOOLS ............................................................. 20 A. Legal Treatises ..................................................................................................... 20 B. Practice Guides .................................................................................................... 21 C. Legal Encyclopedias............................................................................................. 22 D. Law Review/Law Journal Articles ........................................................................ 23 E. American Law Reports (ALR) .............................................................................. 24 F. Case Digests ......................................................................................................... 25 G. Restatements ......................................................................................................... 26 H. Shepard’s/KeyCite for Case Validation ............................................................... 27 I. Shepard’s/KeyCite for Further Research ............................................................. 28 J. Looseleaf Services ................................................................................................ 29 K. Sample Legal Forms ............................................................................................. 30 L. Litigation Resources ............................................................................................. 31 M. Transactional Resource Guides ........................................................................... 32

V. QUESTION POSED: USING FREE INTERNET SOURCES ................................................ 33 A. Google .................................................................................................................. 33 B. Google Scholar ..................................................................................................... 34 C. Court Websites ..................................................................................................... 35 D. Government Legislative Websites ........................................................................ 36 E. Government Agency Websites .............................................................................. 37 F. Legal Blogs ........................................................................................................... 38 G. Law Firm Websites ............................................................................................... 39 H. Law School Library Websites ............................................................................... 40 I. Non-profit Organization Websites ........................................................................ 41

VI. QUESTION POSED: RESEARCH SKILLS OF RECENT LAW SCHOOL GRADUATES ......... 42

Page 3: A Comparison of Responses from Attorneys and from … · from Law Librarians on Legal Research Practices and Expectations for New Associates . ... F. Case Digests ... legal service

TOC

- 3 -

A. Develop Effective Research Plans ........................................................................ 42 B. Using Secondary Sources Effectively ................................................................... 43 C. Critical Thinking in Evaluating Relevance .......................................................... 43 D. Develop Appropriate Search Protocols................................................................ 44 E. Research Case Law .............................................................................................. 45 F. Research Statutes ................................................................................................. 46 G. Research Regulations ........................................................................................... 47 H. Research Administrative Decisions ...................................................................... 48 I. Research Legislative History ................................................................................ 49 J. Understand Difference Between Statutes and Regulations .................................. 50 K. Update Legal Sources with a Citator ................................................................... 51 L. Perform Cost-Effective Research ......................................................................... 52 M. Use Westlaw Efficiently ........................................................................................ 53 N. Use Lexis Efficiently ............................................................................................. 54 O. Use an Online Service Other than Westlaw or Lexis ........................................... 54 P. Research Pleadings, Motions, and Other Court Documents................................ 55 Q. Know When to Stop Researching ......................................................................... 56 R. Finding Non-Legal Information ........................................................................... 57

APPENDIX A: DEMOGRAPHIC DETAILS OF THE COMBINED DATA SETS ... ……..……………58 APPENDIX B: SUMMARY TABLES………………………………………………… ................. 60 APPENDIX C: SURVEY INSTRUMENTS .............................................................. ………...........79

Page 4: A Comparison of Responses from Attorneys and from … · from Law Librarians on Legal Research Practices and Expectations for New Associates . ... F. Case Digests ... legal service

TOC

- 4 -

Introduction In July 2011, the Academic Law Libraries Special Interest Section (“ALL-SIS”) of the American Association of Law Libraries appointed a Task Force on Identifying Skills and Knowledge for Legal Practice. The Task Force was charged with “identify[ing] the current and future research skills that law school graduates need to succeed in legal practice.” The charge observed that “[t]his information will help law schools determine how to develop their curriculum to meet the research needs of their graduates.” To fulfill its charge, the Task Force created a survey that focused on the legal research practices of attorneys. In 2012, two versions of the survey were distributed, one to law firm librarians and one to practicing attorneys. In 2013 the Task Force published the results of the attorney survey.1 Upon examination of the results of both surveys, it became clear that there were significant differences between the responses of the attorneys and the librarians. This Report compares the responses of the law firm librarians with the responses of similarly situated attorneys. To facilitate comparison of attorney and librarian responses, a composite data set was created and restricted by practice setting and office size to ensure the comparison would be between similarly situated individuals.2 A total of 280 respondents, 150 librarians and 130 attorneys, were included in the composite data set.3 The Task Force used standard statistical tests for the data generated from the surveys and has reported the results below, with graphics illustrating significant differences.4 A statistically significant difference between attorney and librarian responses was identified for almost all of the common questions examined. In most instances, these relationships ranged from statistically strong to extremely strong. It is important to note, however, that while there were many statistically significant differences between the attorney and librarian responses, the Task Force cannot state any evidence-based reasons for the differences, but have, in some instances, suggested one or more possible explanations for these differences.

1 For a full discussion, please see A Study of Attorneys’ Legal Research Practices and Opinions of New

Associates’ Research Skills, ALL-SIS Task Force on Identifying Skills and Knowledge for Legal Practice, June 2013, http://www.aallnet.org/sections/all/storage/committees/practicetf/final-report-07102013.pdf. The survey instruments and detailed data about survey distribution are also included in the June 2013 report.

2 Only respondents reporting more than 50 attorneys in their office and who worked in private practice, government agencies, or the judiciary were included in the composite data set. Respondents identifying their setting as in-house/corporate counsel, legal service corporation, public interest, military, law school, other legal education, non-profit organization, consultant, retired, or other, were not included. Subcategories of private practice (litigation, transactional, mixed) and government agency (federal, state, local) were combined into single categories to facilitate statistical analysis, given the smaller sample size.

3 Further demographic detail of the composite data set can be found in Appendix A. 4 The Pearson Chi Square test was used to assess the possible relationships between profession and the

responses to the questions about research practices and opinions about the research knowledge and abilities of recent law school graduates included on both survey instruments. That is the standard test for significance between two categorical variables, where one or both are nominal, as the variable “profession” is in this instance The Cramer’s V statistic was used as a measure of strength of relationship, together with the Lambda statistic.

Page 5: A Comparison of Responses from Attorneys and from … · from Law Librarians on Legal Research Practices and Expectations for New Associates . ... F. Case Digests ... legal service

TOC

- 5 -

The first portion of the survey asked respondents how often attorneys used a particular legal research method, tool, or resource.5 Generally speaking, librarians were significantly more likely to report that their attorneys used these methods, tools, or resources “very frequently,” “frequently,” or at least “occasionally,” than was reported by the attorneys included in the data set. In many of these and other questions, a significantly greater proportion of attorneys reported that they “never” or only “rarely” use the method, tool, or resource at issue.

The second portion of the survey asked respondents how well recent law school graduates performed certain legal research skills. Generally speaking, the librarian respondents reported significantly more negative opinions about the research skills of recent law school graduates than reported by similarly situated attorneys. On question after question, a significantly greater proportion of the librarians rated the ability of recent graduates to carry out the research task at issue as “unacceptable,” “poor” or both, than did the attorneys. In many cases, we also see that a significantly greater proportion of the attorneys rated the recent graduates as performing “very well,” “moderately well,” or at least “adequately,” than was reported by librarians. As an aid to readers who would like to be able to quickly look over the data obtained from the two surveys (practitioner and librarian), or the comparison data aggregating responses from the subset of similarly situated librarians and practitioners, the Task Force has included summary tables for each of the two surveys and the comparison data set in Appendix B.

5 Librarians were asked how often they thought attorneys they worked with used a particular method, tool,

or resource.

Page 6: A Comparison of Responses from Attorneys and from … · from Law Librarians on Legal Research Practices and Expectations for New Associates . ... F. Case Digests ... legal service

TOC

- 6 -

I. Questions Posed: Beginning The Research Process

“When beginning research, how often do you …” | “When the attorneys you work with begin their research, how often do they …” [Q6-practitioner survey; Q8-librarian survey]

A. Start by asking a fellow attorney (218 respondents):

o There is a strong statistically significant relationship between profession and responses to this question.

o Attorney respondents were significantly more likely than librarians describing their attorneys’ practices to report that they “never” begin their research with this approach, or “rarely” do so (i.e. the proportion of attorney respondents who selected each of those answers was significantly greater than the proportion of librarian respondents who did so), as detailed in the chart below.

o Librarian respondents were significantly more likely to report that their attorneys “frequently” begin their research this way (i.e. the proportion of librarian respondents who selected that answer was significantly greater than the proportion of attorney respondents who did so), as detailed in the chart below.

o Note that nearly two thirds of the full cross-section of attorneys who responded to the practitioner survey reported that they “never” (8.9%), “rarely” (27.3%) or only “occasionally” (28.8%) begin a research project by asking another attorney for research advice. This is interesting, as nearly a quarter of responding attorneys across all practice types frequently start by asking a fellow attorney. One possibility is that half of the attorneys who start out by asking a fellow attorney wind up visiting a librarian.

5.6%

22.4% 23.2%

0.0%

8.6%

45.2%

0.00%5.00%

10.00%15.00%20.00%25.00%30.00%35.00%40.00%45.00%50.00%

Never Rarely Frequently

Start by asking fellow attorney

Attorneys

Librarian Observers

Page 7: A Comparison of Responses from Attorneys and from … · from Law Librarians on Legal Research Practices and Expectations for New Associates . ... F. Case Digests ... legal service

TOC

- 7 -

B. Start with Google (228 respondents):

o There is an extremely strong statistically significant relationship between profession and responses to this question.

o Attorney respondents were significantly more likely than librarians describing their attorneys’

practices to report that they “never” begin their research with this approach, “rarely” do so, or only “occasionally” (i.e. the proportions of attorney respondents who selected one of those answers was significantly greater than the proportion of librarian respondents who did so), as detailed in the chart below.

o Librarian respondents were significantly more likely to report that their attorneys begin their research this way “frequently” or “very frequently” (i.e. the proportion of librarian respondents who selected that answer was significantly greater than the proportion of attorney respondents who did so), as detailed in the chart below.

o Note that a combined figure of just over half of the full cross-section of attorneys who responded to the practitioner survey reported that they start legal research with a Google search “occasionally” (30.2%) or “frequently” (21.1 %).

10.4%

25.6% 30.4%

21.6%

12.0%

0.0% 0.0%

15.5%

46.6%

37.9%

0.00%5.00%

10.00%15.00%20.00%25.00%30.00%35.00%40.00%45.00%50.00%

Start with Google

Attorneys

Librarian Observers

Page 8: A Comparison of Responses from Attorneys and from … · from Law Librarians on Legal Research Practices and Expectations for New Associates . ... F. Case Digests ... legal service

TOC

- 8 -

C. Start with statutory database (221 respondents):

o There is a moderate statistically significant relationship between profession and responses to this question.

o Attorney respondents were significantly more likely than librarians describing their attorneys’ practices to report that they “never” begin their research with this approach, as detailed in the chart below.

o Librarian respondents were significantly more likely to report that their attorneys begin their research this way “occasionally,” as detailed in the chart below.

o Note that more than half of the full cross-section of attorneys who responded to the practitioner survey reported beginning research in a statutory database “frequently” (36.4%) or “very frequently” (15.7%).

8.9%

29.3%

1.0%

42.9%

0.00%

5.00%

10.00%

15.00%

20.00%

25.00%

30.00%

35.00%

40.00%

45.00%

50.00%

Never Occasionally

Start with statutory database

Attorneys

Librarian Observers

Page 9: A Comparison of Responses from Attorneys and from … · from Law Librarians on Legal Research Practices and Expectations for New Associates . ... F. Case Digests ... legal service

TOC

- 9 -

D. Start by consulting work product (228 respondents):

o There is a moderate statistically significant relationship between profession and responses to this question.

o Attorney respondents were significantly more likely than librarians describing their attorneys’

practices to report that they begin their research with this approach “very frequently,” as detailed in the chart below.

o Librarian respondents were significantly more likely to report that their attorneys begin their research this way “frequently” or very “frequently,” as detailed in the chart below.

o This is one of the relatively few questions where there was a significant difference between attorney and librarian respondents where the attorneys reported using a resource or approach more “frequently” than the librarians reported as the practice of the attorneys they work with.

o Note that the full cross-section of attorneys who responded to the practitioner survey were divided into three fairly even groups in their responses to this question: Roughly one-third (34.7%) reported beginning research with in-house documents “frequently” or “very frequently.” Roughly one-third (34.3%) reported “rarely” or “never” doing so. The remainder, at just under one-third, reported doing so “occasionally.”

o These results highlight the importance of in-house work product.

21.6% 22.4%

35.0%

7.8%

0.00%

5.00%

10.00%

15.00%

20.00%

25.00%

30.00%

35.00%

40.00%

Frequently Very Frequently

Start by consulting work product

Attorneys

Librarian Observers

Page 10: A Comparison of Responses from Attorneys and from … · from Law Librarians on Legal Research Practices and Expectations for New Associates . ... F. Case Digests ... legal service

TOC

- 10 -

E. Start with a secondary source (234 respondents):

o No statistically significant relationship was observed between profession and responses to this question.

o Note that less than one-third of the full cross-section of attorneys who responded to the practitioner

survey reported that they begin their research with secondary sources “frequently” (25%) or “very frequently” (6.1%). Nearly one-tenth (9.3%) reported that they “never” do so.

F. Start with case law database (235 respondents):

o There is a very strong statistically significant relationship between profession and responses to this

question.

o Attorney respondents were significantly more likely than librarians describing their attorneys’ practices to report that they “never” begin their research with this approach, or only “rarely” do so, as detailed in the chart below.

o Librarian respondents were significantly more likely to report that their attorneys begin their research this way “frequently,” as detailed in the chart below.

o Note that over half of the full cross-section of attorneys who responded to the practitioner survey reported starting research in a case law database “frequently” (30.7%) or “very frequently” (25.9%).

G. Start with a subject specific practice guide (230 respondents):

o No statistically significant relationship was observed between profession and responses to this question.

o Note that more than one-third of the full cross-section of attorneys who responded to the practitioner survey reported beginning with practice guides “frequently” (23.5%) or “very frequently” (10.2%). Combining those two groups with those reporting doing so “occasionally” (23.5%) adds up to more than half of the attorneys who responded to the practitioner survey.

6.3%

19.8%

26.2%

0.0% 3.7%

53.2%

0.00%

10.00%

20.00%

30.00%

40.00%

50.00%

60.00%

Never Rarely Frequently

Start with caselaw database

Attorneys

Librarian Observers

Page 11: A Comparison of Responses from Attorneys and from … · from Law Librarians on Legal Research Practices and Expectations for New Associates . ... F. Case Digests ... legal service

TOC

- 11 -

II. Questions Posed: The Course of Conducting Research

“When you are researching an issue for your practice, how often do you …” | “When the attorneys you work are conducting their research/ how often do they …” [Q7-practitioner survey; Q9-librarian survey]

A. Use a digest or online equivalent to find a topic & key number (238 respondents):

o There is a very strong statistically significant relationship between profession and responses to this question.

o Attorney respondents were significantly more likely than librarians describing their attorneys’ practices to report that they “never” use this resource or approach, and also were significantly more likely at the other extreme to report doing so “very frequently,” as detailed in the chart below.

o Librarian respondents were significantly more likely to report that their attorneys use this resource or approach “occasionally,” as detailed in the chart below.

o Note that more of the full cross-section of attorneys who responded to the practitioner survey reported that they “never” use a digest or online equivalent to find a topic and key number (22.8%) than those who report doing so “frequently” (14.0%) or “very frequently” (5.4%) combined. More than half reported they “rarely” (36.7%) or only “occasionally” (21.0%) do so.

25.4%

17.5%

7.1% 5.4%

36.6%

0.9% 0.00%

5.00%

10.00%

15.00%

20.00%

25.00%

30.00%

35.00%

40.00%

Never Occasionally Very Frequently

Use a digest or online equivalent to find a topic & key number

Attorneys

Librarian Observers

Page 12: A Comparison of Responses from Attorneys and from … · from Law Librarians on Legal Research Practices and Expectations for New Associates . ... F. Case Digests ... legal service

TOC

- 12 -

B. Use headnote & key numbers in case (238 respondents):

o There is a strong statistically significant relationship between profession and responses to this question.

o Attorney respondents were significantly more likely than librarians describing their attorneys’ practices to report that they “never” use this resource or approach, and also were significantly more likely at the other extreme to report doing so “very frequently,” as detailed in the chart below.

o Librarian respondents were significantly more likely to report that their attorneys use this resource or approach “occasionally,” as detailed in the chart below.

o Note that nearly 17% of the full cross-section of attorneys who responded to the practitioner survey reported “never” using headnotes and key numbers in cases to locate other relevant cases. Approximately half of respondents reported doing so “rarely” (26.9%) or “occasionally” (24.1%).

o Note that attorneys are using headnotes and key numbers more than “digests.” See the previous chart on digests. Since headnotes and key numbers are the online equivalent of digests, it is possible attorneys do not realize they are using a “digest.”

o One possibility for this result is that attorneys are using this research technique only “occasionally” or “never” because attorneys are using electronic and internet-based resources.

17.6%

27.2%

11.2%

2.7%

40.7%

2.7%

0.00%

5.00%

10.00%

15.00%

20.00%

25.00%

30.00%

35.00%

40.00%

45.00%

Never Occasionally VeryFrequently

Use headnote & key numbers in a case

Attorneys

Librarian Observers

Page 13: A Comparison of Responses from Attorneys and from … · from Law Librarians on Legal Research Practices and Expectations for New Associates . ... F. Case Digests ... legal service

TOC

- 13 -

C. Follow citations in case (242 respondents):

o There is an extremely strong statistically significant relationship between profession and responses to this question.

o Attorney respondents were significantly more likely than librarians describing their attorneys’ practices to report that they “never” use this resource or approach, or only “rarely,” and also were significantly more likely at the other extreme to report doing so “very frequently,” as detailed in the chart below.

o Librarian respondents were significantly more likely to report that their attorneys use this resource or approach “occasionally,” or “frequently,” as detailed in the chart below.

o Note that nearly three-fourths of the full cross-section of attorneys who responded to the practitioner survey reported following citations in cases “frequently” (38.7%) or “very frequently” (33.7%).

o The librarian figures differ more in emphasis than in numbers. Librarians think attorneys do this frequently, while attorneys believe they do this very frequently.

6.3% 11.0% 9.4%

38.6% 34.6%

0.0% 2.6%

29.6%

53.9%

13.9%

0.00%

10.00%

20.00%

30.00%

40.00%

50.00%

60.00%

Follow citations in a case

Attorneys

Librarian Observers

Page 14: A Comparison of Responses from Attorneys and from … · from Law Librarians on Legal Research Practices and Expectations for New Associates . ... F. Case Digests ... legal service

TOC

- 14 -

D. Use index or table of contents with online statutory database (230 respondents):

o There is an extremely strong statistically significant relationship between profession and responses to this question.

o Attorney respondents were significantly more likely than librarians describing their attorneys’ practices to report that they “frequently” use this resource or approach, or “very frequently” do so, as detailed in the chart below.

o Librarian respondents were significantly more likely to report that their attorneys use this resource or approach “rarely,” or only “occasionally,” as detailed in the chart below.

o Note that more than half of the full cross-section of attorneys who responded to the practitioner survey reported using an index or table of contents “frequently” (33.2%) or “very frequently” (19.4%) when researching statutes online, with “frequently” representing the highest proportion of respondents of any category. Only 5.6% reported “never” using an index or table of contents when researching statutes online.

E. Follow citations in secondary sources (238 respondents):

o No statistically significant relationship was observed between profession and responses to this question.

o Note that more than half of the full cross-section of attorneys who responded to the practitioner survey reported following citations in secondary sources “frequently” (29.9%) or “very frequently” (16.2%). The largest percentage of respondents said they do so “occasionally” (30.3%), while 17.7% do so “rarely” and 6.0% “never” do so.

13.6%

30.4% 29.6%

16.8%

36.2%

43.8%

16.2%

0.0% 0.00%5.00%

10.00%15.00%20.00%25.00%30.00%35.00%40.00%45.00%50.00%

Rarely Occasionally Frequently VeryFrequently

Use index or table of contents with online statutory database

Attorneys

Librarian Observers

Page 15: A Comparison of Responses from Attorneys and from … · from Law Librarians on Legal Research Practices and Expectations for New Associates . ... F. Case Digests ... legal service

TOC

- 15 -

F. Use terms and connectors (240 respondents):

o There is a very strong statistically significant relationship between profession and responses to this question.

o Attorney respondents were significantly more likely than librarians describing their attorneys’ practices to report that they “very frequently” use this resource or approach, as detailed in the chart below.

o Librarian respondents were significantly more likely to report that their attorneys “frequently” use this resource or approach, as detailed in the chart below.

o Note that more than one-third (37.9%) of the full cross-section of attorneys who responded to the practitioner survey reported using terms and connectors searching “very frequently.” More than one-quarter (27.4%) reported using them “frequently,” meaning that nearly two-thirds of respondents reported using terms and connectors searching “frequently” or “very frequently.” Only 6.6% report that they never do so.

o A significant percentage of the responses for “frequently” and “very frequently” from both groups indicate that attorneys continue to use this method of searching when conducting research.

19.7%

48.0% 47.8%

18.6%

0.00%

10.00%

20.00%

30.00%

40.00%

50.00%

60.00%

Frequently Very Frequently

Use terms & connectors

Attorneys

Librarian Observers

Page 16: A Comparison of Responses from Attorneys and from … · from Law Librarians on Legal Research Practices and Expectations for New Associates . ... F. Case Digests ... legal service

TOC

- 16 -

G. Follow citations in annotated code (239 respondents):

o There is a moderately strong statistically significant relationship between profession and responses to this question.

o Attorney respondents were significantly more likely than librarians describing their attorneys’ practices to report that they “never” use this resource or approach, or only “rarely,” as detailed in the chart below.

o Librarian respondents were significantly more likely to report that their attorneys “frequently” use this resource or approach, as detailed in the chart below.

o Note that the majority of the full cross-section of attorneys who responded to the practitioner survey reported following citations in an annotated code “frequently” (32.7%) or “very frequently” (19.7%). A small portion (6.5%) reported that they “never” follow citations in an annotated code to find relevant material, while those that did so “occasionally” outnumber those that did so “rarely” by nearly two to one.

7.2%

19.2%

24.0%

0.9%

5.3%

38.6%

0.00%

5.00%

10.00%

15.00%

20.00%

25.00%

30.00%

35.00%

40.00%

45.00%

Never Rarely Frequently

Follow citations in annotated code

Attorneys

Librarian Observers

Page 17: A Comparison of Responses from Attorneys and from … · from Law Librarians on Legal Research Practices and Expectations for New Associates . ... F. Case Digests ... legal service

TOC

- 17 -

III. Questions Posed: Reporting Resource Usage

When performing research for your practice, how often do you …” | “When engaged in legal research, how often do the attorneys you work with …” [Q8-practioner survey; Q10-librarian survey]

A. Use print materials (242 respondents):

o There is a moderate statistically significant relationship between profession and responses to this question.

o Attorney respondents were significantly more likely than librarians describing their attorneys’ practices to report that they “never” use this resource or approach, as detailed in the chart below.

o Note that more than forty percent of the full cross-section of attorneys who responded to the practitioner survey reported that they use print “frequently” (26.9%) or “very frequently” (15.4%). More than one-third reported doing so “occasionally” (33.9%). Only 5.1% reported “never” doing so.

o One possibility for this result is that attorneys continue, to a certain extent, to use print resources when conducting research.

4.8%

0.0% 0.00%

1.00%

2.00%

3.00%

4.00%

5.00%

6.00%

Never

Use print materials

Attorneys

Librarian Observers

Page 18: A Comparison of Responses from Attorneys and from … · from Law Librarians on Legal Research Practices and Expectations for New Associates . ... F. Case Digests ... legal service

TOC

- 18 -

B. Use free internet sources (239 respondents):

o There is a moderately strong statistically significant relationship between profession and responses to this question.

o Attorney respondents were significantly more likely than librarians describing their attorneys’ practices to report that they “never” use this resource or approach, or only “rarely” do so, as detailed in the chart below.

o Librarian respondents were significantly more likely to report that their attorneys “frequently” use this resource or approach, as detailed in the chart below.

o Note that more than sixty percent of the full cross-section of attorneys who responded to the practitioner survey reported using free internet resources “frequently” (30.7%) or “very frequently (30.7%).” Another twenty-five percent reported doing so “occasionally.” Only 12.6% reported that “rarely” (9.7%) or “never” (2.9%) do so.

4.0%

27.8%

34.1%

0.0%

15.9%

54.0%

0.00%

10.00%

20.00%

30.00%

40.00%

50.00%

60.00%

Never Rarely Frequently

Use free internet resources

Attorneys

Librarian Observers

Page 19: A Comparison of Responses from Attorneys and from … · from Law Librarians on Legal Research Practices and Expectations for New Associates . ... F. Case Digests ... legal service

TOC

- 19 -

C. Use fee-based databases (241 respondents):

o There is a strong statistically significant relationship between profession and responses to this question.

o Attorney respondents were significantly more likely than librarians describing their attorneys’ practices to report that they “rarely” use this resource or approach, or only “occasionally” do so, as detailed in the chart below.

o Librarian respondents were significantly more likely to report that their attorneys “frequently” use this resource or approach, as detailed in the chart below.

o Note that more than two-thirds of the full cross-section of attorneys who responded to the practitioner survey reported using fee-based databases “frequently” (22.7%) or “very frequently” (44.3%). On the other hand, approximately 8% of respondents report “never” using fee-based databases, and nearly 10% “rarely” do so.

o Librarians were almost twice as likely as attorneys to respond that the attorneys they work with use fee-based databases “frequently.”

7.9%

17.5%

26.2%

0.0%

6.1%

45.2%

0.00%

5.00%

10.00%

15.00%

20.00%

25.00%

30.00%

35.00%

40.00%

45.00%

50.00%

Rarely Occasionally Frequently

Use fee based databases

Attorneys

Librarian Observers

Page 20: A Comparison of Responses from Attorneys and from … · from Law Librarians on Legal Research Practices and Expectations for New Associates . ... F. Case Digests ... legal service

TOC

- 20 -

V. Questions Posed: Reporting Resource Usage

“When performing legal research for your practice, how often do you utilize the following research tools?” | “When engaged in legal research, how often do the attorneys you work with use the following research tools?” [Q9-practitioner survey; Q11-librarian survey]

A. Treatises (240 respondents):

o There is a strong statistically significant relationship between profession and responses to this question.

o Attorney respondents were significantly more likely than librarians describing their attorneys’ practices to report that they “never” use this tool, or only “rarely” do so, as detailed in the chart below.

o Librarian respondents were significantly more likely to report that their attorneys use this tool, as detailed in the chart below.

o Note that nearly forty percent of the full cross-section of attorneys who responded to the practitioner survey reported that they “never” (10.2%) or “rarely” (28.8%) use legal treatises for research. Only slightly more than one-quarter of respondents report using legal treatises “frequently” (18.7%) or “very frequently” (8.5%), and one-third report using them “occasionally.”

o Librarians were almost twice more likely than attorneys to respond that the attorneys they work with use treatises “frequently.”

5.6%

20.0%

26.4%

0.0% 3.5%

45.2%

0.00%

5.00%

10.00%

15.00%

20.00%

25.00%

30.00%

35.00%

40.00%

45.00%

50.00%

Never Rarely Frequently

Use treatises

Attorneys

Librarian Observers

Page 21: A Comparison of Responses from Attorneys and from … · from Law Librarians on Legal Research Practices and Expectations for New Associates . ... F. Case Digests ... legal service

TOC

- 21 -

B. Practice guides (237 respondents):

o There is a moderately strong statistically significant relationship between profession and responses to this question.

o Attorney respondents were significantly more likely than librarians describing their attorneys’ practices to report that they “never” use this tool, or only “rarely” do so, as detailed in the chart below.

o Librarian respondents were significantly more likely to report that their attorneys “frequently” use this tool, as detailed in the chart below.

o Note that the majority of the full cross-section of attorneys who responded to the practitioner survey reported that they only use practice guides “rarely” (26.9%) or “occasionally” (31.9%). Almost 10% say they “never” use practice guides. Nearly one-third though reported using practice guides “frequently” (20.4%) or “very frequently” (11.3%).

10.5%

24.2% 26.6%

1.8%

11.5%

38.9%

0.00%

5.00%

10.00%

15.00%

20.00%

25.00%

30.00%

35.00%

40.00%

45.00%

Never Rarely Frequently

Use practice guides

Attorneys

Librarian Observers

Page 22: A Comparison of Responses from Attorneys and from … · from Law Librarians on Legal Research Practices and Expectations for New Associates . ... F. Case Digests ... legal service

TOC

- 22 -

C. Legal encyclopedias (239 respondents):

o There is an extremely strong statistically significant relationship between profession and responses to this question.

o Attorney respondents were significantly more likely than librarians describing their attorneys’ practices to report that they “never” use this tool, as detailed in the chart below.

o Librarian respondents were significantly more likely to report that their attorneys “occasionally” use this tool, or “frequently” do so, as detailed in the chart below.

o Note that the very small percentage of the full cross-section of attorneys who responded to the practitioner survey who reported using legal encyclopedias “frequently” (4.6%) or “very frequently” (1.7%) in their research was very small. The vast majority reported “rarely” (42.9%) or “never” (24.5%) doing so.

o Over fifty percent of both attorneys and librarians responded that attorneys use legal encyclopedias only “occasionally” or “never.”

27.2% 24.8%

7.2% 2.6%

48.2%

15.8%

0.00%

10.00%

20.00%

30.00%

40.00%

50.00%

60.00%

Never Occasionally Frequently

Use legal encyclopedias

Attorneys

Librarian Observers

Page 23: A Comparison of Responses from Attorneys and from … · from Law Librarians on Legal Research Practices and Expectations for New Associates . ... F. Case Digests ... legal service

TOC

- 23 -

D. Law review/law journal articles (234 respondents):

o There is an extremely strong statistically significant relationship between profession and responses to this question.

o Attorney respondents were significantly more likely than librarians describing their attorneys’ practices to report that they “never” use this tool, or “rarely” do so, as detailed in the chart below.

o Librarian respondents were significantly more likely to report that their attorneys “frequently” use this tool, as detailed in the chart below.

o Note that three-quarters of the full cross-section of attorneys who responded to the practitioner survey reported that they only “rarely” (44.3%) or “occasionally” (29.2%) use these articles, and another 17.8% reported that they “never” do so. The percentage who reported doing so “frequently” (7.3%) or “very frequently” (1.4%) was very small.

o There is a large disparity between the responses of attorneys and librarians in terms of how often attorneys use law reviews/law journal articles. 29% of librarians responded that attorneys use these resources “frequently” while only 5.6% of attorneys agreed. 61.1% of attorneys responded that –- they use these resources “rarely” or “never” compared to 25% of librarians who agreed with respect to their attorneys..

13.5%

47.6%

5.6%

0.0%

25.0% 29.6%

0.00%

5.00%

10.00%

15.00%

20.00%

25.00%

30.00%

35.00%

40.00%

45.00%

50.00%

Never Rarely Frequently

Use law review/ law journal articles

Attorneys

Librarian Observers

Page 24: A Comparison of Responses from Attorneys and from … · from Law Librarians on Legal Research Practices and Expectations for New Associates . ... F. Case Digests ... legal service

TOC

- 24 -

E. American Law Reports (235 respondents):

o There is a very strong statistically significant relationship between profession and responses to this question.

o Attorney respondents were significantly more likely than librarians describing their attorneys’ practices to report that they “never” use this tool, as detailed in the chart below.

o Librarian respondents were significantly more likely to report that their attorneys “occasionally” use this tool, as detailed in the chart below.

o Note that three-quarters of the full cross-section of attorneys who responded to the practitioner survey reported that they “never” (31.5%) or only “rarely” (44.3%) use ALRs in their research. Slightly under a quarter (23.3%) say they do so “occasionally.” Only 4.4% reported using ALRs “frequently” or “very frequently.”

o Over fifty percent of both groups (attorneys and librarians) responded that attorneys use American Law Reports “never” or “occasionally.”

28.8% 25.6%

5.5%

48.2%

0.00%

10.00%

20.00%

30.00%

40.00%

50.00%

60.00%

Never Occasionally

Use American Law Reports

Attorneys

Librarian Observers

Page 25: A Comparison of Responses from Attorneys and from … · from Law Librarians on Legal Research Practices and Expectations for New Associates . ... F. Case Digests ... legal service

TOC

- 25 -

F. Case digests (234 respondents):

o There is a strong statistically significant relationship between profession and responses to this question.

o Attorney respondents were significantly more likely than librarians describing their attorneys’ practices to report that they “never” use this tool, as detailed in the chart below.

26.2%

4.6%

0.00%

5.00%

10.00%

15.00%

20.00%

25.00%

30.00%

Never

Use case digests

Attorneys

Librarian Observers

Page 26: A Comparison of Responses from Attorneys and from … · from Law Librarians on Legal Research Practices and Expectations for New Associates . ... F. Case Digests ... legal service

TOC

- 26 -

G. Restatements (235 respondents):

o There is an extremely strong statistically significant relationship between profession and responses to this question.

o Attorney respondents were significantly more likely than librarians describing their attorneys’ practices to report that they “never” use this tool, or only “rarely,” as detailed in the chart below.

o Librarian respondents were significantly more likely to report that their attorneys “occasionally” use this tool, or “frequently” do so, as detailed in the chart below.

o Note that nearly two-thirds of the full cross-section of attorneys who responded to the practitioner survey reported that they “never” (27%) or “rarely” (38.1%) use Restatements. Just over one-quarter (28.4%) report using them “occasionally.” Only 6.5% reported using them “frequently” or “very frequently.”

o Almost seventy-five percent of librarians responded that their attorneys use Restatements only “occasionally,” “rarely,” or “never.” The librarians’ responses were consistent with the attorney responses, 92% of whom responded that - they use this resource “occasionally,” “rarely,” or “never.”

31.0% 34.1%

27.8%

6.3% 1.8%

15.6%

56.0%

18.3%

0.00%

10.00%

20.00%

30.00%

40.00%

50.00%

60.00%

Never Rarely Occasionally Frequently

Use Restatements

Attorneys

Librarian Observers

Page 27: A Comparison of Responses from Attorneys and from … · from Law Librarians on Legal Research Practices and Expectations for New Associates . ... F. Case Digests ... legal service

TOC

- 27 -

H. Shepard’s or KeyCite – for case validation (237 respondents):

o There is an extremely strong statistically significant relationship between profession and responses to this question.

o Attorney respondents were significantly more likely than librarians describing their attorneys’ practices to report that they “never” use this tool, or only “rarely,” as detailed in the chart below.

o Librarian respondents were significantly more likely to report that their attorneys “frequently” use this tool, as detailed in the chart below.

o Note that a somewhat surprising 16.2% of the full cross-section of attorneys who responded to the practitioner survey reported that they “never” use Shepard’s or KeyCite for case validation, while 18.6 % reported that they “rarely” use these citators for this purpose. Still, larger percentages of the respondents reported using Shepard’s or KeyCite for case validation either “frequently” (29.0%) or “very frequently” (23.0%), with another 13.1% reported doing so “occasionally.”

o The attorney “never” and “rarely” responses are possibly due to delegating case validation to paralegals.

16.0% 20.8%

25.6%

0.9% 2.7%

42.9%

0.00%

5.00%

10.00%

15.00%

20.00%

25.00%

30.00%

35.00%

40.00%

45.00%

50.00%

Never Rarely Frequently

Use Shepard's or KeyCite - for case validation

Attorneys

Librarian Observers

Page 28: A Comparison of Responses from Attorneys and from … · from Law Librarians on Legal Research Practices and Expectations for New Associates . ... F. Case Digests ... legal service

TOC

- 28 -

I. Shepard’s or KeyCite – for further research (231 respondents):

o There is a very strong statistically significant relationship between profession and responses to this question.

o Attorney respondents were significantly more likely than librarians describing their attorneys’ practices to report that they “never” use this tool, or only “rarely,” as detailed in the chart below.

o Librarian respondents were significantly more likely to report that their attorneys “occasionally” use this tool, as detailed in the chart below.

Note that close to half of the full cross-section of attorneys who responded to the practitioner survey reported that they used Shepard’s or KeyCite for further research “frequently” (26.9%) or “very frequently” (20.2%). On the other hand, well over a third report they “rarely” (19.6%) or “never” (18.0%) do so, with the remaining 15.3% reporting that they do so “occasionally.”

o The attorney “never” and “rarely” responses are possibly due to delegating citator use to paralegals.

o There is a large disparity between the percentage of attorneys and librarians who responded that attorneys “frequently” use Shepard’s or KeyCite. Librarians were 22.8% more likely to respond that attorneys use this research technique “frequently”.

16.7% 20.6%

14.3%

0.0%

8.6%

37.1%

0.00%

5.00%

10.00%

15.00%

20.00%

25.00%

30.00%

35.00%

40.00%

Never Rarely Frequently

Use Shepard's or KeyCite - for further research

Attorneys

Librarian Observers

Page 29: A Comparison of Responses from Attorneys and from … · from Law Librarians on Legal Research Practices and Expectations for New Associates . ... F. Case Digests ... legal service

TOC

- 29 -

J. Looseleaf services (239 respondents):

o There is an extremely strong statistically significant relationship between profession and responses to this question.

o Attorney respondents were significantly more likely than librarians describing their attorneys’ practices to report that they “never” use this tool, or only “rarely,” as detailed in the chart below.

o Librarian respondents were significantly more likely to report that their attorneys use this tool “very frequently,” “frequently,” or at least “occasionally,” as detailed in the chart below.

o Note that nearly three-quarters of the full cross-section of attorneys who responded to the practitioner survey reported that they “never” (40.3%), or “rarely” (33.3%) use looseleafs services. Another 18.1% reported using them “occasionally.” Of those reporting more frequent use, 6.3% reported using them “frequently” and less than 2% reported using them “very frequently.”

o The survey did not capture “specializations.” One possible reason for the difference in responses between librarians and attorneys could be that the librarians work at firms that have attorneys -specializing in tax or other looseleaf-using practice areas, while the majority of attorney responders do not practice in these areas.

38.6%

28.8% 22.4%

8.8% 3.2% 0.9%

4.4%

45.6%

36.0%

13.2%

0.00%5.00%

10.00%15.00%20.00%25.00%30.00%35.00%40.00%45.00%50.00%

Use looseleaf services

Attorneys

Librarian Observers

Page 30: A Comparison of Responses from Attorneys and from … · from Law Librarians on Legal Research Practices and Expectations for New Associates . ... F. Case Digests ... legal service

TOC

- 30 -

K. Sample legal forms (235 respondents):

o There is an extremely strong statistically significant relationship between profession and responses to this question.

o Attorney respondents were significantly more likely than librarians describing their attorneys’ practices to report that they “never” use this tool, or only “rarely,” as detailed in the chart below.

o Librarian respondents were significantly more likely to report that their attorneys use this resource “frequently,” or at least “occasionally,” as detailed in the chart below.

o Note that more than half of the full cross-section of attorneys who responded to the practitioner survey reported that they “never” (25.8%), or “rarely” (29.7%) use sample legal forms. Another 28.5% reported doing so “occasionally.” Only a small percentage of respondents reported using sample legal forms “frequently” (13.2%) or “very frequently” (2.7%).

o Note also that analysis of office size subgroups represented in the practitioner survey showed a

moderately strong statistically significant negative relationship between office size and frequency of use of sample legal forms, with frequency of use increasing as office size decreases. For example, the proportion of solo practitioners, who were not included in the comparison data set, reporting that they “frequently or very frequently” use sample forms (29.2%) is significantly greater than the percentage of practitioners in offices of 200+ attorneys (5.6%)reporting that they do. This may explain the higher percentage of “never” and “rarely” responses from the limited (50+ attorney offices only) comparative data set reflected in the chart above, but not the discrepancy between attorney and librarian responses.

37.3% 40.5%

15.9%

3.2% 2.8%

19.3%

45.0%

26.6%

0.00%

5.00%

10.00%

15.00%

20.00%

25.00%

30.00%

35.00%

40.00%

45.00%

50.00%

Never Rarely Occasionally Frequently

Use sample legal forms

Attorneys

Librarian Observers

Page 31: A Comparison of Responses from Attorneys and from … · from Law Librarians on Legal Research Practices and Expectations for New Associates . ... F. Case Digests ... legal service

TOC

- 31 -

L. Litigation resources (237 respondents):

o There is an extremely strong statistically significant relationship between profession and responses to this question.

o Attorney respondents were significantly more likely than librarians describing their attorneys’ practices to report that they “never” use this tool, or only “rarely,” as detailed in the chart below.

o Librarian respondents were significantly more likely to report that their attorneys use this resource “very frequently,” “frequently,” or at least “occasionally,” as detailed in the chart below.

o Note that just over a quarter of the full cross-section of attorneys who responded to the practitioner survey reported that they use litigation resources “frequently” (19.7%) or “very frequently” (6.6%). Just under half of the respondents, though, reported that they “rarely” (27.6%) or “never” (22.1%) do so. Another 23.5% reported doing so “occasionally.”

o In both the practitioner and librarian surveys, these resources are described as “Litigation resources/discovery, jury instructions, etc.). One possible explanation for the divergence in response between the professions may be different understandings of what print and online materials are included in the term “litigation resources.”

29.6% 31.2%

22.4%

12.8% 13.4%

0.9% 7.1%

36.6% 42.0%

4.0%

0.00%5.00%

10.00%15.00%20.00%25.00%30.00%35.00%40.00%45.00%

Use litigation resources

Attorneys

Librarian Observers

Page 32: A Comparison of Responses from Attorneys and from … · from Law Librarians on Legal Research Practices and Expectations for New Associates . ... F. Case Digests ... legal service

TOC

- 32 -

M. Transactional resources (233 respondents):

o There is an extremely strong statistically significant relationship between profession and responses to this question.

o Attorney respondents were significantly more likely than librarians describing their attorneys’ practices to report that they “never” use this tool, or only “rarely,” as detailed in the chart below.

o Librarian respondents were significantly more likely to report that their attorneys use this tool “very frequently,” “frequently,” or at least “occasionally,” as detailed in the chart below.

o Note that more than 60% of the full cross-section of attorneys who responded to the practitioner survey reported that they “rarely” (28.1%) or “never” (34.9%) use transactional resources in their research. Another 21.6% reported doing so “occasionally.” Only 15.5% of respondents reported using transactional resources “frequently” (10.8%) or “very frequently” (4.3%).

o In both the practitioner and librarian surveys, these tools are described as “Transactional resources/ guides, etc.” One possible explanation for the divergence in response between the professions may be different understandings of what constitutes a transactional resource or a “guide.” For example, a CLE course material volume may or may not be considered a “guide” by different respondents.

40.5%

31.0%

12.7% 13.5%

2.4% 1.9%

15.0%

38.3% 35.5%

9.3%

0.00%5.00%

10.00%15.00%20.00%25.00%30.00%35.00%40.00%45.00%

Use transactional resources

Attorneys

Librarian Observers

Page 33: A Comparison of Responses from Attorneys and from … · from Law Librarians on Legal Research Practices and Expectations for New Associates . ... F. Case Digests ... legal service

TOC

- 33 -

VI. Question Posed: Using Free Research Tools

“In the course of your normal practice, how often do you use the following free internet sources for legal research?” | “How often are the following free internet resources used by attorneys at your office?” [Q10-practitioner survey; Q12-librarian survey]

A. Google (235 respondents):

o There is an extremely strong statistically significant relationship between profession and responses to this question.

o Attorney respondents were significantly more likely than librarians describing their attorneys’ practices to report that they “never” use this resource, use it “rarely,” or only “occasionally,” as detailed in the chart below.

o Librarian respondents were significantly more likely to report that their attorneys use this resource “very frequently,” as detailed in the chart below.

o Note that well over a quarter (28.7%) of the full cross-section of attorneys who responded to the practitioner survey reported “very frequent” use of Google for legal research, the highest percentage of any category. Another 25.0% reported using Google “frequently,” while 25.1% reported using it “occasionally,” and 14.5% “rarely.” Only 6.7% reported never using Google for legal research.

o The significantly higher frequency of librarians reporting that their attorneys use Google “very frequently” may possibly reflect the greater number of attorneys who seek librarian assistance after trying and reporting failure with Google searches, attorneys underestimating their actual use of Google in conducting legal research, or other factors.

6.3% 11.1%

25.4% 28.6%

0.0% 1.8% 1.8%

72.5%

0.00%

10.00%

20.00%

30.00%

40.00%

50.00%

60.00%

70.00%

80.00%

Never Rarely Occasionally VeryFrequently

Use Google

Attorneys

Librarian Observers

Page 34: A Comparison of Responses from Attorneys and from … · from Law Librarians on Legal Research Practices and Expectations for New Associates . ... F. Case Digests ... legal service

TOC

- 34 -

B. Google Scholar (228 respondents):

o There is an extremely strong statistically significant relationship between profession and responses to this question.

o Attorney respondents were significantly more likely than librarians describing their attorneys’ practices to report that they “never” use this resource, as detailed in the chart below.

o Librarian respondents were significantly more likely to report that their attorneys use this resource “very frequently,” “frequently,” or at least “occasionally,” as detailed in the chart below.

o Note that, perhaps surprisingly, well over half (56.5%) of the full cross-section of attorneys who responded to the practitioner survey reported that they “never” use Google Scholar for legal research. Another 16.5% reported “rarely” using Google Scholar, while 11.4% reported using it “occasionally.” Only 9.0% reported using it “frequently,” and just 6.7% “very frequently.”

o It is possible that the librarian observers are significantly more likely to report more frequent attorney use of Google Scholar than the practitioners themselves because the attorneys who librarians work with and observe directly are more likely to have been instructed in its use by librarians.

53.2%

9.5% 13.5%

7.9% 2.0%

30.4%

39.2%

16.7%

0.00%

10.00%

20.00%

30.00%

40.00%

50.00%

60.00%

Never Occasionally Frequently VeryFrequently

Use Google Scholar

Attorneys

Librarian Observers

Page 35: A Comparison of Responses from Attorneys and from … · from Law Librarians on Legal Research Practices and Expectations for New Associates . ... F. Case Digests ... legal service

TOC

- 35 -

C. Court websites (234 respondents):

o There is an extremely strong statistically significant relationship between profession and responses to this question.

o Attorney respondents were significantly more likely than librarians describing their attorneys’ practices to report that they “never” use this resource, or only “rarely” do so, as detailed in the chart below.

o Librarian respondents were significantly more likely to report that their attorneys use this resource “frequently,” or “very frequently,” as detailed in the chart below.

o Note that nearly half of the full cross-section of attorneys who responded to the practitioner survey reported using court websites “frequently” (29.5%) or “very frequently” (19.2%). Over a quarter reported using court websites “occasionally” (26.8%), while the final one-quarter report using such websites “rarely” (14.9%) or “never” (9.7%).

o This is not surprising considering that the full cross-section of practitioner survey respondents included solo practitioners and practitioners in smaller offices of up to 50 attorneys, who may make cost-effective use of court web sites and may need to consult them more frequently in the litigation process than attorneys in larger offices.

16.7% 20.6%

31.7%

12.7%

0.9% 4.6%

44.4%

31.5%

0.00%5.00%

10.00%15.00%20.00%25.00%30.00%35.00%40.00%45.00%50.00%

Never Rarely Frequently VeryFrequently

Use court web sites

Attorneys

Librarian Observers

Page 36: A Comparison of Responses from Attorneys and from … · from Law Librarians on Legal Research Practices and Expectations for New Associates . ... F. Case Digests ... legal service

TOC

- 36 -

D. Government legislative web sites (233 respondents):

o There is a moderately strong statistically significant relationship between profession and responses to this question.

o Attorney respondents were significantly more likely than librarians describing their attorneys’ practices to report that they “never” use this resource, or only “rarely” do so, as detailed in the chart below.

o Librarian respondents were significantly more likely to report that their attorneys use this resource “frequently,” as detailed in the chart below.

o Note that half of the full cross-section of attorneys who responded to the practitioner survey reported using government legislative websites “rarely” (18.8%) or “occasionally” (31.2%), while another 10.2% reported that they “never” do so. Still, more than one-third report doing so “frequently” (23.9%) or “very frequently” (15.8%).

o It is possible that the librarian observers are significantly more likely to report more frequent attorney use of government legislative websites than the practitioners themselves because the attorneys who librarians work with and observe directly are more likely to have been instructed to use them by librarians.

9.5%

22.2% 19.8%

0.9%

14.0%

39.3%

0.00%

5.00%

10.00%

15.00%

20.00%

25.00%

30.00%

35.00%

40.00%

45.00%

Never Rarely Frequently

Use government legislative web sites

Attorneys

Librarian Observers

Page 37: A Comparison of Responses from Attorneys and from … · from Law Librarians on Legal Research Practices and Expectations for New Associates . ... F. Case Digests ... legal service

TOC

- 37 -

E. Government agency web sites (233 respondents):

o There is a moderately strong statistically significant relationship between profession and responses to this question.

o Attorney respondents were significantly more likely than librarians describing their attorneys’ practices to report that they “never” use this resource, or only “rarely” do so, as detailed in the chart below.

o Librarian respondents were significantly more likely to report that their attorneys use this resource “frequently,” as detailed in the chart below.

o Note that over 40% of the full cross-section of attorneys who responded to the practitioner survey reported using agency websites “frequently” (27.1%) or “very frequently” (15.2%). Less than half reported doing so “occasionally” (29.1%) or “rarely” (18.6%). Similarly to legislative websites, approximately 10% of practitioners say they never use agency websites.

o It is possible that the librarian observers are significantly more likely to report more frequent attorney use of government agency websites than the practitioners themselves because the attorneys who librarians work with and observe directly are more likely to have been instructed to use them by librarians.

9.5%

19.0%

24.6%

0.9%

7.5%

42.1%

0.00%

5.00%

10.00%

15.00%

20.00%

25.00%

30.00%

35.00%

40.00%

45.00%

Never Rarely Frequently

Use government agency web sites

Attorneys

Librarian Observers

Page 38: A Comparison of Responses from Attorneys and from … · from Law Librarians on Legal Research Practices and Expectations for New Associates . ... F. Case Digests ... legal service

TOC

- 38 -

F. Legal blogs (211 respondents):

o There is an extremely strong statistically significant relationship between profession and responses to this question.

o Attorney respondents were significantly more likely than librarians describing their attorneys’ practices to report that they “never” use this resource, or only “rarely” do so, as detailed in the chart below.

o Librarian respondents were significantly more likely to report that their attorneys use this resource “frequently,” or at least “occasionally,” as detailed in the chart below.

o Note that over 40% of the full cross-section of attorneys who responded to the practitioner survey reported “never” (40.5%) using these resources. Only 8.3% of respondents say they use legal blogs “frequently” or “very frequently” for legal research.

40.8%

32.0%

21.6%

4.8% 1.2%

10.5%

46.5%

38.4%

0.00%5.00%

10.00%15.00%20.00%25.00%30.00%35.00%40.00%45.00%50.00%

Never Rarely Occasionally Frequently

Use legal blogs

Attorneys

Librarian Observers

Page 39: A Comparison of Responses from Attorneys and from … · from Law Librarians on Legal Research Practices and Expectations for New Associates . ... F. Case Digests ... legal service

TOC

- 39 -

G. Law firm websites (213 respondents):

o There is an extremely strong statistically significant relationship between profession and responses to this question.

o Attorney respondents were significantly more likely than librarians describing their attorneys’ practices to report that they “never” use this resource, as detailed in the chart below.

o Librarian respondents were significantly more likely to report that their attorneys use this resource “frequently,” or at least “occasionally,” as detailed in the chart below.

o Note that just less than 40% of the full cross-section of attorneys who responded to the practitioner survey reported that they “never” use law firm web sites for research, while approximately the same number (38.7%) reported “rarely” using them. Only 15.8% report even using them “occasionally,” and only 5.5% report using them “frequently” or “very frequently” for research.

40.5%

14.3%

6.3% 4.6%

44.8%

21.8%

0.00%5.00%

10.00%15.00%20.00%25.00%30.00%35.00%40.00%45.00%50.00%

Never Occasionally Frequently

Use law firm web sites

Attorneys

Librarian Observers

Page 40: A Comparison of Responses from Attorneys and from … · from Law Librarians on Legal Research Practices and Expectations for New Associates . ... F. Case Digests ... legal service

TOC

- 40 -

H. Law school library websites (210 respondents):

o There is an extremely strong statistically significant relationship between profession and responses to this question.

o Attorney respondents were significantly more likely than librarians describing their attorneys’ practices to report that they “never” use this resource, as detailed in the chart below.

o Librarian respondents were significantly more likely to report that their attorneys use this resource “rarely,” or “occasionally,” as detailed in the chart below.

o Note that over half of the full cross-section of attorneys who responded to the practitioner survey reported that they “never” use law school library websites for legal research. Nearly half reported doing so “rarely” (32.8%) or only “occasionally” (12.4%). Less than 5% reported using law school library websites “frequently” (2.4%) or “very frequently” (2.0%) for legal research.

o A vast majority of both groups (92.9% of librarians and 97.6% of attorneys) responded that attorneys use law school library websites “occasionally,” “rarely,” or “never.”

54.8%

33.3%

9.5% 14.3%

47.6%

31.0%

0.00%

10.00%

20.00%

30.00%

40.00%

50.00%

60.00%

Never Rarely Occasionally

Use law school library websites

Attorneys

Librarian Observers

Page 41: A Comparison of Responses from Attorneys and from … · from Law Librarians on Legal Research Practices and Expectations for New Associates . ... F. Case Digests ... legal service

TOC

- 41 -

I. Non-profit organization’s website (204 respondents):

o There is an extremely strong statistically significant relationship between profession and responses to this question.

o Attorney respondents were significantly more likely than librarians describing their attorneys’ practices to report that they “never” use this resource, as detailed in the chart below.

o Librarian respondents were significantly more likely to report that their attorneys “frequently” use this resource, or use it “occasionally,” as detailed in the chart below.

o Note that just over three quarters of the full cross-section of attorneys who responded to the practitioner survey reported that they “never” (44.6%) or “rarely” (32.5%) use non-profit websites. Another 15.0% report that they do so “occasionally.” Less than 8% report doing so “frequently” (5.6%) or “very frequently” (2.3%).

o One possible explanation for librarians in offices with over 50 attorneys reporting with significantly greater frequency that their attorneys use these sites may be that the attorneys who work with them on statistical or appellate brief research are directed to these resources by librarians.

50.0%

15.3%

1.6% 3.8%

47.5%

8.8%

0.00%

10.00%

20.00%

30.00%

40.00%

50.00%

60.00%

Never Occasionally Frequently

Use nonprofit organization websites

Attorneys

Librarian Observers

Page 42: A Comparison of Responses from Attorneys and from … · from Law Librarians on Legal Research Practices and Expectations for New Associates . ... F. Case Digests ... legal service

TOC

- 42 -

VII. Question Posed: Research Skills of Recent Graduates

“In your opinion, how well do recent law school graduates perform the following components of legal research? [If you do not work with recent law school graduates, please skip this question]” [Q12-practitioner survey | Q17-Librarian Survey]

A. Develop an effective research plan (186 respondents):

o There is an extremely strong statistically significant relationship between profession and responses to this question.

o Attorney respondents were significantly more likely to report that recent graduates perform the task very well, or moderately well, as detailed in the chart below.

o Librarian respondents were significantly more likely to report that recent graduates perform the task poorly, as detailed in the chart below.

o Note that nearly half of the full cross-section of attorneys who responded to the practitioner survey reported that recent law school graduates were “adequate” (45.8%) at developing an effective research plan. Over one-third went further, saying they did so “moderately well” (27.3%) or “very well” (7.1%), meaning that approximately 80% said recent law school graduates develop effective research plans adequately or better. A very small percentage (1.6%) reported that recent graduates performed “unacceptably” in this area, while 18.2% said they did so “poorly.”

o One possible explanation for the significant difference between librarians and practitioners working

with recent law school graduates in firms of 50 or more attorneys may be the higher standards held by librarians as to what constitutes an “effective research plan.”

13.0%

28.6%

7.8%

47.7%

10.1%

0.0% 0.00%

10.00%

20.00%

30.00%

40.00%

50.00%

60.00%

Poorly Moderately Well Very Well

Ability to develop an effective research plan

Attorneys

Librarians

Page 43: A Comparison of Responses from Attorneys and from … · from Law Librarians on Legal Research Practices and Expectations for New Associates . ... F. Case Digests ... legal service

TOC

- 43 -

B. Use secondary sources effectively (182 respondents):

o There is a very strong statistically significant relationship between profession and responses to this question.

o Attorney respondents were significantly more likely to report that recent graduates perform the task moderately well, as detailed in the chart below.

o Librarian respondents were significantly more likely to report that recent graduates perform the task unacceptably or poorly, as detailed in the chart below.

o Note that just under half of the full cross-section of attorneys who responded to the practitioner survey reported that recent law school graduates were “adequate” at using secondary sources effectively. Nearly 30% went further, saying they did so “moderately well” (22.5%) or “very well” (7.1%), meaning that nearly three-quarters of the respondents rated recent graduates as adequate or better at using secondary sources effectively. The other quarter of respondents said recent graduates performed “poorly” (23.7%) or “unacceptably” (2.5%) in this area.

C. Use critical thinking to evaluate the relevance of case law and other primary sources (182

respondents):

o No statistically significant relationship was observed between profession and responses to this question.

o Note that four-fifths of the full cross-section of attorneys who responded to the practitioner survey

rated recent graduates as performing adequately or better in this area (42.4% “adequately,” 30.6% “moderately well,” and 7% “very well”). Only 2.0% rated them as performing “unacceptably” and 18.0% “poorly” in using critical thinking to evaluate primary sources.

0.0%

28.2% 25.4%

8.1%

54.1%

6.3%

0.00%

10.00%

20.00%

30.00%

40.00%

50.00%

60.00%

Unacceptably Poorly Moderately Well

Ability to use secondary sources effectively

Attorneys

Librarians

Page 44: A Comparison of Responses from Attorneys and from … · from Law Librarians on Legal Research Practices and Expectations for New Associates . ... F. Case Digests ... legal service

TOC

- 44 -

D. Develop appropriate search protocol (181 respondents):

o There is a very strong statistically significant relationship between profession and responses to this question.

o Attorney respondents were significantly more likely to report that recent graduates perform the task moderately well, or very well, as detailed in the chart below.

o Librarian respondents were significantly more likely to report that recent graduates perform the task unacceptably or poorly, as detailed in the chart below.

o Note that nearly 90% of the full cross-section of attorneys who responded to the practitioner survey say recent graduates develop appropriate search protocols “adequately” or better (45.6% of “adequately,” 30.5% “moderately well,” 10.9% “very well”). Only 2.1% say recent graduates perform “unacceptably,” in this area, and 10.9% “poorly.”

o A significant percentage of attorneys (42.8%) responded that recent law school graduates develop search protocols “moderately well” or “very well”; whereas, only 14.4% of librarians agreed. Librarians’ responses were more critical of recent graduates’ ability to perform this task with 47.7% of librarians responding that recent graduates perform this task “unacceptably” or “poorly” compared to a small fraction (5.7%) of attorneys who agreed. One possibility for this discrepancy is that attorneys and librarians have different concepts about what constitutes “effective search protocol.”

0.0%

5.7%

31.4%

11.4%

6.3%

41.4%

14.4%

0.0% 0.00%

5.00%

10.00%

15.00%

20.00%

25.00%

30.00%

35.00%

40.00%

45.00%

Unacceptably Poorly Moderately Well Very Well

Ability to develop an effective search protocol

Attorneys

Librarians

Page 45: A Comparison of Responses from Attorneys and from … · from Law Librarians on Legal Research Practices and Expectations for New Associates . ... F. Case Digests ... legal service

TOC

- 45 -

E. Research case law (182 respondents):

o There is a moderately strong statistically significant relationship between profession and responses to this question.

o Attorney respondents were significantly more likely to report that recent graduates perform the task very well, as detailed in the chart below.

o Note that more than 90% of the full cross-section of attorneys who responded to the practitioner survey said recent law school graduates research case law adequately or better, including nearly two-thirds that say they do so “moderately well” (42.6%) or “very well” (21.7%). Only 6.4% said recent graduates research case law “poorly,” and less than 1.0% said they do so “unacceptably.” Attorney respondents were 18.2% more likely to respond that recent graduates research case law “very well”.

o One possible explanation for the discrepancy in librarian and attorney responses may be that librarians and attorneys have differing opinions about what constitutes conducting case law research “very well.”

27.4%

9.2%

0.00%

5.00%

10.00%

15.00%

20.00%

25.00%

30.00%

Very Well

Ability to research case law

Attorneys

Librarians

Page 46: A Comparison of Responses from Attorneys and from … · from Law Librarians on Legal Research Practices and Expectations for New Associates . ... F. Case Digests ... legal service

TOC

- 46 -

F. Research statutes (183 respondents):

o There is a very strong statistically significant relationship between profession and responses to this question.

o Attorney respondents were significantly more likely to report that recent graduates perform the task very well, as detailed in the chart below.

o Librarian respondents were significantly more likely to report that recent graduates perform the task poorly, as detailed in the chart below.

o Note that nearly 90% of the full cross-section of attorneys who responded to the practitioner survey said recent law school graduates research statutes adequately or better, including more than half that say they do so “moderately well” (38.8%) or “very well” (16.8%); 10.0% said recent graduates research statutes “poorly,” and 2.0% said they do so “unacceptably.”

o One-fifth of attorney respondents (20.5%) believe that recent law school graduates research statutes “very well”; whereas, a mere 4.5% of librarians agreed. Almost one-third (29.1%) of librarians responded that recent graduates perform this task “poorly,” compared to only 4.1% of attorneys who agreed.

4.1%

20.5%

29.1%

4.5%

0.00%

5.00%

10.00%

15.00%

20.00%

25.00%

30.00%

35.00%

Poorly Very Well

Ability to research statutes

Attorneys

Librarians

Page 47: A Comparison of Responses from Attorneys and from … · from Law Librarians on Legal Research Practices and Expectations for New Associates . ... F. Case Digests ... legal service

TOC

- 47 -

G. Research regulations (177 respondents):

o There is a very strong statistically significant relationship between profession and responses to this question.

o Attorney respondents were significantly more likely to report that recent graduates perform the task very well, or at least adequately, as detailed in the chart below. Librarian respondents were significantly more likely to report that recent graduates perform the task unacceptably or poorly, as detailed in the chart below.

o Note that nearly two-thirds of the full cross-section of attorneys who responded to the practitioner survey said recent law school graduates research regulations adequately or better, including nearly a third that say they do so “moderately well” (24.2%) or “very well” (8.5%). However, just over a quarter (25.1%) said recent graduates research regulations “poorly,” and 4.0% said they do so “unacceptably.”

o A majority of attorneys (58.3%) responded that recent law school graduates research regulations “adequately” or “very well”; whereas, approximately one-third (30%) of librarians were in agreement. Librarians’ responses were more critical of recent graduates’ ability to perform this task with 60% of librarians responding that recent graduates perform this task “unacceptably” or “poorly” compared to only one-quarter (25.4%) of attorneys who agreed.

1.5%

23.9%

49.3%

9.0% 12.7%

47.3%

29.1%

0.9% 0.00%

10.00%

20.00%

30.00%

40.00%

50.00%

60.00%

Unacceptably Poorly Adequately Very Well

Ability to research regulations

Attorneys

Librarians

Page 48: A Comparison of Responses from Attorneys and from … · from Law Librarians on Legal Research Practices and Expectations for New Associates . ... F. Case Digests ... legal service

TOC

- 48 -

H. Research administrative decisions (168 respondents):

o There is a strong statistically significant relationship between profession and responses to this question.

o Attorney respondents were significantly more likely to report that recent graduates perform the task adequately, as detailed in the chart below.

o Librarian respondents were significantly more likely to report that recent graduates perform the task unacceptably, as detailed in the chart below.

o Note that more than half of the full cross-section of attorneys who responded to the practitioner survey said that recent law school graduates research administrative decisions “adequately” or better (31.7% “adequately," 20.3% “moderately well,” and 4% “very well”). However, 44% said they perform “poorly” (37.1%) or “unacceptably” (6.9%) in this area.

o A significant percentage of attorneys (44.8%) responded that recent graduates perform this task “adequately” or better. A significantly smaller percentage of librarians, only 28.2%, are in agreement. A large percentage of both groups responded that recent graduates perform this task “poorly” or “unacceptably.”

1.7%

44.8%

15.5%

28.2%

0.00%

5.00%

10.00%

15.00%

20.00%

25.00%

30.00%

35.00%

40.00%

45.00%

50.00%

Unacceptably Adequately

Able to research administrative decisions

Attorneys

Librarians

Page 49: A Comparison of Responses from Attorneys and from … · from Law Librarians on Legal Research Practices and Expectations for New Associates . ... F. Case Digests ... legal service

TOC

- 49 -

I. Research legislative history (172 respondents):

o There is an extremely strong statistically significant relationship between profession and responses to this question.

o Attorney respondents were significantly more likely to report that recent graduates perform the task moderately well, or at least adequately, as detailed in the chart below.

o Librarian respondents were significantly more likely to report that recent graduates perform the task unacceptably or poorly, as detailed in the chart below.

o Note that nearly half of the full cross-section of attorneys who responded to the practitioner survey said recent graduates researched legislative history “poorly” (35.7%) or “unacceptably” (12.0%); 30.9% said they do so “adequately,” and less than a quarter said they do so “moderately well” (17.1%) or “very well” (4.3%).

o A majority of attorneys (58%) responded that recent law school graduates research legislative history “adequately” or “moderately well”; whereas, only a small fraction of librarian respondents (14.5%) agreed. A large majority (85.5%) of librarians responded that recent graduates perform this task “unacceptably” or “poorly” compared to 38.7% of attorneys who agreed.

4.8%

33.9%

41.9%

16.1%

26.4%

59.1%

13.6%

0.9% 0.00%

10.00%

20.00%

30.00%

40.00%

50.00%

60.00%

70.00%

Unacceptably Poorly Adequately Moderately Well

Able to research legislative history

Attorneys

Librarians

Page 50: A Comparison of Responses from Attorneys and from … · from Law Librarians on Legal Research Practices and Expectations for New Associates . ... F. Case Digests ... legal service

TOC

- 50 -

J. Understand the difference between statutes and regulations (177 respondents):

o There is an extremely strong statistically significant relationship between profession and responses to this question.

o Attorney respondents were significantly more likely to report that recent graduates perform the task moderately well, or at least adequately, as detailed in the chart below.

o Librarian respondents were significantly more likely to report that recent graduates perform the task unacceptably or poorly, as detailed in the chart below.

o Note that slightly more than half of the full cross-section of attorneys who responded to the practitioner survey said recent graduates understand this difference “moderately well” (25.3%) or “very well” (24.9%). Over one-third said this understanding was “adequate,” while only 14.4% indicated it was “poor” (13.1%) or “unacceptable” (1.3%)

o One-half of attorney respondents (50%) believe that recent law school graduates understand the difference between statutes and regulations “moderately well” or “very well”; whereas, only a fraction of librarian respondents (7.2%) agreed. Librarians’ responses were more critical of recent graduates’ ability to perform this task with nearly half (45%) of librarians responding that recent graduates perform this task “unacceptably” or “poorly” compared to only 12.1% of attorneys who agreed.

0.0%

12.1%

24.2% 25.8%

11.7%

33.3%

0.9%

6.3%

0.00%

5.00%

10.00%

15.00%

20.00%

25.00%

30.00%

35.00%

Unacceptably Poorly Moderately Well Very Well

Understand difference between statutes and regulations

Attorneys

Librarians

Page 51: A Comparison of Responses from Attorneys and from … · from Law Librarians on Legal Research Practices and Expectations for New Associates . ... F. Case Digests ... legal service

TOC

- 51 -

K. Update legal sources using a citator (Shepard’s/KeyCite) (172 respondents):

o There is a moderately strong statistically significant relationship between profession and responses to this question.

o Attorney respondents were significantly more likely to report that recent graduates perform the task very well, as detailed in the chart below.

o Librarian respondents were significantly more likely to report that recent graduates perform the task unacceptably or poorly, as detailed in the chart below.

o Note that more than 90% of the full cross-section of attorneys who responded to the practitioner survey said recent law school graduates update legal sources with a citator at an adequate or better level. Over 60% indicated recent graduates do this research task “moderately well” (33.2%) or “very well” (27.1%), while 30% said they did so “adequately.” Only 7.7% of these respondents indicated recent graduates’ performance in this area was “poor” (6.0%) or “unacceptable” (1.7%).

o Overall, a large majority of attorneys and librarians (100% of attorneys and 82.6% of librarians)

believe that recent graduates update legal sources using a citator “adequately” or better. Less than one-fifth (17.4%) of librarians responded that recent graduates perform this take “unacceptably” or “poorly.” Not a single attorney responded that recent graduates perform this task unacceptably or poorly.

0.0% 0.0%

28.6%

6.4%

11.0%

15.6%

0.00%

5.00%

10.00%

15.00%

20.00%

25.00%

30.00%

35.00%

Unacceptably Poorly Very Well

Able to update legal sources using a citator

Attorneys

Librarians

Page 52: A Comparison of Responses from Attorneys and from … · from Law Librarians on Legal Research Practices and Expectations for New Associates . ... F. Case Digests ... legal service

TOC

- 52 -

L. Perform cost-effective research (180 respondents):

o There is an extremely strong statistically significant relationship between profession and responses to this question.

o Attorney respondents were significantly more likely to report that recent graduates perform the task very well, moderately well, or at least adequately, as detailed in the chart below.

o Librarian respondents were significantly more likely to report that recent graduates perform the task unacceptably or poorly, as detailed in the chart below.

o Note that nearly 40% of the full cross-section of attorneys who responded to the practitioner survey said recent law school graduates perform cost-effective research “poorly” (30.3%) or “unacceptably” (7.4%). Just over a third said they perform “adequately” in this regard (34%), and nearly 20% said they do so “moderately well” (19.7%), or “very well” (8.1%).

o Almost three-quarters of attorneys (71.3%) responded that recent law school graduates perform cost-effective research “adequately,” “moderately well” or “very well;” whereas, just over one-third (34.6%) of librarians are in agreement. 65.4% of librarians responded that recent graduates perform this task “unacceptably” or “poorly” as compared to only 28.5% of attorneys who agreed.

7.1%

21.4%

47.1%

17.1%

7.1%

22.7%

42.7%

29.1%

5.5%

0.0% 0.00%

5.00%

10.00%

15.00%

20.00%

25.00%

30.00%

35.00%

40.00%

45.00%

50.00%

Unacceptably Poorly Adequately ModeratelyWell

Very Well

Able to perform cost effective research

Attorneys

Librarians

Page 53: A Comparison of Responses from Attorneys and from … · from Law Librarians on Legal Research Practices and Expectations for New Associates . ... F. Case Digests ... legal service

TOC

- 53 -

M. Use Westlaw online services efficiently (168 respondents):

o There is an extremely strong statistically significant relationship between profession and responses to this question.

o Attorney respondents were significantly more likely to report that recent graduates perform the task very well, as detailed in the chart below.

o Librarian respondents were significantly more likely to report that recent graduates perform the task unacceptably or poorly, as detailed in the chart below.

o Note that more than 90% of the full cross-section of attorneys who responded to the practitioner survey said recent law school graduates use Westlaw services at an adequate or better level. One-third of these respondents said recent graduates do so “adequately,” 31.0% said they do so “moderately well,” and 19.2% said they do so “very well.” Just over 15% said recent graduates perform “poorly” (13.6%) or “unacceptably” (2.8%) in this regard.

o There was a drastic difference between how attorneys and librarians perceive law school graduates’ use of Westlaw online services. Approximately one-quarter (24.3%) of attorneys responded that recent law school graduates use Westlaw online services efficiently “very well”. No librarians responded that recent graduates use Westlaw online services efficiently “very well.” Rather, librarians’ responses were more critical of recent graduates’ ability to perform this task with 43.5% of librarians responding that recent graduates perform this task “unacceptably” or “poorly” as compared to a mere 17.1% of attorneys who agreed.

1.4%

15.7%

24.3%

9.2%

34.3%

0.0% 0.00%

5.00%

10.00%

15.00%

20.00%

25.00%

30.00%

35.00%

40.00%

Unacceptably Poorly Very Well

Able to use Westlaw online services efficiently

Attorneys

Librarians

Page 54: A Comparison of Responses from Attorneys and from … · from Law Librarians on Legal Research Practices and Expectations for New Associates . ... F. Case Digests ... legal service

TOC

- 54 -

N. Use Lexis online services efficiently (146 respondents):

o There is an extremely strong statistically significant relationship between profession and responses to this question.

o Attorney respondents were significantly more likely to report that recent graduates perform the task very well, as detailed in the chart below.

o Librarian respondents were significantly more likely to report that recent graduates perform the task poorly, as detailed in the chart below.

o Note that more than 80% of the full cross-section of attorneys who responded to the practitioner survey said recent law school graduates use Lexis services at an adequate or better level. Just over one-third of these respondents said recent graduates do so “adequately” (35.2%), 20.2% said they do so “moderately well,” and 15.9% said they do so “very well.” Approximately 20% said recent graduates perform “poorly” (15.9%) or “unacceptably” (2.7%) in this regard.

o 17.5% of attorneys responded that recent law school graduates use Lexis online services efficiently “very well.” No librarians responded that recent graduates use Lexis online services efficiently “very well”. Rather, librarians’ responses were more critical of recent graduates’ ability to perform this task with 39.3% believing that recent graduates perform this task “poorly” as compared to only 15.8% of attorneys who agreed.

15.8% 17.5%

39.3%

0.0% 0.00%

5.00%

10.00%

15.00%

20.00%

25.00%

30.00%

35.00%

40.00%

45.00%

Poorly Very Well

Able to use Lexis online services efficiently

Attorneys

Librarians

Page 55: A Comparison of Responses from Attorneys and from … · from Law Librarians on Legal Research Practices and Expectations for New Associates . ... F. Case Digests ... legal service

TOC

- 55 -

O. Research using an online service other than Westlaw or Lexis (147 respondents):

o No statistically significant relationship was observed between profession and responses to this question.

Note that more than two-thirds of the full cross-section of attorneys who responded to the practitioner survey said recent law school graduates use such services at an adequate or better level. Just over one-third of these respondents said recent graduates do so “adequately,” with 33% saying they do so “adequately,” 20.2% “moderately well,” and 15.9% “very well.” Over one-third of respondents, however, said that recent graduates’ use of an online service other than Westlaw or Lexis was “poor” (28.2%) or “unacceptable” (8%).

P. Research pleadings, motions, or other court documents:

o No statistically significant relationship was observed between profession and responses to this question.

o Note that more than two-thirds of the full cross-section of attorneys who responded to the practitioner

survey said recent law school graduates research pleadings, motions, and other court documents adequately or better. Nearly 40% said they do so “adequately,” 24.4% “moderately well,” and 5.0% “very well.” However, a full -third of the respondents indicated recent graduates researched these types of documents “poorly” (24.4%) or “unacceptably” (8.6%).

Page 56: A Comparison of Responses from Attorneys and from … · from Law Librarians on Legal Research Practices and Expectations for New Associates . ... F. Case Digests ... legal service

TOC

- 56 -

Q. Know when to stop researching:

o There is a moderate statistically significant relationship between profession and responses to this question.

o Attorney respondents were significantly more likely to report that recent graduates perform the task moderately well, as detailed in the chart below.

o Librarian respondents were significantly more likely to report that recent graduates perform the task unacceptably, as detailed in the chart below.

o Note that more than half of the full cross-section of attorneys who responded to the practitioner survey said recent graduates perform adequately or better at knowing when to stop researching appropriately (35.0% “adequately,” 20.3% “moderately well,” and 2.4% “very well”). More than 42%, however, said recent grads do this “poorly” (30.5%) or “unacceptably” (11.8%). More than 4 times as many respondents rated recent graduates as “unacceptable” than rated them as performing “very well” in this regard.

o 17.8% of attorneys responded that recent graduates know when to stop researching “moderately well”, compared to only 5.6% of librarians who were in agreement. Librarians were more critical of recent graduates’ ability to know when to stop researching, as 13.1% of librarians responded that recent graduates performed this task “unacceptably” compared to 4.1% of attorneys who agreed.

4.1%

17.8%

13.1%

5.6%

0.00%

2.00%

4.00%

6.00%

8.00%

10.00%

12.00%

14.00%

16.00%

18.00%

20.00%

Unacceptable Moderately Well

Know when to stop researching

Attorneys

Librarians

Page 57: A Comparison of Responses from Attorneys and from … · from Law Librarians on Legal Research Practices and Expectations for New Associates . ... F. Case Digests ... legal service

TOC

- 57 -

R. Able to find a variety of other non-legal information:

o There is an extremely strong statistically significant relationship between profession and responses to this question.

o Attorney respondents were significantly more likely to report that recent graduates perform the task moderately well, as detailed in the chart below.

o Librarian respondents were significantly more likely to report that recent graduates perform the task unacceptably or poorly, as detailed in the chart below.

o Note that more than 40% of the full cross-section of attorneys who responded to the practitioner survey said recent graduates did this “poorly” (31.1%) or “unacceptably” (12.2%). Slightly more than one-third indicated they did so “adequately,” while 18.4% said they did so “moderately well” and 3.8% said “very well.”

o 22.2% of attorneys responded that recent graduates are able to find a variety of other-non-legal information “moderately well” compared to just 1.9% of librarians who were in agreement. Librarians were much more critical of recent graduates’ ability to find a variety of other non-legal information, with a large majority (69.8%) of librarians responding that recent graduates performed this task “unacceptably” or “poorly” compared to 36.5% of attorneys who agreed.

7.9%

28.6%

22.2% 25.5%

44.3%

1.9%

0.00%

5.00%

10.00%

15.00%

20.00%

25.00%

30.00%

35.00%

40.00%

45.00%

50.00%

Unacceptable Poorly Moderately Well

Able to find variety of other non-legal information

Attorneys

Librarians

Page 58: A Comparison of Responses from Attorneys and from … · from Law Librarians on Legal Research Practices and Expectations for New Associates . ... F. Case Digests ... legal service

TOC

- 58 -

APPENDICES

A COMPARISON OF RESPONSES FROM ATTORNEYS AND FROM LAW LIBRARIANS ON LEGAL RESEARCH PRACTICES AND

EXPECTATIONS FOR NEW ASSOCIATES

June 2014

ALL-SIS Task Force on Identifying Skills and Knowledge for Legal Practice

Page 59: A Comparison of Responses from Attorneys and from … · from Law Librarians on Legal Research Practices and Expectations for New Associates . ... F. Case Digests ... legal service

TOC

- 59 -

Appendix A

Demographic Details of Combined Data Set

Page 60: A Comparison of Responses from Attorneys and from … · from Law Librarians on Legal Research Practices and Expectations for New Associates . ... F. Case Digests ... legal service

Appendix B

Summary Tables

Data from Survey of Legal Practitioners (603 respondents)

Data from Survey of Law Librarians (184 respondents)

Data Comparing Similarly Situated Practitioners and Law Librarians (280 respondents)

Page 61: A Comparison of Responses from Attorneys and from … · from Law Librarians on Legal Research Practices and Expectations for New Associates . ... F. Case Digests ... legal service

TOC

- 61 -

DATA FROM 2012 ALL-SIS SURVEY OF LEGAL PRACTITIONERS6

INTRODUCTORY AND DEMOGRAPHIC QUESTIONS:

QUESTION 1. In what state do you currently conduct the majority of your practice? (574) State Valid Percent State Valid Percent State Valid Percent State Valid Percent

Alabama 0.5 Indiana 0.9 Nevada 14.0 South Dakota 0.3 Alaska 0.5 Iowa 0.3 New Hampshire 0 Tennessee 0.5 Arizona 3.8 Kansas 0.2 New Jersey 0.9 Texas 8.7 Arkansas 0 Kentucky 0.3 New Mexico 15.0 Utah 11.1 California 7.5 Louisiana 0 New York 7.3 Vermont 0.3 Colorado 4.4 Maine 0 North Carolina 0.5 Virginia 1.4 Connecticut 0.2 Maryland 0.3 North Dakota 0 Washington 1.9 Delaware 0 Massachusetts 1.0 Ohio 0.5 West Virginia 0 D.C. 2.8 Michigan 0.5 Oklahoma 0.3 Wisconsin 0.7 Florida 0.9 Minnesota 0.3 Oregon 1.2 Wyoming 0.2 Georgia 0.7 Mississippi 0.2 Pennsylvania 0.3 International 1.4 Hawaii 0.9 Missouri 2.6 Puerto Rico 0.2 Idaho 1.2 Montana 0 Rhode Island 0 Illinois 14.5 Nebraska 0 South Carolina 0

QUESTION 2. What is the size of the office in which you work? (588) Solo

Practitioner 2-5 attorneys 6-10 attorneys 11-20 attorneys 21-50 attorneys 51-100

attorneys 101-150 attorneys

151-200 attorneys

201+ attorneys

18.7 25.3 11.7 11.1 8.5 6.5 3.1 2.6 12.6

QUESTION 3. Which setting best describes your working environment? (602)

6 A total of 603 individuals responded to the practitioner survey. All figures reported represent the percentage of respondents selecting a relevant response. The number of respondents that answered a given question and were counted for purposes of calculating the reported percentages varies by question and is listed for each (see # in parentheses). Each of the questions that asked respondents for their opinions about the research abilities of recent law school graduates included the option of responding “n/a”. Respondents who skipped a question or selected “n/a” were not included in calculating the relevant response percentages for that question as reported here. The figures reported are thus considered “valid percentages” for purposes of statistical analysis.

Page 62: A Comparison of Responses from Attorneys and from … · from Law Librarians on Legal Research Practices and Expectations for New Associates . ... F. Case Digests ... legal service

TOC

- 62 -

Private Practice / Litigation

Private Practice / Transactional

Private Practice / Mixed

In House / Corporate Counsel

Government / Federal

Government / State

Government / Local7

Legal Services Corporation

27.2 9.0 21.1 8.3 5.6 12.3 1.5 1.7

Public Interest Judiciary Military Consulting8 Law School Other Legal

Education9 Non Law Related Retired10

3.3 6.0 0.8 0.5 0.7 0.3 0.3 0.3

Other

1.0

QUESTION 4. How many years have you been in legal practice? (601) 0-4 years 5-9 years 10-19 years 20-29 years 30+ years

11.5 17.0 24.6 19.0 17.0

QUESTIONS ABOUT RESEARCH PROCESS:

QUESTION 5. In an average week in your practice, how much of your time do you spend doing legal research? (600) None At least 15 Percent 15-25 Percent 25-50 Percent 50-75 Percent More than 75 Percent 5.76 47.3 24.3 12.3 7.0 3.3

QUESTION 6. When beginning research, how often do you … Never Rarely Never or

Rarely Occasionall

y Frequently Very

Frequently Frequently

or Very Frequently

Start by asking a fellow attorney for advice (586) 8.9 27.3 36.2 28.8 19.5 15.5 35.0 Start with Google (587) 12.8 23.2 36.0 30.2 21.1 12.8 33.9 Start with statutory database (585) 6.2 14.5 20.7 27.2 36.4 15.7 52.1 Start by consulting work product (584) 15.8 18.5 34.3 31.0 22.9 11.8 34.7 Start with a secondary source (581) 9.3 25.3 34.6 34.1 25.0 6.4 31.4 7 Identified by respondents who selected “other” from the response categories listed and added a narrative description in the space provided. 8 Identified by respondents who selected “other” from the response categories listed and added a narrative description in the space provided. 9 Identified by respondents who selected “other” from the response categories listed and added a narrative description in the space provided. 10 Identified by respondents who selected “other” from the response categories listed and added a narrative description in the space provided.

Page 63: A Comparison of Responses from Attorneys and from … · from Law Librarians on Legal Research Practices and Expectations for New Associates . ... F. Case Digests ... legal service

TOC

- 63 -

Start with case law database (590) 5.3 17.1 22.4 20.8 30.7 25.9 56.6 Start with a subject specific practice guide (127) 8.2 26.2 34.4 32.0 23.5 10.2 33.7

QUESTION 7. When you are researching an issue for your practice, how often do you __________________? Never Rarely Never or

Rarely Occasionally Frequently Very

Frequently Frequently or Very Frequently

Use digest or online equivalent to find a topic & key number (591)

22.8 36.7 59.5 21.0 14.0 5.4 19.4

Use headnote & key numbers in case (588) 16.7 26.9 43.6 24.1 21.4 10.9 32.3 Follow citations in case (594) 3.7 7.9 11.6 16.0 38.7 33.7 72.4 Use index or TOC with online statutory DB (588) 5.6 13.3 18.9 28.6 33.2 19.4 52.6 Follow citations in secondary sources (588) 6.0 17.7 23.7 30.3 29.9 16.2 46.1 Use terms and connectors (594) 6.6 10.6 17.2 17.5 27.4 37.9 65.3 Follow citations in annotated code (588) 6.5 13.8 20.3 27.4 32.7 19.7 52.4

QUESTIONS ABOUT RESOURCE USAGE:

QUESTION 8. When performing legal research for your practice, how often do you __________________? Never Rarely Never or

Rarely Occasionally Frequently Very

Frequently Frequently or

Very Frequently

Use print materials (590) 5.1 18.6 23.7 33.9 26.9 15.4 42.3 Use free internet sources (590) 2.9 9.7 12.6 26.1 30.7 30.7 61.4 Use fee-based databases (591) 8.1 9.6 17.7 15.2 22.7 44.3 67.0

QUESTION 9. When performing legal research for your practice, how often do you utilize the following research tools? Never Rarely Never or

Rarely Occasionally Frequently Very

Frequently Frequently or

Very Frequently

Treatises (587) 10.2 28.8 39.0 33.7 18.7 8.5 27.2 Practice guides (583) 9.4 26.9 36.3 31.9 20.4 11.3 31.7 Legal encyclopedias (583) 24.5 42.9 67.4 26.2 4.6 1.7 6.3 Law review/law journal articles (589) 17.8 44.3 62.1 29.2 7.3 1.4 8.7 American Law Reports (588) 31.5 40.8 72.3 23.3 3.7 0.7 4.4 Case digests (585) 25.1 37.6 62.7 23.8 10.6 2.9 13.5

Page 64: A Comparison of Responses from Attorneys and from … · from Law Librarians on Legal Research Practices and Expectations for New Associates . ... F. Case Digests ... legal service

TOC

- 64 -

Restatements (585) 27.0 38.1 65.1 28.4 5.5 1.0 6.5 Shepard’s or KeyCite – for case validation (586) 16.2 18.6 34.8 13.1 29.0 23.0 52.0 Shepard’s or KeyCite – for further research (588) 18.0 19.6 37.6 15.3 26.9 20.2 47.1 Looseleaf services (585) 40.3 33.3 73.7 18.1 6.3 1.9 8.2 Sample legal forms (582) 25.8 29.7 55.5 28.5 13.2 2.7 15.9 Litigation resources (585) 22.6 27.6 50.2 23.5 19.7 6.6 26.3 Transactional resources (588) 34.9 28.1 62.9 21.6 11.1 4.4 15.5 QUESTION 10. In the course of your normal practice, how often do you use the following free internet sources for legal research?

Never Rarely Never or Rarely

Occasionally Frequently Very Frequently

Frequently or Very

Frequently Google (593) 6.7 14.5 21.2 25.1 25.0 28.7 53.7 Google Scholar (581) 56.5 16.5 73.0 11.4 9.0 6.7 15.7 Court web sites (590) 9.7 14.9 24.6 26.8 29.5 19.2 48.7 Government legislative web sites (589) 10.2 18.8 29.0 31.2 23.9 15.8 39.7 Government agency web sites (587) 10.1 18.6 28.6 29.1 27.1 15.2 42.3 Legal blogs (587) 40.5 32.2 72.7 18.9 7.0 1.4 8.4 Law firm web sites (587) 39.7 38.7 78.4 15.8 4.4 1.4 5.8 Law school library websites (589) 50.4 32.8 83.2 12.4 2.4 2.0 4.4 Non-profit organization’s website (572) 44.6 32.5 77.1 15.0 5.6 2.3 7.9

QUESTION 11. Which of the following databases, if any, do you use when performing legal research? Select all that apply. Yes No Additional analysis derived from the data

Yes No Use at least one of these platforms 63.3 36.7 Use more than one of these platforms 10.1 89.1

Bloomberg Law (588) 2.0 98.0 Lexis Advance (588) 23.8 76.2 WestlawNext (588) 49.5 50.5

QUESTIONS ABOUT RECENT LAW SCHOOL GRADUATES

QUESTION 12. In your opinion, how well do recent law school graduates perform the following components of legal research? [If you do not work with recent law school graduates, please skip this question] Very Well Moderately Adequately Poorly Unacceptably Poorly or

Page 65: A Comparison of Responses from Attorneys and from … · from Law Librarians on Legal Research Practices and Expectations for New Associates . ... F. Case Digests ... legal service

TOC

- 65 -

Well Unacceptably Develop an effective research plan (253) 7.1 27.3 27.3 18.2 1.6 19.4 Use secondary sources effectively (240) 7.1 22.5 22.5 23.7 2.5 26.2 Use critical thinking to evaluate the relevance of case law and other primary sources (255)

7.0 30.6 30.6 18.0 2.0 20.0

Develop appropriate search protocol (239) 10.9 30.5 30.5 10.9 2.1 13.0 Research case law (249) 21.7 42.6 42.6 6.4 0.8 7.2 Research statutes (250) 16.8 38.8 38.8 10.0 2.0 12.0 Research regulations (223) 8.5 24.2 24.2 25.1 4.0 29.1 Research administrative decisions (202) 4.0 20.3 20.3 37.1 6.9 44.0 Research legislative history (210) 4.3 17.1 17.1 35.7 12.0 47.7 Understand the difference between statutes and regulations (229)

24.9 25.3 25.3 13.1 1.3 14.4

Update legal sources using a citator (Shepard’s/KeyCite) (229)

27.1 33.2 33.2 6.0 1.7 7.7

Perform cost-effective research (231) 8.1 19.9 19.9 30.3 7.4 37.7 Use Westlaw online services efficiently (213) 19.2 31.0 31.0 13.6 2.8 16.4 Use Lexis online services efficiently (182) 15.9 30.2 30.2 15.9 2.7 18.6 Research using an online service other than Westlaw or Lexis (174)

10.3 20.1 20.1 28.2 8.0 36.2

Research pleadings, motions, or other court documents (221)

5.0 24.0 24.0 24.4 8.6 33.0

Know when to stop researching (246) 2.4 20.3 20.3 30.5 11.8 42.3 Able to find a variety of non-legal information (212) 3.8 18.4 18.4 31.1 12.2 43.3

Page 66: A Comparison of Responses from Attorneys and from … · from Law Librarians on Legal Research Practices and Expectations for New Associates . ... F. Case Digests ... legal service

TOC

- 66 -

DATA FROM 2012 ALL-SIS SURVEY OF LAW LIBRARIANS11

INTRODUCTORY AND DEMOGRAPHIC QUESTIONS:

QUESTION 1: Which setting best describes your working environment? (182) Law Firm Court/Judicial Govt. Agency –

Federal Govt. Agency –

State In-house/ Corporate

County12 Consultant13 Non-profit Organization

83.5 4.9 4.9 2.2 1.6 1.6 0.5 0.5

QUESTION 2: In which state is your library? (176) State Valid

Percent State Valid

Percent State Valid Percent State Valid Percent

Alabama 1.1 Illinois 6.3 Montana 0 Rhode Island 0 Alaska 0 Indiana 0.5 Nebraska 0.5 South Carolina 0 Arizona 0.5 Iowa 0 Nevada 0 South Dakota 0 Arkansas 0.5 Kansas 0 New Hampshire 0 Tennessee 0.5 California 10.8 Kentucky 1.1 New Jersey 3.8 Texas 2.7 Colorado 0.5 Louisiana 0 New Mexico 0 Utah 0 Connecticut 1.1 Maine 0.5 New York 8.2 Vermont 0.5 Delaware 0 Maryland 2.7 North Carolina 1.1 Virginia 2.7 D.C. 16.3 Massachusetts 4.3 North Dakota 0 Washington 2.2 Florida 1.6 Michigan 1.6 Ohio 2.2 West Virginia 0 Georgia 2.7 Minnesota 2.2 Oklahoma 0.5 Wisconsin 2.7 Hawaii 0 Mississippi 0.5 Oregon 2.2 Wyoming 1.1

11 A total of 184 individuals responded to the law librarian survey. All figures reported represent the percentage of respondents selecting a relevant response. The number of respondents that answered a given question and were counted for purposes of calculating the reported percentages varies by question and is listed for each (see # in parentheses). Each question that asked respondents how frequently their attorneys use particular research tools and resources included the option of responding with “I don’t know.” Each of the questions that asked respondents for their opinions about the research abilities of recent law school graduates included the option of responding “n/a”. Respondents who skipped a question or selected “I don’t know” or “n/a” were not included in calculating the relevant response percentages for that question as reported here. The figures reported are thus considered “valid percentages” for purposes of statistical analysis. A summary is included at the end separately listing for each question the number of respondents who skipped the question or selected “I don’t know” or “n/a” in response. 12 Identified by respondents who selected “other” from the response categories listed and added a narrative description in the space provided. 13 Identified by respondents who selected “other” from the response categories listed and added a narrative description in the space provided.

Page 67: A Comparison of Responses from Attorneys and from … · from Law Librarians on Legal Research Practices and Expectations for New Associates . ... F. Case Digests ... legal service

TOC

- 67 -

Idaho 0.5 Missouri 1.6 Pennsylvania 5.4 International 2.7

QUESTION 3: What is the size of the office in which you work? (180) Solo

Practitioner 2-5 attorneys 6-10 attorneys 11-20

attorneys 21-50

attorneys 51-100

attorneys 101-150 attorneys

151-200 attorneys

201+ attorneys

1.1 0.6 0.6 0.6 12.2 27.2 16.7 11.7 % 29.4

QUESTION 4: How many years have you been employed as a law librarian? (183) 0-4 years 5-9 years 10-19 years 20-29 years 30+ years

7.7 19.1 28.4 31.1 13.7

QUESTION 5: Do you have an MLS, MIS, or equivalent degree? (182)

QUESTION 6: Do you have a JD or equivalent degree? (182)

Yes No Yes No 89.0 11.0 19.8 80.2

QUESTIONS ABOUT RESEARCH PROCESS:

QUESTION 7: In an average week, how much of your time is spent working directly with attorneys on legal research issues? (150) None At least 15 Percent 15-25 Percent 25-50 Percent 50-75 Percent More than 75 Percent 0.7 10.7 14.7 30.0 30.0 14.0

QUESTION 8. When the attorneys you work with begin their legal research, how often do they _____________________? Never Rarely Never or

Rarely Occasionally

Frequently Very Frequently

Frequently or Very Frequently

Start by asking a fellow attorney (109) 0 9.2 9.2 31.2 45.0 14.7 59.7 Start by asking a librarian (136) 0 4.4 4.4 44.9 43.4 7.4 50.8 Start with Google (124) 0 0.8 0.8 17.7 45.2 36.3 81.5 Start with statutory database (116) 0.9 20.7 21.6 43.1 31.9 3.4 35.3 Start by consulting work product (122) 2.5 15.6 18.1 38.5 34.4 9.0 43.4 Start with a secondary source (133) 0.8 17.3 18.1 45.9 30.8 5.3 36.1 Start with case law database (132) 0 4.5 4.5 18.9 54.5 22.0 74.5

Page 68: A Comparison of Responses from Attorneys and from … · from Law Librarians on Legal Research Practices and Expectations for New Associates . ... F. Case Digests ... legal service

TOC

- 68 -

Start with a subject specific practice guide (127) 3.9 22.8 26.7 41.7 28.3 3.1 31.4

QUESTION 9. When the attorneys you work with are conducting their research, how often do they __________________? Never Rarely Never or

Rarely Occasionally

Frequently Very Frequently

Frequently or Very Frequently

Use digest or online equivalent to find a topic & key number (134)

5.2 42.5 47.7 35.8 15.7 0.7 16.4

Use headnote & key numbers in case (135) 3.0 32.6 35.6 41.5 20.7 2.2 22.9 Follow citations in case (138) 0 3.6 3.6 27.5 55.1 13.8 68.9 Use index or TOC with online statutory DB (125)

3.2 35.2 38.4 44.8 15.2 1.6 16.8

Follow citations in secondary sources (134) 2.2 6.7 8.9 31.3 46.3 13.4 59.7 Use terms and connectors (134) 3.0 5.2 8.2 26.1 48.5 17.2 65.7 Follow citations in annotated code (137) 0.7 5.1 5.8 38.0 40.9 15.3 56.2

QUESTIONS ABOUT RESOURCE USAGE:

QUESTION 10. When engaged in legal research, how often do the attorneys you work with __________________? Never Rarely Never or

Rarely Occasionally Frequently Very

Frequently Frequently or Very Frequently

Use print materials (141) 0 9.2 9.2 35.5 44.0 11.3 55.3 Use free internet sources (137) 0 2.9 2.9 16.1 51.8 29.2 81.0 Use fee-based databases (140) 0 0 0 5.7 45.7 48.6 94.3

QUESTION 11. When engaged in legal research, how often do the attorneys you work with use the following research tools ___________________? Never Rarely Never or

Rarely Occasionally Frequently Very

Frequently Frequently or Very Frequently

Treatises (140) 0 3.6 3.6 31.4 45.7 19.3 65.0 Practice guides (137) 3.6 11.7 15.3 32.1 38.0 14.6 52.6 Legal encyclopedias (138) 2.9 30.4 33.3 44.9 17.4 4.3 21.7 Law review/law journal articles (133) 0.8 24.8 25.6 42.1 29.3 3.0 32.3

Page 69: A Comparison of Responses from Attorneys and from … · from Law Librarians on Legal Research Practices and Expectations for New Associates . ... F. Case Digests ... legal service

TOC

- 69 -

American Law Reports (133) 8.3 36.1 44.4 45.9 9.0 0.8 9.8 Case digests (131) 5.3 38.9 44.2 38.9 15.3 1.5 16.8 Restatements (133) 3.8 16.5 20.3 51.9 21.1 6.8 27.9 Shepard’s or KeyCite – for case validation (137)

1.5 2.9 4.4 15.3 42.3 38.0 60.3

Shepard’s or KeyCite – for further research (126)

0.8 9.5 10.3 35.7 33.3 20.6 53.9

Looseleaf services (138) 2.2 7.2 9.4 39.9 37.0 13.8 50.8 Sample legal forms (131) 3.8 20.6 24.4 44.3 22.9 8.4 31.3 Litigation resources (135) 3.0 6.7 9.6 37.8 39.3 13.3 52.6 Transactional resources (129) 4.7 17.1 21.7 35.7 33.3 9.3 42.6

QUESTION 12. How often are the following free internet resources used by attorneys at your office?

Never Rarely Never or Rarely

Occasionally Frequently Very Frequently

Frequently or Very Frequently

Google (133) 0 1.5 1.5 3.0 23.0 72.2 95.5 Google Scholar (125) 1.6 12.0 13.6 33.6 35.2 17.6 52.8 Court web sites (132) 1.5 5.3 6.8 17.4 41.7 34.1 75.1 Government legislative web sites (129) 1.6 13.2 14.7 30.2 41.1 14.0 55.0 Government agency web sites (130) 0.8 6.9 7.7 29.2 44.6 18.5 63.1 Legal blogs (107) 0.9 13.1 14.0 45.8 37.4 2.8 40.2 Law firm web sites (106) 4.7 34.9 39.6 38.7 21.7 0 21.7 Law school library websites (103) 13.6 52.4 66.0 27.2 6.8 0 6.8 Non-profit organization’s website (100) 3.0 40.0 43.0 46.0 9.0 2.0 11.0

QUESTION 13. Which of the following databases, if any, do attorneys have access to in your library? Select all that apply. Yes No Additional analysis derived from the data

Yes No Provide access to at least one of these platforms 68.3 31.7 Provide access to more than one of these platforms 28.2 71.8

Bloomberg Law (141) 12.8 87.2 Lexis Advance (142) 36.6 63.4 WestlawNext (142) 50.0 50.0

Page 70: A Comparison of Responses from Attorneys and from … · from Law Librarians on Legal Research Practices and Expectations for New Associates . ... F. Case Digests ... legal service

TOC

- 70 -

QUESTIONS 14-16. Please check the answer that best describes your subscription plans for [WestlawNext (Q14)] [LexisAdvance (Q15)] [Bloomberg Law (Q16)] at your office.14 My office does not

plan to subscribe to ___________

My office is undecided about subscribing to ___________

My office plans to subscribe to ___________ within the next two years

My office plans to subscribe to _________ within the next year

My office currently subscribes to ____________

WestlawNext (140 respondents) 23.6 20.0 1.4 5.0 50 Lexis Advance (140 respondents) 24.3 25.7 5.0 8.4 36.6 Bloomberg Law (140 respondents)

39.1 37.7 1.4 3.6 18.1

QUESTIONS ABOUT RECENT LAW SCHOOL GRADUATES

QUESTION 17. In your opinion, how well do recent law school graduates perform the following components of legal research? [If you do not work with recent law school graduates, please skip this question] Very Well Moderately

Well Adequately Poorly Unacceptably Poorly or

Unacceptably Develop an effective research plan (130) 0 10 38.5 46.9 4.6 51.5 Use secondary sources effectively (132) 1.5 6.8 32.6 51.5 7.6 59.1 Use critical thinking to evaluate the relevance of case law and other primary sources (125)

11.2 24.8 44.8 14.4 4.8 19.2

Develop appropriate search protocol (132) 0 15.9 39.4 38.6 6.1 44.7 Research case law (130) 8.5 34.8 47.7 7.7 1.5 9.2 Research statutes (131) 4.6 21.4 38.9 32.1 3.1 35.1 Research regulations (131) 0.8 9.9 26.7 50.4 12.2 62.6 Research administrative decisions (130) 0 8.5 26.9 49.2 15.4 64.6 Research legislative history (130) 0 1.5 12.3 57.7 28.5 86.2

14 The first three subscription planning response options were the same for Questions 14, 15, and 16, asking about WestlawNext, LexisAdvance, and Bloomberg Law respectively. However, where Questions 14 and 16 each allowed a respondent to distinguish between plans to either subscribe to WestlawNext or Bloomberg Law within the next year, or indicate that that the respondents office already subscribes to the service, the options provided in Question 13 did not allow a respondent to do so with respect to LexisAdvance, only offering the option of stating “[my office plans to continue with or start a subscription to LexisAdvance within the next year” instead. The percentages reported here, which do distinguish between those already subscribing to LexisAdvance and those who planned to do so within one year, were derived using responses to Question 13 which separately asked respondents if attorneys in their office have access to LexisAdvance in their library.

Page 71: A Comparison of Responses from Attorneys and from … · from Law Librarians on Legal Research Practices and Expectations for New Associates . ... F. Case Digests ... legal service

TOC

- 71 -

Understand the difference between statutes and regulations (132)

7.6 9.1 39.4 34.1 9.8 43.9

Update legal sources using a citator (Shepard’s/KeyCite) (130)

16.2 28.5 38.5 10.8 6.2 16.9

Perform cost-effective research (129) 1.6 5.4 27.9 45.7 19.4 65.1 Use Westlaw online services efficiently (115) 1.7 16.5 40 33.9 7.8 41.7 Use Lexis online services efficiently (108) 1.9 14.8 38.9 38.0 6.5 44.4 Research using an online service other than Westlaw or Lexis (114)

3.5 8.8 39.5 39.5 8.8 48.2

Research pleadings, motions, or other court documents (124)

3.2 8.9 42.7 36.3 8.9 45.2

Know when to stop researching (128) 1.6 7.0 42.2 37.5 11.7 49.2 Able to find a variety of non-legal information (127)

1.6 3.1 27.6 42.5 25.2 67.7

SUMMARY DATA: FREQUENCY OF MISSING, IRRELEVANT, AND RELEVANT RESPONSES

Survey Question # that did not respond

# selecting “I don’t know”

# selecting “n/a”

# selecting relevant response

Q1: Which setting best describes your working environment? 2 n/a n/a 182 Q2: In which state is your library? 8 n/a n/a 176 Q3: What is the size of the office in which you work? 4 n/a n/a 180 Q4: How many years have you been employed as a law librarian? 1 n/a n/a 183 Q5: Do you have an MLS, MIS, or equivalent degree? 2 n/a n/a 182 Q6: Do you have an JD or equivalent degree? 2 n/a n/a 182 Q7: In an average week, how much of your time is spent working directly with attorneys on legal research issues?

34 n/a n/a 150

Q8: When the attorneys you work with begin their legal research, how often do they …

Start by asking a fellow attorney? 37 38 n/a 109 Start by asking a librarian? 37 11 n/a 136 Start with Google? 38 22 n/a 124 Start with statutory database? 37 31 n/a 116 Start by consulting work product? 37 25 n/a 122

Page 72: A Comparison of Responses from Attorneys and from … · from Law Librarians on Legal Research Practices and Expectations for New Associates . ... F. Case Digests ... legal service

TOC

- 72 -

Start with secondary source? 36 15 n/a 133 Start with case law database? 36 16 n/a 132 Start with subject specific practice guide? 38 19 n/a 127 Q9: When the attorneys you work with are conducting their research, how often do they …

Use digest or online equivalent to find topic & key number 37 13 n/a 134 Use headnote and key numbers in case 36 13 n/a 135 Follow citations in a case 37 9 n/a 138 Use index or TOC with online statutory database 36 23 n/a 125 Follow citations in secondary source 37 13 n/a 134 Use terms and connectors 36 14 n/a 134 Follow citations in annotated code 36 11 n/a 137 Q10: When engaged in legal research, how often do the attorneys you work with … Use print materials 42 1 n/a 141 Use free internet sources 42 5 n/a 137 Use fee-based databases 43 1 n/a 140 Q11. When engaged in legal research, how often do the attorneys you work with use the following research tools?

Treatises 42 2 n/a 140 Practice guides 42 5 n/a 137 Legal encyclopedias 42 4 n/a 138 Law review/law journal articles 44 7 n/a 133 American Law Reports 42 9 n/a 133 Case digests 43 10 n/a 131 Restatements 42 9 n/a 133 Shepard’s or KeyCite for case validation 42 5 n/a 137 Shepard’s or KeyCite for additional research 44 14 n/a 126 Looseleaf services 43 3 n/a 138 Sample legal forms 44 9 n/a 131 Litigation resources 43 6 n/a 135 Transactional resources 42 13 n/a 129 Q12: How often are the following free internet resources used by attorneys at your office?

Page 73: A Comparison of Responses from Attorneys and from … · from Law Librarians on Legal Research Practices and Expectations for New Associates . ... F. Case Digests ... legal service

TOC

- 73 -

Google 42 9 n/a 133 Google Scholar 43 16 n/a 125 Court web sites 42 10 n/a 132 Government legislative web sites 42 13 n/a 129 Government agency web sites 44 10 n/a 130 Legal Blogs 42 35 n/a 107 Law firm web sites 42 36 n/a 106 Law school library web sites 43 38 n/a 103 Non-profit organization web sites 43 41 n/a 100 Q13: Which of the following databases, if any, do attorneys have access to in your library?

Bloomberg Law 43 n/a n/a 141 LexisAdvance 42 n/a n/a 142 WestlawNext 42 n/a n/a 142 Q14: Subscription plans for WestlawNext 44 n/a n/a 140 Q15: Subscription plans for LexisAdvance 44 n/a n/a 140 Q16: Subscription plans for Bloomberg Law 44 n/a n/a 140 Q17: In your opinion, how well do recent law school graduates perform the following components of legal research? [If you do not work with recent law school graduates, please skip the question]

Develop an effective research plan 53 n/a 1 130 Use secondary sources effectively 52 n/a 0 132 Use critical thinking to evaluate the relevance of [authorities] 55 n/a 4 125 Develop appropriate search protocol 52 n/a 5 132 Research case law 53 n/a 1 130 Research statutes 52 n/a 1 131 Research regulations 52 n/a 1 131 Research administrative Decisions 53 n/a 1 130 Research legislative history 53 n/a 1 130 Understand the difference between statutes and regulations 52 n/a 0 132 Update using a citator 53 n/a 1 130 Perform cost effective research 52 n/a 3 129 Use Westlaw online services efficiently 54 n/a 15 115

Page 74: A Comparison of Responses from Attorneys and from … · from Law Librarians on Legal Research Practices and Expectations for New Associates . ... F. Case Digests ... legal service

TOC

- 74 -

Use Lexis online services efficiently 55 n/a 21 108 Research using an online service other than Westlaw or Lexis 54 n/a 16 114 Research pleadings, motions, or other court documents 53 n/a 7 124 Know when to stop researching 52 n/a 2 128 Able to find a variety of other non-legal information 53 n/a 4 127

Page 75: A Comparison of Responses from Attorneys and from … · from Law Librarians on Legal Research Practices and Expectations for New Associates . ... F. Case Digests ... legal service

TOC

- 75 -

DATA COMPARING SIMILARLY SITUATED ATTORNEYS AND LAW LIBRARIANS15

DEMOGRAPHICS (composite table)

Profession (280) Valid Percent Setting (280) Valid Percent Office Size Valid Percent Experience (280) Valid Percent Librarian 53.6 Law Firm 76.4 51-100 attorneys 29.6 0-4 years 12.5 Attorney 46.4 Govt. Agency &

Judicial/Court 23.6 101-200 attorneys 29.3 5-9 years 19.6

201+ attorneys 41.1 10-19 years 30.4 20-29 years 23.9 30+ years 13.6

QUESTIONS ABOUT RESEARCH PROCESS:

“In an average week in your practice, how much of your time do you spend doing legal research? | “In an average week, how much of your time is spent working directly with attorneys on legal research issues?” [Q5 –practitioner survey; Q7-librarian survey] (253)

None At least 15 Percent 15-25 Percent 25-50 Percent 50-75 Percent More than 75 Percent 5.76 47.3 24.3 12.3 7.0 3.3

“When beginning research, how often do you …” | “When the attorneys you work with begin their research, how often do they …” [Q6-practitioner survey; Q8-librarian survey] Never Rarely Never or

Rarely Occasionall

y Frequently Very

Frequently Frequently

or Very Frequently

Start by asking a fellow attorney for advice (218) 3.2 16.5 19.7 29.4 32.6 18.3 50.9 Start with Google (228) 5.7 14.0 19.7 23.7 32.9 23.7 56.6 Start with statutory database (221) 5.4 22.2 27.6 35.3 32.6 4.5 37.1 Start by consulting work product (228) 3.5 17.1 20.6 36.0 27.6 15.8 43.4 Start with a secondary source (234) 2.6 22.8 25.4 41.0 27.8 6.4 34.2 Start with case law database (235) 3.4 12.3 15.7 20.9 38.7 24.7 63.4 Start with a subject specific practice guide (230) 3.0 23.0 25.0 38.7 29.6 5.7 35.3

15 A total of 280 respondents from were included in the composite data set, drawn from the practitioner and librarian surveys to facilitate comparison between similarly situated individuals on common questions included on both surveys as described in the Task Force report. As with the tables above, all figures reported represent the percentage of respondents selecting a relevant response (see notes 1 and 6, supra, regarding treatment of respondents who skipped a question or selected “I don’t know” or “n/a”). The number of respondents that answered a given question and were counted for purposes of calculating the reported percentages varies by question and is listed for each (see # in parentheses).

Page 76: A Comparison of Responses from Attorneys and from … · from Law Librarians on Legal Research Practices and Expectations for New Associates . ... F. Case Digests ... legal service

TOC

- 76 -

“When you are researching an issue for your practice, how often do you …” | “When the attorneys you work are conducting their research/ how often do they …” [Q7-practitioner survey; Q9-librarian survey] Never Rarely Never or

Rarely Occasionally Frequently Very

Frequently Frequently or

Very Frequently

Use digest or online equivalent to find a topic & key number (238)

16.0 41.2 57.2 26.5 12.2 4.2 14.4

Use headnote & key numbers in case (238) 10.5 29.0 39.5 33.6 19.7 7.1 26.8 Follow citations in case (242) 3.3 7.0 10.3 19.0 45.9 24.8 70.7 Use index or TOC with online statutory DB (230) 7.0 23.9 30.9 36.5 23.5 9.1 32.6 Follow citations in secondary sources (238) 4.6 12.2 16.8 28.6 38.7 16.0 54.7 Use terms and connectors (240) 5.4 6.7 12.1 20.8 32.9 34.2 67.1 Follow citations in annotated code (239) 4.2 12.6 16.8 33.5 31.0 18.8 49.8

QUESTIONS ABOUT RESOURCE USAGE:

“When performing research for your practice, how often do you …” | “When engaged in legal research, how often do the attorneys you work with …” [Q8-practioner survey; Q10-librarian survey] Never Rarely Never or

Rarely Occasionally Frequently Very

Frequently Frequently or

Very Frequently

Use print materials (242) 2.5 14.5 17.0 31.0 38.8 13.2 52.0 Use free internet sources (239) 2.1 6.3 8.4 22.2 43.5 25.9 69.4 Use fee-based databases (241) 1.7 4.1 5.8 12.0 35.3 46.9 82.2

“When performing legal research for your practice, how often do you utilize the following research tools?” | “When engaged in legal research, how often do the attorneys you work with use the following research tools?” [Q9-practitioner survey; Q11-librarian survey] Never Rarely Never or

Rarely Occasionally Frequently Very

Frequently Frequently or

Very Frequently

Treatises (240) 2.9 12.1 15.0 32.9 35.4 16.7 52.1 Practice guides (237) 6.3 18.1 24.4 33.3 32.5 9.7 42.2 Legal encyclopedias (239) 15.5 33.9 49.4 36.0 11.3 3.3 14.6 Law review/law journal articles (234) 7.7 37.2 44.9 37.2 16.7 1.3 18.0 American Law Reports (235) 17.9 38.3 56.2 36.2 7.2 0.4 7.6 Case digests (234) 16.2 36.3 68.7 33.8 11.5 2.1 13.6 Restatements (235) 17.4 25.5 42.9 40.9 11.9 4.3 16.2

Page 77: A Comparison of Responses from Attorneys and from … · from Law Librarians on Legal Research Practices and Expectations for New Associates . ... F. Case Digests ... legal service

TOC

- 77 -

Shepard’s or KeyCite – for case validation (237) 8.9 12.2 21.1 13.1 33.8 32.1 65.9 Shepard’s or KeyCite – for further research (231) 9.1 15.2 24.3 24.7 28.6 22.5 51.1 Looseleaf services (239) 19.7 17.2 36.9 33.5 21.8 7.9 29.7 Sample legal forms (235) 21.3 30.6 51.9 29.4 14.0 4.7 18.7 Litigation resources (237) 16.0 19.8 35.8 29.1 26.6 8.4 35.0 Transactional resources (233) 22.7 23.6 46.3 24.5 23.6 5.6 29.2

“In the course of your normal practice, how often do you use the following free internet sources for legal research?” | “How often are the following free internet resources used by attorneys at your office?” [Q10-practitioner survey; Q12-librarian survey]

Never Rarely Never or Rarely

Occasionally Frequently Very Frequently

Frequently or Very

Frequently Google (235) 3.4 6.8 10.2 14.5 26.4 48.9 75.3 Google Scholar (228) 30.3 14.0 44.3 18.9 25.0 11.8 36.8 Court web sites (234) 9.4 13.2 22.6 18.4 37.6 21.4 59.0 Government legislative web sites (233) 5.6 18.5 24.1 33.9 28.8 13.3 42.1 Government agency web sites (233) 5.6 13.7 19.3 30.0 32.6 18.0 50.6 Legal blogs (211) 24.6 23.2 47.8 31.8 18.5 1.9 20.4 Law firm web sites (213) 25.8 34.7 60.5 26.8 12.7 0 12.7 Law school library websites (210) 38.6 39.0 77.6 18.1 4.3 0 4.3 Non-profit organization’s website (204) 31.9 34.8 66.7 27.9 4.4 1.0 5.4

QUESTION 11. Which of the following databases, if any, do you use when performing legal research? Select all that apply. Yes No Additional analysis derived from the data

Yes No Use/provide access to at least one of these platforms 73.0 27.0 Use/provide access to more than one of these platforms 26.2 73.8

Bloomberg Law (236) 8.1 91.9 Lexis Advance (237) 34.6 65.4 WestlawNext (237) 58.2 41.8

Page 78: A Comparison of Responses from Attorneys and from … · from Law Librarians on Legal Research Practices and Expectations for New Associates . ... F. Case Digests ... legal service

TOC

- 78 -

QUESTIONS ABOUT RECENT LAW SCHOOL GRADUATES

“In your opinion, how well do recent law school graduates perform the following components of legal research? [If you do not work with recent law school graduates, please skip this question]” [Q12-practitioner survey | Q17-Librarian Survey] Very Well Moderately

Well Adequately Poorly Unacceptably Poorly or

Unacceptably Develop an effective research plan (186) 3.2 17.7 43.0 33.3 2.7 36.0 Use secondary sources effectively (182) 2.7 13.7 34.6 44.0 4.0 48.0 Use critical thinking to evaluate the relevance of case law and other primary sources (182)

11.0 22.5 47.3 16.5 2.7 19.2

Develop appropriate search protocol (181) 4.4 21.0 43.1 27.6 3.9 31.5 Research case law (182) 16.5 34.6 42.9 4.9 1.1 6.0 Research statutes (183) 10.9 23.5 43.7 19.1 2.7 21.8 Research regulations (177) 4.0 12.4 36.7 38.4 8.5 46.9 Research administrative decisions (168) 0.6 12.5 33.9 42.3 10.7 53.0 Research legislative history (172) 1.2 6.4 23.8 50.0 18.6 68.6 Understand the difference between statutes and regulations (177)

13.6 14.7 39.0 25.4 7.3 32.7

Update legal sources using a citator (Shepard’s/KeyCite) (172)

20.3 29.7 39.0 7.0 4.1 11.1

Perform cost-effective research (180) 2.8 10.0 36.1 34.4 16.7 51.1 Use Westlaw online services efficiently (168) 10.1 17.3 39.9 26.8 6.0 32.8 Use Lexis online services efficiently (146) 6.8 15.8 41.8 30.1 5.5 35.6 Research using an online service other than Westlaw or Lexis (147)

4.1 12.2 42.9 32.0 8.8 40.8

Research pleadings, motions, or other court documents (161)

3.7 14.9 39.8 32.9 8.7 41.6

Know when to stop researching (180) 2.2 10.6 41.1 36.7 9.4 46.1 Able to find a variety of non-legal information (169) 1.8 9.5 31.4 38.5 18.9 57.4

Page 79: A Comparison of Responses from Attorneys and from … · from Law Librarians on Legal Research Practices and Expectations for New Associates . ... F. Case Digests ... legal service

Appendix C

Survey Instruments

Survey on Identifying Skills and Knowledge for Legal Practice (Practitioner Survey Instrument)

Survey on Identifying Skills and Knowledge for Legal Practice – Librarian (Librarian Survey Instrument)