how automation is revolutionizing ediscovery€¦ · technology consulting, technical project...

How Automation is Revolutionizing eDiscovery

Association of Certified eDiscovery Specialists (ACEDS) | Webinar – August 10, 2016 Talking Points

Panel Overview High-Level Consideration:

• Drivers for eDiscovery Automation

• From TREC (2009) to Da Silva Moore (2012) and Beyond

What is the State-of TAR-Art?:

• Comparing TAR Methodologies

• No Legally Sufficient Definition

• Strong Marketing and an Accommodating Judiciary

• Improper Measures of Performance

Dealing with TAR’s Limitations

Other Areas of Automation

A Generational View of eDiscovery Technology

2

Moderator Seminar Panelists for Today’s Guided Discussion

Mary Mack

E-discovery pioneer Mary Mack leads the Association of Certified eDiscovery Specialists (ACEDS) as the executive director. Mary

provides ACEDS and its membership more than a decade of strong credibility and sound leadership within the e-discovery

community.

Mary is known for her skills in relationship and community building as well as for the depth of her e-discovery knowledge. Mary is

frequently sought out by media for comment on industry issues and is the author of A Process of Illumination: The Practical Guide

to Electronic Discovery, considered by many to be the first popular book on e-discovery. She is also the co-editor of the Thomson

Reuters West treatise, eDiscovery for Corporate Counsel. Mary is a member of the Illinois State Bar and the Association of

Corporate Counsel. She received her bachelor’s degree from Le Moyne College and her law degree from Northwestern University School of Law.

Mary Mack| @acedsonline | Association of Certified eDiscovery Specialists (ACEDS) 3

Panelist Seminar Panelists for Today’s Guided Discussion

Doug Austin

Doug Austin is the Vice President of Professional Services for CloudNine. At CloudNine, Doug manages professional services consulting projects for CloudNine clients. Overall, Doug has over 25 years of experience providing legal technology consulting, technical project management and software development services to numerous commercial and

government clients. Doug has managed projects in all phases of the EDRM eDiscovery life cycle. Doug is also the editor of the CloudNine sponsored eDiscovery Daily blog, which has become a trusted resource for eDiscovery news and analysis and is an EDRM Education partner. Since its inception in 2010, eDiscovery Daily has published over 1,400-lifetime posts regarding eDiscovery case law, trends and best practices, which has included case law for over 300 unique cases.

Doug Austin| @Cloud9Discovery | eDiscovery Daily Blog 4

Panelist Seminar Panelists for Today’s Guided Discussion

Bill Dimm

Bill Dimm is the founder and CEO of Hot Neuron LLC. He developed the algorithms for predictive coding,

conceptual clustering, and near-dupe detection used in the company's Clustify software. He is currently writing a

book that is tentatively titled Predictive Coding: Theory & Practice. He has over two decades of experience in the

development and application of sophisticated mathematical models to solve real-world problems in the fields of theoretical physics, mathematical finance, information retrieval, and e-discovery. He has a Ph.D. in theoretical

elementary particle physics from Cornell University.

Bill Dimm| @Clustify | Hot Neuron LLC 5

Panelist Seminar Panelists for Today’s Guided Discussion

Bill Speros

An attorney consulting in evidence management, in Cleveland, Bill Speros advises lawyers on issues of

technologies and techniques to manage evidence, litigation support staff and vendors.

He has served more than 4,000 hours as interim Director of Litigation Support and E-Discovery for the trustee

administering bankruptcy proceedings in the largest Ponzi scheme in history, Madoff Investment Securities.

More commonly, Bill helps develop detailed project plans, marshals staffing and vendor alternatives, and measures

and manages burdens for corporate clients and supports outside counsels’ meet-and-confer negotiations as a

“whispering” expert.

Bill Speros| Speros & Associates, LLC 6

Disclaimer

Ideas expressed here are not necessarily those of our clients or employers and may

simply represent ideas intended to be helpful in the context of this seminar.

7

Budgetary Constraints - 28.9%

Increasing Types of Data - 10.5%

Data Security – 15.8%

Lack of Personnel - 13.2%

Increasing Volumes of Data - 26.3%

Inadequate Tech - 5.3%

N=76

eDiscovery Business Confidence Survey - Spring 2016 ComplexDiscovery.

Drivers: eDiscovery Challenges Top Challenges in Managing eDiscovery Requests

8

Drivers: Business Opportunity

2015 2016 2017 2018 2019 2020

$13.597B

$7.332B

Total Worldwide Market - $13.597B Estimated 63% U.S. | 37% Rest Of World

eDiscovery Software + Services Market Estimated 13.15% CAGR 2015-20

Software Comprises Approximately 28.31% And

Services Comprise Approximately 71.69% Of

Total eDiscovery Market Spending

$3.849B

Software

$9.748B

Services $6.092B

$2.193B

Source: ComplexDiscovery

Automation Opportunity Indicators

• M&A Acceleration

• Venture Capital Investment

• Automation Announcements

Notable (and Non-All Inclusive) TAR Milestones

TREC Study Effectiveness of Technology-Assisted Review (2009)

DaSilva Moore v. Publicis Groupe & MSL Group (February 2012)

Pyrrho Investments Ltd v MWB Property Ltd

(February 2016)

Article - Andrew J. Peck, Magistrate Judge, SDNY: Search, Forward: Will manual document

review and keyword searches be replaced by computer-assisted coding? (October 2011)

White Paper - Maura R. Grossman & Gordon V. Cormack: Technology-Assisted Review in E-Discovery

Can Be More Effective and More Efficient Than Exhaustive Manual Review (April 2011)

10

Time

Acceptance

So, What about Market “Acceptance”?

According to Gartner:

“Predictive coding…has not gained mainstream adoption.

The estimated rate of adoption among enterprises is [only]

about 10% to 15%, while the service providers may reach

50% to 60%.” [emphasis added]

Source: Gartner Market Guide for E-Discovery Solutions, June 2016 11

What is the State-of-TAR-Art? Illustration of TAR 1.0, 2.0 & 3.0

All Documents

TAR 1.0

TAR 2.0

TAR 3.0

12

Review Required (All Candidates Reviewed)

13

Review Required (Produce Without Review)

14

What is the State-of-TAR-Art? As Advertised

“Court orders parties to employ TAR”

“Federal court adopts TAR”

“Judges approve TAR”

Monique Da Silva Moore, et al., v. Publicis Groupe & MSL Group, Civ. No. 11-

1279 (ALC)(AJP) (S.D.N.Y. February 24, 2012), aff’d (S.D.N.Y. April 26, 2012) 15

What is the State-of-TAR-Art? Ruling

“Technology-assisted review can (and does) yield more

accurate results than exhaustive manual review, with much

lower effort.”

“Concerns” that the predictive coding “method lacks the necessary standards for assessing whether its results are

accurate… [or] reliable…or actually works…are premature…better decided ‘down the road,’ when real information is available to the parties and the Court”

Monique Da Silva Moore, et al., v. Publicis Groupe & MSL Group, Civ. No. 11-1279

(ALC)(AJP) (S.D.N.Y. February 24, 2012), aff’d (S.D.N.Y. April 26, 2012)

What is the State-of-TAR-Art? TAR “Defined”

“A technology-assisted review process involves the

interplay of humans and computers to identify the

documents … [and] may involve, in whole or in part, the

use of one or more approaches including, but not limited

to, keyword search, Boolean search, conceptual search,

clustering, machine learning, relevance ranking, and

sampling.”

Grossman & Cormack: Tech. Assisted Review Can Be More Effective and

More Efficient Than Exhaustive Manual Review. XVII Rich. J.L. & Tech 11 (2011) 17

What is the State-of-TAR-Art? TAR “Defined”

[A] computerized system that harnesses human

judgments of one or more Subject Matter Expert(s)… Some TAR methods use Machine Learning Algorithms

[or] other TAR methods derive systematic Rules that

emulate the expert(s)’ decision-making process.

The Grossman-Cormack Glossary of Technology-Assisted Review 7 FED.COURTS.L.REV. 1, 32 (2013)

18

“[Common definitions fail to] designate TAR’s capabilities, operating requirements and constraints…

Instead, those definitions are essentially aspirations and un-

testable puffery.”

Speros: Despite Early Success, Technology Assisted Review’s General Acceptance Is Limited by Its Lack of Definition and, Therefore, Its Lack of Justification:

Accepted for presentation by the March, 2016 ASU-Arkfeld eDiscovery and Digital Evidence Conference

What is the State-of-TAR-Art? TAR “Defined”

19

What is the State-of-TAR-Art? TAR “Defined”

Limitations recognized by its authors 1. For 3 of 5 topics “no significant difference in recall” between TAR and manual reviews (G&C Jolt, p. 44)

2. “[C]onsidered…only the two of eleven teams most likely to demonstrate that TAR can improve upon exhaustive manual

review” (G&C Jolt, p. 48) 3. “The manual reviews were the ‘First-Pass Assessments’” (G&C Jolt, p. 24) by loosely managed volunteers (p. 28).

Other limitations (selected): • William Webber, Re-examining the Effectiveness of Manual Review,

http://www.williamwebber.com/research/papers/w11sire.pdf

• Gerard J. Britton, Courts must reassess assumptions underlying current predictive

• coding protocols, http://postmodern-ediscovery.blogspot.com/2014/07/courts-must-reassess-assumptions_7.html

• William C. Dimm, Predictive Coding: Theory & Practice: http://www.predictivecodingbook.com/

Speros: Despite Early Success, Technology Assisted Review’s General Acceptance Is Limited by Its Lack of Definition and, Therefore, Its Lack of Justification:

Accepted for presentation by the March, 2016 ASU-Arkfeld eDiscovery and Digital Evidence Conference 20

Grossman & Cormack: Tech. Assisted Review Can Be More Effective and

More Efficient Than Exhaustive Manual Review. XVII Rich. J.L. & Tech 11 (2011)

As reported

21

Actual

Underlying

Data

22

What is Minimum Necessary Recall Performance? [Google “tar 80% recall” = About 435,000 results]

• “Thus, a goal of 80% recall — a common TAR target — could well be

reasonable when reviewing for responsive documents.”

• “I'll set the target recall level at 80%.”

• “Thus…you have found 80% of the relevant documents. Your argument is that the cost…to find the remaining 20% of the relevant

documents is unduly burdensome.”

. 23

What is Minimum Necessary Recall Performance?

• I spelt eighty percent of this sentence’s words are korrectly.

• Ninety-nine percent of the distance the student drove was perfect.

24

Total “Recall” Aggregates Importance Risks relate to probative weight

Respon-

sive? Risk Probative

Low High

Yes

High Fatal Hot

Prejudice Foundational

Costs Relevant

Minor Redundantly Relevant

No Low Avoid Data

Dump Junk

Common Rare

25

Total “Recall” Aggregates Topics

Average Topic-By-Topic

26

Dealing with TAR’s Limitations Anecdotal Assessment when AI was Inferior to Human Intelligence

• Capabilities / reliability • Who is the witness? • How do you know you’re done?

•Operating Requirements • Consistency and clarity of language • Properly informed, motivated reviewers

• Limitations • Doc. numbers, images, parent:child, etc. • Single vs multi issues • Doc size: too large, too small • Population richness: too rich, too poor

27

What is the State-of-TAR-Art? Asking TAR to Perform Alchemy

“First state court to order Predictive coding”

“Court orders TAR over the objections of

requesting parties”

Global Aerospace, Inc. v. Landow Aviation, L.P., No. CL61040, 2012 WL 1431215

(Va. Cir. Ct. Apr. 23, 2012); 28

What is the State-of-TAR-Art? Asking TAR to Perform Alchemy

Global Aerospace, Inc. v. Landow Aviation, L.P., No. CL61040, 2012 WL 1431215

(Va. Cir. Ct. Apr. 23, 2012) 29

What is the State-of-TAR-Art? Asking TAR to Perform Alchemy

Global Aerospace, Inc. v. Landow Aviation, L.P., No. CL61040, 2012 WL 1431215

(Va. Cir. Ct. Apr. 23, 2012)

[Requesting party’s brief]

• Company of 5 employees

• Internal network

• No communication with its own attorneys

TAR “Problem of the Producing Party’s

own making”

30

From Review to TAR to Other Artificial Intelligence

Automation

Acceptance

Manual Review of Individual Documents

Technology-Assisted Review

Artificial Intelligence 31

Following the Money – SaaS and Automation

VC Investment in eDiscovery Automation Providers

• Multi-million dollar investments in providers like Logikcull and Everlaw

Emergence of Other Automation Providers

• Other providers also making a splash

Big Boys Taking Note

• Larger Providers like kCura, Ipro and Thomson Reuters have announced SaaS and automation initiatives

Bottom Line: Self-service automation is beginning to change the market – in a big way

32

A Generational View of eDiscovery Technology

1 • Adapted for eDiscovery

• No Task Integration

• No Task Automation

4 • Designed for eDiscovery

• Designed for Task Integration

• Designed for Task Automation

2 • Designed for eDiscovery

• Adapted for Task Integration

• No Task Automation

3 • Designed for eDiscovery

• Designed for Task Integration

• No Task Automation

Automation

Acceptance

33

So, Is Automation Revolutionizing eDiscovery?

Disruptive Innovation – Defined

A disruptive innovation is an innovation that

helps create a new market and value network,

and eventually disrupts an existing market and

value network (over a few years or decades),

displacing established market leaders.

Source: Wikipedia 34

Resources • Is Automation What’s Next (and What’s Necessary) for E-discovery?, Legaltech News, March 23, 2016

• TAR 3.0 Performance, Clustify Blog, January 28, 2016

• Considering Fourth Generation eDiscovery Technology Offerings: Two Approaches, Part One and Part Two,

Complex Discovery, January 8, 2016

• Welcome to 2016! The Age of eDiscovery Automation is Upon Us!: eDiscovery Trends, eDiscovery Daily,

January 4, 2016

• Predictive Coding Performance and the Silly F1 Score, Clustify Blog, May 2013

• Not Only is the Age of eDiscovery Automation Upon Us, But So is the Age of Cloud-Based eDiscovery:

eDiscovery Trends, eDiscovery Daily, January 8, 2016

• eDiscovery Vendor Viability: Comparisons Beyond Technology and Talent, Complex Discovery, May 13, 2015

• E-discovery: Effects of automated technologies on electronic document preservation and review obligations,

Inside Counsel, December 18, 2012

35

Follow Up

mary.mack@barbripa.com

Association of Certified eDiscovery

Specialists (ACEDS)

daustin@ediscovery.co

CloudNine™

speros@speros.net

Speros & Associates, LLC

bdimm@hotneuron.com

Hot Neuron LLC

36 Association of Certified eDiscovery Specialists (ACEDS) | Webinar – August 10, 2016