how automation is revolutionizing ediscovery€¦ · technology consulting, technical project...
TRANSCRIPT
How Automation is Revolutionizing eDiscovery
Association of Certified eDiscovery Specialists (ACEDS) | Webinar – August 10, 2016 Talking Points
Panel Overview High-Level Consideration:
• Drivers for eDiscovery Automation
• From TREC (2009) to Da Silva Moore (2012) and Beyond
What is the State-of TAR-Art?:
• Comparing TAR Methodologies
• No Legally Sufficient Definition
• Strong Marketing and an Accommodating Judiciary
• Improper Measures of Performance
Dealing with TAR’s Limitations
Other Areas of Automation
A Generational View of eDiscovery Technology
2
Moderator Seminar Panelists for Today’s Guided Discussion
Mary Mack
E-discovery pioneer Mary Mack leads the Association of Certified eDiscovery Specialists (ACEDS) as the executive director. Mary
provides ACEDS and its membership more than a decade of strong credibility and sound leadership within the e-discovery
community.
Mary is known for her skills in relationship and community building as well as for the depth of her e-discovery knowledge. Mary is
frequently sought out by media for comment on industry issues and is the author of A Process of Illumination: The Practical Guide
to Electronic Discovery, considered by many to be the first popular book on e-discovery. She is also the co-editor of the Thomson
Reuters West treatise, eDiscovery for Corporate Counsel. Mary is a member of the Illinois State Bar and the Association of
Corporate Counsel. She received her bachelor’s degree from Le Moyne College and her law degree from Northwestern University School of Law.
Mary Mack| @acedsonline | Association of Certified eDiscovery Specialists (ACEDS) 3
Panelist Seminar Panelists for Today’s Guided Discussion
Doug Austin
Doug Austin is the Vice President of Professional Services for CloudNine. At CloudNine, Doug manages professional services consulting projects for CloudNine clients. Overall, Doug has over 25 years of experience providing legal technology consulting, technical project management and software development services to numerous commercial and
government clients. Doug has managed projects in all phases of the EDRM eDiscovery life cycle. Doug is also the editor of the CloudNine sponsored eDiscovery Daily blog, which has become a trusted resource for eDiscovery news and analysis and is an EDRM Education partner. Since its inception in 2010, eDiscovery Daily has published over 1,400-lifetime posts regarding eDiscovery case law, trends and best practices, which has included case law for over 300 unique cases.
Doug Austin| @Cloud9Discovery | eDiscovery Daily Blog 4
Panelist Seminar Panelists for Today’s Guided Discussion
Bill Dimm
Bill Dimm is the founder and CEO of Hot Neuron LLC. He developed the algorithms for predictive coding,
conceptual clustering, and near-dupe detection used in the company's Clustify software. He is currently writing a
book that is tentatively titled Predictive Coding: Theory & Practice. He has over two decades of experience in the
development and application of sophisticated mathematical models to solve real-world problems in the fields of theoretical physics, mathematical finance, information retrieval, and e-discovery. He has a Ph.D. in theoretical
elementary particle physics from Cornell University.
Bill Dimm| @Clustify | Hot Neuron LLC 5
Panelist Seminar Panelists for Today’s Guided Discussion
Bill Speros
An attorney consulting in evidence management, in Cleveland, Bill Speros advises lawyers on issues of
technologies and techniques to manage evidence, litigation support staff and vendors.
He has served more than 4,000 hours as interim Director of Litigation Support and E-Discovery for the trustee
administering bankruptcy proceedings in the largest Ponzi scheme in history, Madoff Investment Securities.
More commonly, Bill helps develop detailed project plans, marshals staffing and vendor alternatives, and measures
and manages burdens for corporate clients and supports outside counsels’ meet-and-confer negotiations as a
“whispering” expert.
Bill Speros| Speros & Associates, LLC 6
Disclaimer
Ideas expressed here are not necessarily those of our clients or employers and may
simply represent ideas intended to be helpful in the context of this seminar.
7
Budgetary Constraints - 28.9%
Increasing Types of Data - 10.5%
Data Security – 15.8%
Lack of Personnel - 13.2%
Increasing Volumes of Data - 26.3%
Inadequate Tech - 5.3%
N=76
eDiscovery Business Confidence Survey - Spring 2016 ComplexDiscovery.
Drivers: eDiscovery Challenges Top Challenges in Managing eDiscovery Requests
8
Drivers: Business Opportunity
2015 2016 2017 2018 2019 2020
$13.597B
$7.332B
Total Worldwide Market - $13.597B Estimated 63% U.S. | 37% Rest Of World
eDiscovery Software + Services Market Estimated 13.15% CAGR 2015-20
Software Comprises Approximately 28.31% And
Services Comprise Approximately 71.69% Of
Total eDiscovery Market Spending
$3.849B
Software
$9.748B
Services $6.092B
$2.193B
Source: ComplexDiscovery
Automation Opportunity Indicators
• M&A Acceleration
• Venture Capital Investment
• Automation Announcements
Notable (and Non-All Inclusive) TAR Milestones
TREC Study Effectiveness of Technology-Assisted Review (2009)
DaSilva Moore v. Publicis Groupe & MSL Group (February 2012)
Pyrrho Investments Ltd v MWB Property Ltd
(February 2016)
Article - Andrew J. Peck, Magistrate Judge, SDNY: Search, Forward: Will manual document
review and keyword searches be replaced by computer-assisted coding? (October 2011)
White Paper - Maura R. Grossman & Gordon V. Cormack: Technology-Assisted Review in E-Discovery
Can Be More Effective and More Efficient Than Exhaustive Manual Review (April 2011)
10
Time
Acceptance
So, What about Market “Acceptance”?
According to Gartner:
“Predictive coding…has not gained mainstream adoption.
The estimated rate of adoption among enterprises is [only]
about 10% to 15%, while the service providers may reach
50% to 60%.” [emphasis added]
Source: Gartner Market Guide for E-Discovery Solutions, June 2016 11
What is the State-of-TAR-Art? Illustration of TAR 1.0, 2.0 & 3.0
All Documents
TAR 1.0
TAR 2.0
TAR 3.0
12
Review Required (All Candidates Reviewed)
13
Review Required (Produce Without Review)
14
What is the State-of-TAR-Art? As Advertised
“Court orders parties to employ TAR”
“Federal court adopts TAR”
“Judges approve TAR”
Monique Da Silva Moore, et al., v. Publicis Groupe & MSL Group, Civ. No. 11-
1279 (ALC)(AJP) (S.D.N.Y. February 24, 2012), aff’d (S.D.N.Y. April 26, 2012) 15
What is the State-of-TAR-Art? Ruling
“Technology-assisted review can (and does) yield more
accurate results than exhaustive manual review, with much
lower effort.”
“Concerns” that the predictive coding “method lacks the necessary standards for assessing whether its results are
accurate… [or] reliable…or actually works…are premature…better decided ‘down the road,’ when real information is available to the parties and the Court”
Monique Da Silva Moore, et al., v. Publicis Groupe & MSL Group, Civ. No. 11-1279
(ALC)(AJP) (S.D.N.Y. February 24, 2012), aff’d (S.D.N.Y. April 26, 2012)
What is the State-of-TAR-Art? TAR “Defined”
“A technology-assisted review process involves the
interplay of humans and computers to identify the
documents … [and] may involve, in whole or in part, the
use of one or more approaches including, but not limited
to, keyword search, Boolean search, conceptual search,
clustering, machine learning, relevance ranking, and
sampling.”
Grossman & Cormack: Tech. Assisted Review Can Be More Effective and
More Efficient Than Exhaustive Manual Review. XVII Rich. J.L. & Tech 11 (2011) 17
What is the State-of-TAR-Art? TAR “Defined”
[A] computerized system that harnesses human
judgments of one or more Subject Matter Expert(s)… Some TAR methods use Machine Learning Algorithms
[or] other TAR methods derive systematic Rules that
emulate the expert(s)’ decision-making process.
The Grossman-Cormack Glossary of Technology-Assisted Review 7 FED.COURTS.L.REV. 1, 32 (2013)
18
“[Common definitions fail to] designate TAR’s capabilities, operating requirements and constraints…
Instead, those definitions are essentially aspirations and un-
testable puffery.”
Speros: Despite Early Success, Technology Assisted Review’s General Acceptance Is Limited by Its Lack of Definition and, Therefore, Its Lack of Justification:
Accepted for presentation by the March, 2016 ASU-Arkfeld eDiscovery and Digital Evidence Conference
What is the State-of-TAR-Art? TAR “Defined”
19
What is the State-of-TAR-Art? TAR “Defined”
Limitations recognized by its authors 1. For 3 of 5 topics “no significant difference in recall” between TAR and manual reviews (G&C Jolt, p. 44)
2. “[C]onsidered…only the two of eleven teams most likely to demonstrate that TAR can improve upon exhaustive manual
review” (G&C Jolt, p. 48) 3. “The manual reviews were the ‘First-Pass Assessments’” (G&C Jolt, p. 24) by loosely managed volunteers (p. 28).
Other limitations (selected): • William Webber, Re-examining the Effectiveness of Manual Review,
http://www.williamwebber.com/research/papers/w11sire.pdf
• Gerard J. Britton, Courts must reassess assumptions underlying current predictive
• coding protocols, http://postmodern-ediscovery.blogspot.com/2014/07/courts-must-reassess-assumptions_7.html
• William C. Dimm, Predictive Coding: Theory & Practice: http://www.predictivecodingbook.com/
Speros: Despite Early Success, Technology Assisted Review’s General Acceptance Is Limited by Its Lack of Definition and, Therefore, Its Lack of Justification:
Accepted for presentation by the March, 2016 ASU-Arkfeld eDiscovery and Digital Evidence Conference 20
Grossman & Cormack: Tech. Assisted Review Can Be More Effective and
More Efficient Than Exhaustive Manual Review. XVII Rich. J.L. & Tech 11 (2011)
As reported
21
Actual
Underlying
Data
22
What is Minimum Necessary Recall Performance? [Google “tar 80% recall” = About 435,000 results]
• “Thus, a goal of 80% recall — a common TAR target — could well be
reasonable when reviewing for responsive documents.”
• “I'll set the target recall level at 80%.”
• “Thus…you have found 80% of the relevant documents. Your argument is that the cost…to find the remaining 20% of the relevant
documents is unduly burdensome.”
. 23
What is Minimum Necessary Recall Performance?
• I spelt eighty percent of this sentence’s words are korrectly.
• Ninety-nine percent of the distance the student drove was perfect.
24
Total “Recall” Aggregates Importance Risks relate to probative weight
Respon-
sive? Risk Probative
Low High
Yes
High Fatal Hot
Prejudice Foundational
Costs Relevant
Minor Redundantly Relevant
No Low Avoid Data
Dump Junk
Common Rare
25
Total “Recall” Aggregates Topics
Average Topic-By-Topic
26
Dealing with TAR’s Limitations Anecdotal Assessment when AI was Inferior to Human Intelligence
• Capabilities / reliability • Who is the witness? • How do you know you’re done?
•Operating Requirements • Consistency and clarity of language • Properly informed, motivated reviewers
• Limitations • Doc. numbers, images, parent:child, etc. • Single vs multi issues • Doc size: too large, too small • Population richness: too rich, too poor
27
What is the State-of-TAR-Art? Asking TAR to Perform Alchemy
“First state court to order Predictive coding”
“Court orders TAR over the objections of
requesting parties”
Global Aerospace, Inc. v. Landow Aviation, L.P., No. CL61040, 2012 WL 1431215
(Va. Cir. Ct. Apr. 23, 2012); 28
What is the State-of-TAR-Art? Asking TAR to Perform Alchemy
Global Aerospace, Inc. v. Landow Aviation, L.P., No. CL61040, 2012 WL 1431215
(Va. Cir. Ct. Apr. 23, 2012) 29
What is the State-of-TAR-Art? Asking TAR to Perform Alchemy
Global Aerospace, Inc. v. Landow Aviation, L.P., No. CL61040, 2012 WL 1431215
(Va. Cir. Ct. Apr. 23, 2012)
[Requesting party’s brief]
• Company of 5 employees
• Internal network
• No communication with its own attorneys
TAR “Problem of the Producing Party’s
own making”
30
From Review to TAR to Other Artificial Intelligence
Automation
Acceptance
Manual Review of Individual Documents
Technology-Assisted Review
Artificial Intelligence 31
Following the Money – SaaS and Automation
VC Investment in eDiscovery Automation Providers
• Multi-million dollar investments in providers like Logikcull and Everlaw
Emergence of Other Automation Providers
• Other providers also making a splash
Big Boys Taking Note
• Larger Providers like kCura, Ipro and Thomson Reuters have announced SaaS and automation initiatives
Bottom Line: Self-service automation is beginning to change the market – in a big way
32
A Generational View of eDiscovery Technology
1 • Adapted for eDiscovery
• No Task Integration
• No Task Automation
4 • Designed for eDiscovery
• Designed for Task Integration
• Designed for Task Automation
2 • Designed for eDiscovery
• Adapted for Task Integration
• No Task Automation
3 • Designed for eDiscovery
• Designed for Task Integration
• No Task Automation
Automation
Acceptance
33
So, Is Automation Revolutionizing eDiscovery?
Disruptive Innovation – Defined
A disruptive innovation is an innovation that
helps create a new market and value network,
and eventually disrupts an existing market and
value network (over a few years or decades),
displacing established market leaders.
Source: Wikipedia 34
Resources • Is Automation What’s Next (and What’s Necessary) for E-discovery?, Legaltech News, March 23, 2016
• TAR 3.0 Performance, Clustify Blog, January 28, 2016
• Considering Fourth Generation eDiscovery Technology Offerings: Two Approaches, Part One and Part Two,
Complex Discovery, January 8, 2016
• Welcome to 2016! The Age of eDiscovery Automation is Upon Us!: eDiscovery Trends, eDiscovery Daily,
January 4, 2016
• Predictive Coding Performance and the Silly F1 Score, Clustify Blog, May 2013
• Not Only is the Age of eDiscovery Automation Upon Us, But So is the Age of Cloud-Based eDiscovery:
eDiscovery Trends, eDiscovery Daily, January 8, 2016
• eDiscovery Vendor Viability: Comparisons Beyond Technology and Talent, Complex Discovery, May 13, 2015
• E-discovery: Effects of automated technologies on electronic document preservation and review obligations,
Inside Counsel, December 18, 2012
35
Follow Up
Association of Certified eDiscovery
Specialists (ACEDS)
CloudNine™
Speros & Associates, LLC
Hot Neuron LLC
36 Association of Certified eDiscovery Specialists (ACEDS) | Webinar – August 10, 2016