Panelists
David Horrigan, Esq.
Analyst, eDiscovery and Information Governance – The 451 Group
Doug Stewart
Vice President of Technology and Innovation, Daegis
Anita Engles
Vice President of Product Marketing,
Daegis
3
Learning from Big Data:
Simplify Your Workflow Using Technology Assisted Review
David Horrigan, 451 Research
451 Research
Global research analyst firm
A division of The 451 Group
451 founded in 1999, 200+ staff
Daily qualitative & quantitative research
1,000+ clients served
Publications, analyst advice, global events
Technology Assisted Review: A Necessity?
Sample eDiscovery Costs
$17,183 in an Intellectual Property Case
$27.1 Million in a Product Liability Case
$1.8 Million Median Cost
$20,000 per GB in About a Third of Cases
$900,000 per GB in One Case (Less than a DVD)
Source: Rand Corporation, 2012
Technology Assisted Review: A Necessity?
Breakdown of eDiscovery Costs
Review 73%
Processing 19%
Collection 8%
Source: Rand Corporation, 2012
Technology Assisted Review: A Risk?
Important Judicial Observations on Assisted Review:
1) IT’S NOT MAGIC “Computer-assisted review is not a magic, Staples-Easy-Button
solution appropriate for all cases.”
2) IT’S NOT MYSTERIOUS “Every person who uses email uses predictive coding, even if they
do not realize it. The "spam filter" is an example of predictive coding.”
--U.S. Magistrate Judge Andrew Peck
in Da Silva Moore v. Publicis Groupe
Technology Assisted Review: A Risk?
2006 Amendments Federal Rules of Civil Procedure
•Amended Rules 16, 26, 33, 34, 37, and 45
•Introduced “Electronically Stored Information” (ESI)
•Important Provisions: • Initial Disclosures, Rule 26(a)(1)(B) • Meet and Confer, Rule 26(f) • Form of Production, Rule 34(b) • Reasonably Accessible, Undue Burden Cost, Rule 26(b)(2) • Clawback “Light,” Rule 26(b)(5)(b) • Safe Harbor for Good Faith, Routine Destruction, Rule 37(f)
Technology Assisted Review: A Risk?
2008 Enactment
Federal Rule of Evidence 502
•Limits Waivers of Attorney-Client Privilege and Work Product Doctrine
•Rule 502(b)-No Waiver for Unintentional Disclosure if:
• “Reasonable attempts to prevent disclosure,” Rule 502(b)(2)
AND
• Prompt, reasonable steps to rectify error, Rule 502(b)(3)
Technology Assisted Review: A Risk?
2008 Enactment
Federal Rule of Evidence 502
•Rule 502(d)-Controlling Effect of a Court Order. A federal court may order that the privilege or protection is not waived by disclosure connected with the litigation pending before the court — in which event the disclosure is also not a waiver in any other federal or state proceeding.
•The “Get Out of Jail Free” Card for Inadvertent Disclosure
•“If You’re not asking for a 502(d) order, it’s close to malpractice.”
--U.S. Magistrate Judge Andrew Peck
Technology Assisted Review
Notable Current Cases:
Da Silva Moore v. Publicis Groupe SA, No. 11-CV-1279 (S.D.N.Y. filed Feb. 24, 2011)
--The Landmark Case: First known judicial approval of computer assisted review
--Plaintiff and Defendant Agree on the use of computer assisted review. They disagree on methodology.
Technology Assisted Review
Notable Current Cases:
Kleen Prods. LLC v. Packaging Corp. of Am., No. 1:11-CV-01279-ALC-AJP (N.D. Ill. filed Sept. 9, 2010)
--Plaintiffs Attempts to Force Defendants to redo production using “Content Based Advanced Analytics.”
--Judge Nan Nolan Pushes for Keyword Compromise as she attempts to get parties to agree on a Boolean approach.
--Keyword v. TAR Tussle is avoided for now.
Technology Assisted Review
Notable Current Cases:
Global Aerospace, Inc. v. Landow Aviation, L.P., No. CL 61040 (Va. Cir. Ct. Loudoun Co. Apr. 23, 2012)
--Very Significant as Court Orders TAR Over Objections: The Defendants sought to use TAR, but Plaintiffs objected.
--Also Significant as a State Court Case
Technology Assisted Review
Notable Current Cases:
EOHB, Inc. v. HOL Holdings, LLC, No. CA-7409-VCL (Del. Ch. Oct. 15, 2012)
--Very Significant as Court Orders TAR Over Objections of Both Parties
--Significant as a State Court Case
--Significant in the Order to Use the Same Technology Vendor
Technology Assisted Review: Moving Forward
Remember, TAR has been used in hundreds of matters, but we have only five known matters where there’s been judicial intervention. At the same time, question the headlines…
Why Hire a Lawyer? Computers are Cheaper
--The Wall Street Journal, June 18, 2012
Will Computers Soon Replace Your Lawyer?
--Thomson Reuters, June 5, 2012
18
Doug Stewart
Vice President of Technology and Innovation, Daegis
About Daegis • Provides collections, processing, managed revenue, project management
and eDiscovery Analytic consulting services.
• Daegis eDiscovery Platform delivers cost-saving solutions for corporations and law firms.
• Acumen, a Predictive coding tool for Technology Assisted Review and Cross-Matter Management, enables retention and reuse of case documents and attorney work product.
Process Driven
• Enforces Process to Ensure Defensibility
• Validates Results Automatically
• Fits Technology to the Review Process
• Optimizes the Investment in Human Review
• Plays to the Strengths of Humans and Computers
• Leverages Proven “Big Data” Tools and Approaches
20
Human Review
Recall Precision F1 Score
TREC 2009-1 75.6% 5.0% 9.5%
TREC 2009-2 79.9% 26.7% 40.0%
TREC 2009-3 25.2% 12.5% 16.7%
TREC 2009-4 36.9% 25.5% 30.2%
TREC 2009-5 79.0% 89.0% 83.7%
TREC 2009 Average 59.3% 31.7% 36.0%
21
Familiar Process
Target review set through filtering
Supervising Attorney w/ Case Knowledge to answer questions
Project Management
Qualified, well trained reviewers
Random sampling for QC and accuracy assessments Identify privileged documents
Compile performance metrics
Compile accuracy metrics
QC
100% eyes-on review by reviewers
Traditional Review
TAR
Approaches
• Seed Sets • If validated can provide good training • Introduces bias / expensive
• Random Sample • No bias • Ignores useful knowledge
• Search Terms • Validated terms provide inexpensive training • Introduces bias
• Other Methods • Combine Approaches / Hybrid • Sample from document map / clusters • Other Big Data techniques
24
TAR Adoption
RAND Institute Report:
We propose that the best way to … bring predictive coding into the mainstream is for innovative, public-spirited litigants to take bold steps by using this technology for large-scale e-discovery efforts and to proclaim its use in an open and transparent manner.
Pace, Nicholas M., and Laura Zakaras. "Where the Money Goes." (2012). Summary p. XIX
25
Advice
• Process Driven
• Support document review best practices
• Not for Every Matter
• Types, volumes and formats
• Cost
• Should be set to encourage use
• Decision to Stop
• Prepare and plan for this before starting
26
Thank You
For more information, contact us at: [email protected] San Francisco 235 Montgomery Street, Suite 350 San Francisco, CA 94104 415.364.7300
New York 17 Battery Place, Suite 300 New York, NY 10004 212.867.3044
28