sf iia – emerging it risks: the road ahead · ontology engineering. neuralnetworks....

Artificial Intelligence in AuditingBrian Crowley, Audit & Assurance Senior Manager, Deloitte & Touche LLP

SF IIA – Emerging IT Risks: The Road Ahead

March 28, 2019

AI Overview

© 2019 Deloitte Development LLC. All rights reserved.

Technology advances and the digital landscape are facilitating a fourth industrial revolution and a seismic shift to a data driven era of real-time auditing.

A shift in the Audit Profession

Artificial Intelligence

3


Why Now?

90% of all data that exists today was generated in the past two years1

1 “10 Key Marketing Trends for 2017” IBM Marketing Cloudhttps://public.dhe.ibm.com/common/ssi/ecm/wr/en/wrl12345usen/watson-customer-engagement-watson-marketing-wr-other-papers-and-reports-wrl12345usen-20170719.pdf

Source: Cisco Complete Visual Networking Index Forecast, 2017-2022

4

https://public.dhe.ibm.com/common/ssi/ecm/wr/en/wrl12345usen/watson-customer-engagement-watson-marketing-wr-other-papers-and-reports-wrl12345usen-20170719.pdf

Creating business value with artificial intelligence© 2017 Deloitte Development LLC. 5


A useful definition of AI

The theory and development of computer systems able to perform tasks that normally require human intelligence.1

1 Oxford Dictionaries, “Definition of artificial intelligence.”

Defining artificial intelligence

5

© 2019 Deloitte Development LLC. All rights reserved. 6

Pattern Recognition

Supervised Learning

Knowledge Representation

Reinforced Learning

Natural Language Understanding

Artificial intelligence is really an umbrella term that covers a number of technologies and disciplines:

Machine Learning

Deep Learning

Natural Language Processing

Machine Translation

Speech Recognition

Text to Speech

Image Recognition

Language Modeling

Classification and Ranking algorithms

Probabilistic Methods

Un-supervised Learning

Ontology engineering

Neural Networks

Expert Systems

neural machine translation

Recursive NeuralNetworks

Clustering

Parsing

Artificial Intelligence

Machine LearningSystems that can extractpatterns and predictoutcomes

Deep LearningMulti-layer neural networks that learn complexfeatures by building up layers of representations of simpler features

Expert Systems

Make inferences from pre-

programmed knowledge

Research advances in techniques known as Deep Learning have been behind many of the recent thought-provoking applications of AI

Hierarchy of Artificial Intelligence



Modern AI learns to map inputs to outputs based on patterns (“features”) in the dataR

ule

s-b

ased

sy

stem

Define fields to describe fruit

<Sweetness= ? >

A system to predict whether someone will like a specific fruit…

<Shape= ? >

<Color= ? >

Pre-program rules

If <round> & <sweet>

Or if <red> & not <sour>

Or if <green> & <sour>

Provide input and get fixed output or error<Apple> = Like

<Kiwi> = Like

<Banana> = ???

Define features to describe fruit

<Sweetness= ? >

<Shape= ? >

<Color= ? >

Provide training examples

Like <Sweet, Round, Red>

Like <Sweet, Round, Yellow>

Dislike <Sour, Round, Green>

Learn to distinguish likes from dislikes

<Sweetness>

<Shape>

Dislike

Like

Provide training examples Learn features to describe fruit

<Sweetness= ? >

<Shape= ? >

<Color= ? >

<Citrus= ? >

<Berry= ? >

<Texture = ? >

Learn to distinguish likes from dislikesLike Dislike Like Dislike

Cla

ssic

M

ach

ine

Lear

nin

g

Dee

p L

earn

ing


2007Chess is solved by

memorizing 39 trillion potential end game positions over 18

years.

2017AlphaGo Zero learns

from scratch by playing against itself, beating the original after only

three days.Repurposed for chess, it

then defeats the best chess engine after only

4 hours.

This has allowed AI to overcome the ambiguity and complexity required for real-world applications

Infer what happened to characters in a story

Identifying objects in a

scene

Drive on city streets and

highways

Determine if a growth is cancerous

Generate images from a description

AI in the Audit Profession (Internal & External)


11

How auditing has changed

Now

• Data-driven using predictive algorithms and proprietary models

• Risk-responsive; stratified populations; entire populations, focus on outliers

• Data flows directly from client through integrated innovation applications

• Efficient, repeatable processes leveraging automation and artificial intelligence

• Value-added insights that are below the surface and provide an objective outside-in perspective

Then

Risk assessments that are broad; based on dollar value

Audit Procedures that are sample based with Excel-based tools. Manually intensive; time consuming

Conclusions that focus on compliance

11


Use of AI in an Audit Setting

Providing suggestions based on rules or previous examples Using known rules and past decisions to suggest an outcome that is the most likely match for a new fact pattern

1

12


Current StateProviding suggestions based on rules or previous examples

Judgment/DecisionInput Data

Training ExperienceKnowledge

13


Future StateProviding suggestions based on rules or previous examples

Judgment/Decision

Input Data

Experience

Training

KnowledgeOutput Data (Suggestion)

Context Causation

FEEDBACK LOOP(Redesign, machine

learning)14

AI Technology




Extracting information from unstructured sources Creating usable data from unstructured documents such as scanned documents or websites

1

2

15


Machine encoded text

Name entity recognition:

detect names & classify their

category

Relation extraction:

detect & classify relations between

entities

Event extraction:

who did what to whom, when &

where

Coreference resolution:

detect multiple mentions of the

same entity

Extracting information from unstructured sources

Could be given via a digital document,

website, etc

Underlying Techniques

• OCR (bounding box detection and character recognition

• Name entity recognition

• Relation extraction

• Event extraction

• Coreference resolution

OCR

Bounding box detection

Characterrecognition

2

ABC Bank shall bear an interest rate annually equal to theAdjusted LIBOR Rate for the Interest Period in effect plus 2% from Company XYZ.

Who What When Why Whom


Document analysis with ArgusExtracting information from unstructured sources

Benefits

Greater efficiency

Higher quality testing

Insights and summaries

Heat map summary of compared documents

Automatic key field extraction

Simple interface foranalysis of key fields

17




Extracting information from unstructured sources Creating usable data from unstructured documents such as scanned documents or websites

Identifying items that are abnormal without explicit rules Learning normal patterns within a process, client, industry, or other dataset to then identify unusual deviations that should be investigated

1

2

318


Example: Identify unusual trends in account balances and financial ratiosIdentifying items that are abnormal without explicit rules

Context anomalyPoint anomalyCollective anomaly

0

2

4

Q1 Q2 Q3 Q4

Cost of Sales ($M)

Revenue ($M)

Decrease in costs of sales, increase in revenues

Example: In 2017, Company A sells $2M of products to Customer B. Historical margin for this product is 50% (implying costs of sales of $1M). Actual recorded costs of sales is $0.5M.

Cause: The Company inappropriately recorded revenue when it still retained the risks and rewards of ownership of the product (i.e., the inventory was still in their warehouse; an inappropriate bill-and-hold arrangement – costs are typically recorded when inventory is reduced). Fraud!

AI would flag usual trends of this nature to help the auditor spend their time in effective areas of investigation

3

AI Governance & Controls Discussion


AI Governance – What’s Happening

• Organizations are now capitalizing on “data as an asset”; implementing functions and to drive progress and accountability on a variety of data initiatives, with an emphasis on AI-enabled solutions

• Challenges of deploying AI solutions without enterprise governance and oversight is emerging as an enterprise-wide risk

• Solutions often lack coordination with the enterprise, and/or across lines of business (e.g. as it relates to data, techniques, monitoring, etc.)

• Companies are exploring ways to extend existing governance constructs to address this gap – industry-specific regulations have provided accelerators in some areas (e.g. Model Risk – financial services) but there is no silver bullet or one-size-fits-all solution

• Boards and C-suite leaders (CIO, COO, CRO, CDO, CISO, etc.) are becoming aware of the need to have an accountable leader and supporting function to be focused on data, AI implementation, and AI governance

Organizations are realizing voluntary self-governance is a risk at the business unit level; governance & control over AI solutions, along with ethical use, requires both enterprise-wide

coordination and Board-level visibility

A 2017 Deloitte University Press study indicated that as many as 86% of US respondents would sever ties with an organization if it were revealed that the company used their data unethically.“To share or not to share: What consumers really think about sharing their personal information”, Deloitte University Press, 2017


These seven components for AI Governance address existing and emerging risks across the lifecycle of an AI model/algorithm

AI Governance Components

1. TechniqueSpecific technique or combination of techniques that is used to address a specific use case or business problem (e.g., language processing, neural network, image recognition)

2. DataData sets (internal or external) used to build and train AI models/algorithms, and their level of curation and fit-for-use (i.e., availability of vectors, weights, results)

3. Policies, Standards & Controls Organizational constructs that establish the design principles and guardrails for the development, deployment, and dispositioning of AI models/algorithms

4. Validation & Testing Mechanism to review, test, and monitor the development and deployment of AI models/algorithms

5. Data Science Platform & Infrastructure

Operational and technological resources leveraged to build, operate and/or monitor AI models/algorithms

6. Talent & Workforce Skills and people required to drive and sustain the development, operation, and monitoring of AI

7. Industry & Regulatory Alignment Awareness and alignment with relevant regulations and/or industry standards related with the use of AI models/algorithms


AI Governance: “Notoriously Tough” Problems

Vendor “Black Boxes”

• Who is responsible for testing vendor AI?• Do we allow the use of AI-enabled vendor

“black boxes”?• How should (can?) we test vendor AI?

• Explicit AI Governance expectations written in vendor contracts

• Maintain inventory of AI usage by vendors• Periodic testing schedule over sample of vendor AI,

prioritized by risk level

Role of Policy & Controls

• What is the right balance between policy and controls?

• What existing structures can be leveraged for AI (e.g., MRM)?

• AI should be added to existing policies; however, unique control expectations must also be developed for AI

• MRM infrastructure may be leveraged, but specific testing procedures often lack necessary sophistication to govern AI

Use of Human and Non-Human Decisioning

• When and how frequent should human review be required?

• Should we invest in AI challenger models?

• Use a spectrum to determine appropriate control structures for range of AI use cases

• Challenger systems and bias detection monitoring are leading practice

Operating Model

• What is the right operating model (centralized vs. decentralized)?

• How do we solve for the skills/knowledge gap in second and third lines of defense?

• Centralized governance model, including a central pool of data scientists; performing always-on monitoring from intake to disposition

• Institute review and gating processes

New Control Structures

• What new control structures/ controls do we need to manage the new risks driven by AI?

• Considerations of surveillance systems• Incorporating AI-driven monitoring solutions to

check on AI models/algorithms

Challenges / Questions Industry Trends / Approaches


AI Governance will need coordinated effort across all lines of defense

Example AI Risks Line of Defense Example Governance & Control activities

• Increased risks of both benign and malicious cyber intrusions / breaches.

• Significant risk of disruption to the company’s operations from unintended machine-made decisions or actions.

• Lack of accountability for outcomes due to inadequate control and responsibility structure

• Faulty financial projections or calculations that undermine the integrity of financial planning and reporting.

• Competitive disadvantage resulting from bias-replication and blind spots due to hidden assumptions and biases in data.

• Heightened impact of threats related to safety, trust, and alignment with the ethics and values of the organization.

• Compliance violations and reputational damage resulting from poorly designed or monitored AI.

• Violating the safety, trust, fairness, or transparency expectations of the organization or its stakeholders

1st LOD

Develop standards for AI development and “kill switch” mechanism

Leverage enterprise sandbox for AI to shape governance and controls

Data Curation for AI (volume, velocity, variety)

2nd LOD

Incorporate bias detection and monitoring

Use control networks to monitor/surveil outputs from AI solutions

Make risk management nimble and dynamic to adopt/deploy AI applications with business units

3rd LOD

Internal Audit using independent neural networks or comparable techniques to test AI solutions

Adopt Governance by Design philosophy through defined boundaries of transparency and accountability

Board Committee (Data)

Periodic review of the performance measures/scorecard and key decisions associated with AI models/algorithms

Review of design principles and guardrails associated with data sets, techniques, and use cases

Questions?

This presentation contains general information only and Deloitte is not, by means of this presentation, rendering accounting, business, financial, investment, legal, tax, or other professional advice or services. This presentation is not a substitute for such professional advice or services, nor should it be used as a basis for any decision or action that may affect your business. Before making any decision or taking any action that may affect your business, you should consult a qualified professional advisor. Deloitte shall not be responsible for any loss sustained by any person who relies on this presentation.

About DeloitteDeloitte refers to one or more of Deloitte Touche Tohmatsu Limited, a UK private company limited by guarantee (“DTTL”), its network of member firms, and their related entities. DTTL and each of its member firms are legally separate and independent entities. DTTL (also referred to as “Deloitte Global”) does not provide services to clients. In the United States, Deloitte refers to one or more of the US member firms of DTTL, their related entities that operate using the “Deloitte” name in the United States and their respective affiliates. Certain services may not be available to attest clients under the rules and regulations of public accounting. Please see www.deloitte.com/about to learn more about our global network of member firms.

Copyright © 2019 Deloitte Development LLC. All rights reserved.

http://www.deloitte.com/about

sf iia – emerging it risks: the road ahead · ontology engineering. neuralnetworks....

Documents