assurance scoring: using machine learning and analytics to reduce risk in the public sector

Assurance Scoring: Using Machine Learning and Analytics to Reduce Risk in the Public Sector

Matt Thomson17/11/2016

2Copyright © Capgemini 2012. All Rights Reserved

Presentation Title | Date

Outline

IntroductionTraditional Fraud DetectionAssurance ScoringMachine LearningBusiness RulesAnomaly DetectionGraph Links



Who am I?

Matt ThomsonSenior Data Scientist at CapgeminiPhD in Astrophysics (http://arxiv.org/abs/1010.3315)Several years experience in fraud detection

CapgeminiBig Data Analytics team~100 Data Scientists, Big Data Engineers and Data AnalystsFocus on Open Source and Big Data technologies to solve client

problemsSponsor the meetup today!



Introduction to the Problem

Public sector constantly working in an environment of reduced resources

Want to provide a better service but with greater efficiency

Therefore very important that limited resources are focussed correctly

Assurance Scoring Use ML and other analytical methods to identify the least risky people or applications so

that investigators resources can be targeted on the most risky



Hypothetical Example – 2016 Olympics tickets

Imagine running the application process for selling tickets to the 2016 Olympics

Avoid selling tickets to touts/resellers Vast majority of people applying for tickets are genuine Fraud detection with big class imbalance problem (<0.1%) Avoid approach of investigating each person applying

Lets say we know from 2012 Olympics which people ended up reselling their tickets – training data

Use ML to identify the 30% (say) least likely to be touts – fast tracked

Investigators focus on the high risk



Traditional Fraud Detection

Identify Historical

Training Data

Feature Engineering

Model Training and Evaluation

Model Execution

Feedback



Assurance Scoring

Focus on low-risk

Allows resources to be better focussed

Not limited to Machine Learning

Built using Python! Pandas, Scikit-learn etc Scala version using Spark MLlib



Assurance Scoring



POLE ‘Analytical’ Data Layer

Disparate data sources - Atomic Layer

Atomic data is Transformed and Loaded into POLE

POLE Layer

EventLocationObjectPerson



POLE ‘Analytical’ Data Layer

POLE contains ALL entities from the Atomic Layer, plus their inter-linkages



Assurance Scoring



Machine learning

Transform Selection Model

Training

Validation

Test

Feature extraction and selection Model Building

Variety of output files: logs, graphics, pickle models, etcTesting: Unit tests, monitoring tests and integration tests

Vector BuildInput Data

Manipulate, ExploreData

Framework: Structure, flexibility, consistency



Machine learning : Feature Engineering

SQL, Python

Transform

Explore

Select

Ask questions, validate

Refine features

• Feature Extraction

• Data exploration

• Feature selection

Historical Data



Machine Learning: Model Building

Training

Validation

Test

Split Datasets

Build Models

Hyper-parameter tuning

Selectedfeatures Models

Training results

Validation results

Testsresults

Compare Models



Low risk? High risk? Depends on classifier’s threshold

• True-positives : applications the model correctly classifies as high risk

• True negatives: applications model correctly classifies as low risk

• False-positives: applications the model scores as high risk but are not

• False-negatives: applications the model scores as low risk but were in fact high risk



Assurance Scoring



Business Rules

Identifying Fraud often been done using deterministic rules

Look for transactions near a threshold or at the end of the day

Primarily data queries on your feature vector

Olympics example – Anyone applying for more than £10,000 tickets



Assurance Scoring



Anomaly Detection

Use the training data to create a baseline of applications by postcode (say)

If a particular postcode has a larger than expected number of applications then those cases pushed into high-risk bucket



Assurance Scoring



Graph Links - Matching

Key part of assurance scoring – bringing data together from disparate sources

Probability of Match: 80%

Attribute Data Source 1 Data Source 2

Name Matt Thomson Matthew Thosmon

Phone Number 07123 456 789 07123 456 798

Favourite Sport Football Cricket



Assurance Scoring



Further Details

Come and find [email protected] / @MattGThomsonAssurance Scoring brochure: http://ow.ly/4nbEUIBlogs:

• Introduction: https://www.capgemini.com/node/1380596• Integrating multiple techniques: http://bit.ly/24BmszV • Machine Learning: http://bit.ly/1QTMGnq• Many more on other topics



We’re Hiring!

Data Sciencehttps://www.uk.capgemini.com/careers/jobs/data-scientist-0

Big Data Engineerhttps://www.uk.capgemini.com/careers/jobs/big-data-engineer

[email protected]

The information contained in this presentation is proprietary.© 2012 Capgemini. All rights reserved.

www.capgemini.com

About CapgeminiWith more than 120,000 people in 40 countries, Capgemini is one of the world's foremost providers of consulting, technology and outsourcing services. The Group reported 2011 global revenues of EUR 9.7 billion.Together with its clients, Capgemini creates and delivers business and technology solutions that fit their needs and drive the results they want. A deeply multicultural organization, Capgemini has developed its own way of working, the Collaborative Business ExperienceTM, and draws on Rightshore ®, its worldwide delivery model.

Rightshore® is a trademark belonging to Capgemini

http://www.facebook.com/Capgemini

http://www.linkedin.com/company/capgemini

http://www.twitter.com/capgemini

http://www.youtube.com/capgemini

http://www.slideshare.net/capgemini

assurance scoring: using machine learning and analytics to reduce risk in the public sector

Data & Analytics