hw09 large scale transaction analysis
TRANSCRIPT
Hadoop World: NYC 1
Hadoop@Visa
Joe Cunningham Technology Strategy and InnovationOctober 2, 2009
Hadoop World: NYC 2
Agenda
1 Visa
2 Value-Added Information Products
3 Hadoop@Visa – research results
VisaNet – an overview
Hadoop World: NYC 3
Keeping the Brand Promise
ZeroLiability
goSecurity
Flexibility
Hadoop World: NYC 4
ZeroLiability
goSecurity
Flexibility
Keeping the Brand Promise
3
GlobalAcceptance
Reliability
GuaranteedPayment
Hadoop World: NYC 5
VisaNet Today
fast secure flexible reliable scalablefast secure flexible reliable scalable
• VisaNet is the world’s largest, most advanced payments network.
28 millionacceptancelocations
130 millionauthorizationsper day
1,600endpoints
Processes transactions in 1 second
16,300financial institutions
Less than 2 secondsunavailability per year
Processes in 175 currencies
1.4 millionATMs
Hadoop World: NYC 6
1 VisaNet – an overview
2
3
Value-Added Information Products
Hadoop@Visa – research results
Agenda
Hadoop World: NYC 7
Visa Processing Architecture
Security/Access Services
Message File Web
VisaNet Services Integration
Common Data Infrastructure
Authorization Clearing & Settlement
InformationDisputes Risk
Hadoop World: NYC 8
Value-Added Information Products
Information Services
Client • Portfolio Analysis• Visa Incentive Network
Accountholder• Transaction Alerts• Account Updater• Tailored Rewards
Risk Management Services
Authentication • Advanced Auth• Verified by Visa• Address Verification
Account Monitoring• Fraud Detection• Compromised Accounts• Consumer Alerts
Encryption• Dynamic Key Exchange• Dynamic Cryptograms
Hadoop World: NYC 9
1 VisaNet – an overview
2 Value-Added Information Products
3 Hadoop@Visa – research results
Agenda
Hadoop World: NYC 10
Hadoop SystemsVM System
Research Lab Setup
Relational Database
Custom Analytic Stacks
Encryption Processing
Management Stack
Hadoop #2~300Tb / 28 nodes
Hadoop #1~40Tb / 42 node
Hadoop World: NYC 11
Risk Product Use CaseCreate critical data model elements, such as keys and transaction statistics, which feed our real-time risk-scoring systems
Input: Transactions
Merchant Category
Country / Zip
Output: Key & Statistics
MCCZIP KeyStatistics related to account, transaction type, approval, fraud, IP address, tx. counts
HadoopCluster
Research Sample (Synthetic Transactions)
• 500 million distinct accounts
• 100 million transactions per day
• 200 bytes per transaction
• 2 years: ~ 73 billion transactions, ~ 36 TB
• One Month 13 minutes!
Hadoop World: NYC 12
Financial Enterprise Fit
Some key questions we are researching…
• What will the Hadoop Solution Stack(s) look like?File System
Transaction Sample System
Relational Back-End
Analytics Processing
• Internal vs. External cloudHow do I get data into a cloud in a secure way?
• How does HSM and security integration work in Hadoop?What are the missing pieces?
Hadoop World: NYC 13
Why Hadoop@Visa?
fast secure flexible reliable scalablefast secure flexible reliable scalable
• For our information products, we see promise in combining Visa’s data analytics capability with the power of Hadoop
• Analyze volumes of data with response times that are not possible today…
… and apply analytic models to individual client, not just client segment
• But, need to answer questions relating to stack configuration and support…
… and, eventually must fit with our financial enterprise paradigm:
Hadoop World: NYC 14
Keeping the Brand Promise
ZeroLiability
goSecurity
Flexibility
Hadoop World: NYC 15
ZeroLiability
goSecurity
Flexibility
Keeping the Brand Promise
3
GlobalAcceptance
Reliability
GuaranteedPayment