![Page 1: TEXAS A&M CAPSTONE PROJECT - Texas A&M | Texas A&M ...online.stat.tamu.edu/dist/analytics/capstone/gov3.pdf · FRAMEWORK Airforce.com Clickstream Data Jan 2016 – Dec 2016 118,489,114](https://reader035.vdocuments.us/reader035/viewer/2022062603/5f0316387e708231d407780c/html5/thumbnails/1.jpg)
TEXAS A&M
CAPSTONE PROJECT
P R E D I C T I V E S E Q U E N T I A L A S S O C I A T I O N R U L E M I N I N G F O R
T R A N S A C T I O N A L C L I C K S T R E A M D A T A
Texas A&M | March 2017
![Page 2: TEXAS A&M CAPSTONE PROJECT - Texas A&M | Texas A&M ...online.stat.tamu.edu/dist/analytics/capstone/gov3.pdf · FRAMEWORK Airforce.com Clickstream Data Jan 2016 – Dec 2016 118,489,114](https://reader035.vdocuments.us/reader035/viewer/2022062603/5f0316387e708231d407780c/html5/thumbnails/2.jpg)
![Page 3: TEXAS A&M CAPSTONE PROJECT - Texas A&M | Texas A&M ...online.stat.tamu.edu/dist/analytics/capstone/gov3.pdf · FRAMEWORK Airforce.com Clickstream Data Jan 2016 – Dec 2016 118,489,114](https://reader035.vdocuments.us/reader035/viewer/2022062603/5f0316387e708231d407780c/html5/thumbnails/3.jpg)
![Page 4: TEXAS A&M CAPSTONE PROJECT - Texas A&M | Texas A&M ...online.stat.tamu.edu/dist/analytics/capstone/gov3.pdf · FRAMEWORK Airforce.com Clickstream Data Jan 2016 – Dec 2016 118,489,114](https://reader035.vdocuments.us/reader035/viewer/2022062603/5f0316387e708231d407780c/html5/thumbnails/4.jpg)
Flow Satellites
![Page 5: TEXAS A&M CAPSTONE PROJECT - Texas A&M | Texas A&M ...online.stat.tamu.edu/dist/analytics/capstone/gov3.pdf · FRAMEWORK Airforce.com Clickstream Data Jan 2016 – Dec 2016 118,489,114](https://reader035.vdocuments.us/reader035/viewer/2022062603/5f0316387e708231d407780c/html5/thumbnails/5.jpg)
![Page 6: TEXAS A&M CAPSTONE PROJECT - Texas A&M | Texas A&M ...online.stat.tamu.edu/dist/analytics/capstone/gov3.pdf · FRAMEWORK Airforce.com Clickstream Data Jan 2016 – Dec 2016 118,489,114](https://reader035.vdocuments.us/reader035/viewer/2022062603/5f0316387e708231d407780c/html5/thumbnails/6.jpg)
MULTI-TOUCH ATTRIBUTION
![Page 7: TEXAS A&M CAPSTONE PROJECT - Texas A&M | Texas A&M ...online.stat.tamu.edu/dist/analytics/capstone/gov3.pdf · FRAMEWORK Airforce.com Clickstream Data Jan 2016 – Dec 2016 118,489,114](https://reader035.vdocuments.us/reader035/viewer/2022062603/5f0316387e708231d407780c/html5/thumbnails/7.jpg)
PRIMARY GOAL:
OPTIMIZE THE SITE BY DYNAMICALLY SERVING CONTENT THAT
MAXIMIZES THE LIKELIHOOD OF A VISITOR TO APPLY
SECONDARY GOAL:
EVOLVE MARKETING MULTI-TOUCH ATTRIBUTION MODELS TO
INCLUDE NON-MEDIA EVENTS
![Page 8: TEXAS A&M CAPSTONE PROJECT - Texas A&M | Texas A&M ...online.stat.tamu.edu/dist/analytics/capstone/gov3.pdf · FRAMEWORK Airforce.com Clickstream Data Jan 2016 – Dec 2016 118,489,114](https://reader035.vdocuments.us/reader035/viewer/2022062603/5f0316387e708231d407780c/html5/thumbnails/8.jpg)
FRAMEWORK
Airforce.com
Clickstream Data
Jan 2016 –
Dec 2016
118,489,114 Obs
50 GB
![Page 9: TEXAS A&M CAPSTONE PROJECT - Texas A&M | Texas A&M ...online.stat.tamu.edu/dist/analytics/capstone/gov3.pdf · FRAMEWORK Airforce.com Clickstream Data Jan 2016 – Dec 2016 118,489,114](https://reader035.vdocuments.us/reader035/viewer/2022062603/5f0316387e708231d407780c/html5/thumbnails/9.jpg)
STRUCTURAL CHALLENGES
• TRANSACTIONAL
• 3,841,025 VISITORS
• 52 VARIABLES
• 216 PAGES
• 8 MARKETING CHANNELS
![Page 10: TEXAS A&M CAPSTONE PROJECT - Texas A&M | Texas A&M ...online.stat.tamu.edu/dist/analytics/capstone/gov3.pdf · FRAMEWORK Airforce.com Clickstream Data Jan 2016 – Dec 2016 118,489,114](https://reader035.vdocuments.us/reader035/viewer/2022062603/5f0316387e708231d407780c/html5/thumbnails/10.jpg)
RARE EVENT CHALLENGES
• 3,687,633 VISITORS
• 153,392 APPLICANTS
• 4% APPLICATION RATE
![Page 11: TEXAS A&M CAPSTONE PROJECT - Texas A&M | Texas A&M ...online.stat.tamu.edu/dist/analytics/capstone/gov3.pdf · FRAMEWORK Airforce.com Clickstream Data Jan 2016 – Dec 2016 118,489,114](https://reader035.vdocuments.us/reader035/viewer/2022062603/5f0316387e708231d407780c/html5/thumbnails/11.jpg)
FRAMEWORK
Airforce.com
Clickstream Data
Jan 2016 –
Dec 2016
118,489,114 Obs
50 GB
![Page 12: TEXAS A&M CAPSTONE PROJECT - Texas A&M | Texas A&M ...online.stat.tamu.edu/dist/analytics/capstone/gov3.pdf · FRAMEWORK Airforce.com Clickstream Data Jan 2016 – Dec 2016 118,489,114](https://reader035.vdocuments.us/reader035/viewer/2022062603/5f0316387e708231d407780c/html5/thumbnails/12.jpg)
FRAMEWORK
Airforce.com
Clickstream Data
Jan 2016 –
Dec 2016
Association
Rules
118,489,114 Obs
50 GB
![Page 13: TEXAS A&M CAPSTONE PROJECT - Texas A&M | Texas A&M ...online.stat.tamu.edu/dist/analytics/capstone/gov3.pdf · FRAMEWORK Airforce.com Clickstream Data Jan 2016 – Dec 2016 118,489,114](https://reader035.vdocuments.us/reader035/viewer/2022062603/5f0316387e708231d407780c/html5/thumbnails/13.jpg)
ASSOCIATION RULES
![Page 14: TEXAS A&M CAPSTONE PROJECT - Texas A&M | Texas A&M ...online.stat.tamu.edu/dist/analytics/capstone/gov3.pdf · FRAMEWORK Airforce.com Clickstream Data Jan 2016 – Dec 2016 118,489,114](https://reader035.vdocuments.us/reader035/viewer/2022062603/5f0316387e708231d407780c/html5/thumbnails/14.jpg)
FRAMEWORK
Airforce.com
Clickstream Data
Jan 2016 –
Dec 2016
Association
Rules
Sequence
Rules
118,489,114 Obs
50 GB
![Page 15: TEXAS A&M CAPSTONE PROJECT - Texas A&M | Texas A&M ...online.stat.tamu.edu/dist/analytics/capstone/gov3.pdf · FRAMEWORK Airforce.com Clickstream Data Jan 2016 – Dec 2016 118,489,114](https://reader035.vdocuments.us/reader035/viewer/2022062603/5f0316387e708231d407780c/html5/thumbnails/15.jpg)
SEQUENCING RULES
![Page 16: TEXAS A&M CAPSTONE PROJECT - Texas A&M | Texas A&M ...online.stat.tamu.edu/dist/analytics/capstone/gov3.pdf · FRAMEWORK Airforce.com Clickstream Data Jan 2016 – Dec 2016 118,489,114](https://reader035.vdocuments.us/reader035/viewer/2022062603/5f0316387e708231d407780c/html5/thumbnails/16.jpg)
RARE EVENT RULES50 RULES WERE IDENTIFIED INDIVIDUALLY FOR EVENTS AND NON-
EVENTS AND SCORED ON THE ORIGINAL DATASET.
![Page 17: TEXAS A&M CAPSTONE PROJECT - Texas A&M | Texas A&M ...online.stat.tamu.edu/dist/analytics/capstone/gov3.pdf · FRAMEWORK Airforce.com Clickstream Data Jan 2016 – Dec 2016 118,489,114](https://reader035.vdocuments.us/reader035/viewer/2022062603/5f0316387e708231d407780c/html5/thumbnails/17.jpg)
FRAMEWORK
Airforce.com
Clickstream Data
Jan 2016 –
Dec 2016
Association
Rules
Sequence
Rules
118,489,114 Obs
50 GB
100 Rules
![Page 18: TEXAS A&M CAPSTONE PROJECT - Texas A&M | Texas A&M ...online.stat.tamu.edu/dist/analytics/capstone/gov3.pdf · FRAMEWORK Airforce.com Clickstream Data Jan 2016 – Dec 2016 118,489,114](https://reader035.vdocuments.us/reader035/viewer/2022062603/5f0316387e708231d407780c/html5/thumbnails/18.jpg)
MECE TREE
• 1 DEPENDENT (APPLICATION FLAG)
• 100 BINARY (RULES)
• 2 NOMINAL (DEVICE, LOCATION)
• 2 INTERVAL (TIME ON SITE, TIME SINCE 1ST VISIT)
• 1 ORDINAL (NUMBER OF VISITS)
![Page 19: TEXAS A&M CAPSTONE PROJECT - Texas A&M | Texas A&M ...online.stat.tamu.edu/dist/analytics/capstone/gov3.pdf · FRAMEWORK Airforce.com Clickstream Data Jan 2016 – Dec 2016 118,489,114](https://reader035.vdocuments.us/reader035/viewer/2022062603/5f0316387e708231d407780c/html5/thumbnails/19.jpg)
LOGISTIC LASSO
SAS DOES NOT NATIVELY
SUPPORT LASSO FOR LOGISTIC
MODELS.
PROC HPGENSELECT FOR A
BINOMIAL DEPENDENT WAS
USED.
![Page 20: TEXAS A&M CAPSTONE PROJECT - Texas A&M | Texas A&M ...online.stat.tamu.edu/dist/analytics/capstone/gov3.pdf · FRAMEWORK Airforce.com Clickstream Data Jan 2016 – Dec 2016 118,489,114](https://reader035.vdocuments.us/reader035/viewer/2022062603/5f0316387e708231d407780c/html5/thumbnails/20.jpg)
DECISION TREE
![Page 21: TEXAS A&M CAPSTONE PROJECT - Texas A&M | Texas A&M ...online.stat.tamu.edu/dist/analytics/capstone/gov3.pdf · FRAMEWORK Airforce.com Clickstream Data Jan 2016 – Dec 2016 118,489,114](https://reader035.vdocuments.us/reader035/viewer/2022062603/5f0316387e708231d407780c/html5/thumbnails/21.jpg)
MODEL DIAGNOSTICS
Confusion Matrix
False
Negative28,929 1%
True
Negative228,6263 95%
False
Positive14,301 1%
True
Positive67,841 3%
30% HOLD-OUT
![Page 22: TEXAS A&M CAPSTONE PROJECT - Texas A&M | Texas A&M ...online.stat.tamu.edu/dist/analytics/capstone/gov3.pdf · FRAMEWORK Airforce.com Clickstream Data Jan 2016 – Dec 2016 118,489,114](https://reader035.vdocuments.us/reader035/viewer/2022062603/5f0316387e708231d407780c/html5/thumbnails/22.jpg)
FRAMEWORK
Airforce.com
Clickstream Data
Jan 2016 –
Dec 2016
Association
Rules
Sequence
Rules
DECISION TREE
118,489,114 Obs
50 GB
100 Rules
![Page 23: TEXAS A&M CAPSTONE PROJECT - Texas A&M | Texas A&M ...online.stat.tamu.edu/dist/analytics/capstone/gov3.pdf · FRAMEWORK Airforce.com Clickstream Data Jan 2016 – Dec 2016 118,489,114](https://reader035.vdocuments.us/reader035/viewer/2022062603/5f0316387e708231d407780c/html5/thumbnails/23.jpg)
LEARNINGS• APPLICANTS SPEND AT LEAST 10M ON THE SITE BEFORE APPLYING
• MORE THAN 50% OF APPLICANTS APPLY ON THE DAY OF THEIR FIRST VISIT
• IMPORTANT PAGE SEQUENCES ARE DEPENDENT ON DEVICE
• GOOD FOR BUSINESS:
• Recruiter Chat
• Browsing Careers
• Searching for Recruiters
• Requirements
• BAD FOR BUSINESS:
• Pay and Benefits
• BMT
• Education
![Page 24: TEXAS A&M CAPSTONE PROJECT - Texas A&M | Texas A&M ...online.stat.tamu.edu/dist/analytics/capstone/gov3.pdf · FRAMEWORK Airforce.com Clickstream Data Jan 2016 – Dec 2016 118,489,114](https://reader035.vdocuments.us/reader035/viewer/2022062603/5f0316387e708231d407780c/html5/thumbnails/24.jpg)
IMPLEMENTATION
• FLOW SATELLITE RULES ARE UPDATED BASED ON MODEL RESULTS
• MARKETING LANDING PAGES DYNAMICALLY DETERMINED BY MODEL RESULTS
• MARKETING RE-ENGAGEMENT IS SHIFTED TOWARDS DAY ONE VISITORS
• ‘APPLY NOW’ BUTTON FEATURED ON EVERY PAGE OF THE SITE
• ADJUSTED CONTENT ON:
• Pay and Benefits
• BMT
• Education
![Page 25: TEXAS A&M CAPSTONE PROJECT - Texas A&M | Texas A&M ...online.stat.tamu.edu/dist/analytics/capstone/gov3.pdf · FRAMEWORK Airforce.com Clickstream Data Jan 2016 – Dec 2016 118,489,114](https://reader035.vdocuments.us/reader035/viewer/2022062603/5f0316387e708231d407780c/html5/thumbnails/25.jpg)
RESULTS
• AIRFORCE.COM SITE CONVERSION RATE HAS DOUBLED
• MARKETING IS NOW TWICE AS EFFICIENT
• ESTIMATED IMPACT EQUAL TO 10% OF MARKETING BUDGET
![Page 26: TEXAS A&M CAPSTONE PROJECT - Texas A&M | Texas A&M ...online.stat.tamu.edu/dist/analytics/capstone/gov3.pdf · FRAMEWORK Airforce.com Clickstream Data Jan 2016 – Dec 2016 118,489,114](https://reader035.vdocuments.us/reader035/viewer/2022062603/5f0316387e708231d407780c/html5/thumbnails/26.jpg)