imc 2011, measurement and evaluation of a real world deployment of a challenge-response spam filter

8/3/2019 IMC 2011, Measurement and Evaluation of a Real World Deployment of a Challenge-Response Spam Filter

http://slidepdf.com/reader/full/imc-2011-measurement-and-evaluation-of-a-real-world-deployment-of-a-challenge-response 1/23

Measurement and Evaluation

of a Real World Deployment

of a Challenge-Response Spam Filter

Jelena Isacenkova and Davide Balzarott



12/22/11 Eurecom

Background

― In 2010 spam accounted for 89.1% of the emails

― Traditional email filtering is based on:

→ Content-based classification

→ Sender's properties, IP reputation, domain authenticity

Thorsten Holz, Honeyblog b



12/22/11 Eurecom

Traditional email filtering



12/22/11 Eurecom

Challenge-Response (CR) filtering


http://slidepdf.com/reader/full/imc-2011-measurement-and-evaluation-of-a-real-world-deployment-of-a-challenge-response 5/2312/22/11 Eurecom


The responsibility of the email delivery is shifted from the recipient

to the sender of the message




The responsibility of the email delivery is shifted from the recipient

to the sender of the message



Myths and critics of CR Systems

― Annoying to other users

― Email delivery delay

― Email backscatter

― Mail server blacklisting

― Newsletter non-delivery

― Burdens CR system email users

― Some real emails may go missing

― Deadlocks between two CR serversGarriss et al. “RE: Reliable email ”

Erickson et al, “The effectiveness of Whitelisting: a User-Study



Dataset and system architecture

― 47 mail servers (13 open relay)

― Monitoring period: 6 months

― 19,426 protected users

― Total messages: 90M

― Total challenges: 4,3M

― Solved challenges: 151K



12/22/11 Eurecom

System architecture



12/22/11 Eurecom 1

Our contribution

― Internet viewpoint

―

Users' viewpoint

― Administrator's

viewpoint

– Email backscatter phenomenon

– False negatives in CR system

– Message delivery delay

– CR server blacklisting

– Misdirected challenges



12/22/11 Eurecom 1

Backscattered messages

― Misdirected automated bounce messages(e.g. Non-Delivery Notification)

― The cause is the usage of forged email

addresses (spoofing)

― CR systems work as reflectors:

the unknown incoming messages are challenged with an

automated message aiming to verify the source



12/22/11 Eurecom 1

Backscatter ratio evaluation

How much and what emails CR system pours into the Internet?

― Reflection ratio:

R = RefC/IncE = 19,3%

― Backscattered ratio(spam sent by the CR system)

β <= 8,7%



12/22/11 Eurecom 1

19,3%: good or bad ?

R → 0 %R → 89 %

Harmful Useless



12/22/11 Eurecom 1

False negatives in CR systems

Do CR systems provide 100% spam protection in a real world deployment?

― To answer, we clustered challenged emails by their subject and sender similarity



12/22/11 Eurecom 1

False Negatives

― Requirements for the High-Volume (HV) spam to reach CR

protected user's mailbox are high:

1. Pass reverse DNS, DNSBL, AntiVirus checks

2. Spoof an existing user address(HV spam often uses non-existing email addresses)

3. The innocent user needs to solve a CAPTCHA

― In our experiment a phenomenon of an innocent user solving a

CAPTCHA occurred in 1/10,000 challenges sent



12/22/11 Eurecom 1

Email delivery delay

―

What is the actual email delivery delay for the new communications?

→ 94% of the performed communications are between known contacts

→ 50% of the ever whitelisted senders get whitelisted in the first 30 minutes, mostly by

senders solving CAPTCHA

→ 0.6% of white emails are delayed more than 1 day



12/22/11 Eurecom 1

Server blacklisting

― Spam traps

→ Designed to lure spam

→ BL services use them to update their blacklists

―

If a challenge hits a spamtrap, the server may get blacklisted

― Two experiments:

→ Mail server logs monitoring for 'blacklisted' error messages

→ Tracking the MTA-OUT IPs in 8 known blacklisting services



12/22/11 Eurecom 1

Server blacklisting

―

In our 3 months experiment 75% of the serversnever appeared in any blacklist

― But some servers were blacklisted:

→ During the holidays, stayed unnoticed

→

Several times in a row→ For the most monitoring period

― No correlation found between number of

challenges sent and server blacklisting



12/22/11 Eurecom 1

Conclusions

― CR systems are capable of providing 99.99% spam protection(on top of other deployed anti-spam filters)

― The filter employs a number of side effects:

→ Email traffic increase (~ 0.62%)

→ Useless challenges sent (96%)

→ User email delays:

° 3% delayed for 30 minutes

° 0.6% delayed for more than one day

― CR servers need to be configured and maintained very carefully

→ To lower the probability of blacklisting (and act to get the IP out of the blacklists)

→ To reduce the effect of a backscattered phenomenon



12/22/11 Eurecom 2

Thank you!



12/22/11 Eurecom 2

CAPTCHA and filter performance



12/22/11 Eurecom 2

Burden on users

―

What is a burden on users that maintain theirwhitelists?

→ 51% of the users has not more than 10 chang

in their whitelists during 2 months period

→ Number of very active users is very low, aroun

1-2% of the users

→ Average daily digest size varies greatly betwe

users, thus burdening some of them

→ User responsibility to check their digest



12/22/11 Eurecom 2

Histograms and correlations

imc 2011, measurement and evaluation of a real world deployment of a challenge-response spam filter

Documents