exchange deployment at cern and new ideas for spam fighting michel christaller, emmanuel ormancey,...

Exchange deployment at CERN Exchange deployment at CERN and new ideas for SPAM fightingand new ideas for SPAM fighting

Michel Christaller, Emmanuel Ormancey, Alberto Pace

CERN Mail infrastructureCERN Mail infrastructure

14 Servers 8 “Mailbox” stores, 2 Public Folder Stores, 2 Front-end servers, 2 Spare

IMAP (secure), POP (secure), MAPI and secure HTTP MAPI with Outlook on Windows/Mac MAPI open (in theory) outside CERN using Microsoft ISA Server IMAP and POP work with almost any client HTTP works with any Web browser Collaborative tools available with MAPI and HTTP

Office XP recommended for collaborative features Not possible to switch Outlook 2000 from IMO to CW Allows multi protocol (pop, imap, mapi, webdav) All information stored at server level, no more PST file problems

Office 2003 being evaluated MAPI over HTTP Seamless connected/disconnected/online/offline feature Optimized for slow network connections

Migration overviewMigration overview Nothing changes for the user

Legacy ServerNew Server

user.mailbox.cern.chMail Server

Mail ClientMail User

The server is replaced,The server is replaced,Nothing changes for the clientNothing changes for the clientAdditional interfaces availableAdditional interfaces available

imap

mapihttp

imapspops

webdav

Migration: what is doneMigration: what is done

User are invited to migrate by filling a migration form The password is kept on the new service and synchronized

with the windows password Unresponsive users are forced to migrate and the password is

reset

All folders and mails are copied from the old servers to Exchange

Mail Forwarding configuration is kept if any Mailbox is not functional during at most 10

minutes, while rebuilding configuration files

Migration WorkflowMigration Workflow

Migration Form

Mailbox migrated

Keep password typed in migration form

Nice and Mail password synchronized

Mailbox migrated

Password reset

Nice and Mail password synchronized

“Ask for migration”

mail

Accept / Delay

FormReminder Mail (3) Accept

After n remindersForce migration

No answerClick on link

Click on link

Migration StatusMigration Status 10000 Exchange Users, 14774 Total Only inactive and a few “non cooperative” users remaining Cleanup: More than 700 Mail accounts deleted following user approval

Current statusCurrent status

1 year of production Exchange software stable and scalable No major disaster, only normal hardware failures, solved

in operational delays Usage: 50 % Outlook XP, other 50 % with IMAP, POP and

HTTP access 1’000’000 Incoming mails per week, 30% is Spam

Next step, currently in testNext step, currently in test

Move SMTP Gateways to Exchange Implement automatic anti flood system

Any server, sender or recipient sending or receiving more than 500 mails in 5 minutes will be banned (numbers to define)

Only solution to improve quality of service, and reduce impact of loops on “regular” mails

Migrate Mailing lists system from majordomo to Exchange You will hear about this next year

Spam Fighting at Cern

Evolution

Legacy systemLegacy system

Sendmail checks: Lists of banned IP addresses, domains, subject,

senders or recipients, and words Header “consistency” tests (i.e. message id format)

Mail rejected if identified as Spam Heavy manual work:

Update local banned lists from abuse reports Remove entries when users report false positive

rejections

Current serviceCurrent service

Existing market products were reviewed: Technology too young Results are not accurate Missing a per user basis configuration

While the market consolidates … CERN developed his own Anti-Spam filter

Based on SpamAssassin Less effort than running after immature commercial

technology Now in production for 1 year Easy to modify and update detection techniques

How it worksHow it works

The anti-spam filter calculates the probability for a message to be spam Regular expressions “Intelligent” content parsing Statistical heuristics (Bayesian Filters)

The user sets the threshold at which he wants spam to be rejected Rejected message can be seen by the user

(CERN Spam folder) Per user configuration (!) Allows rejection of foreign languages mail

(Chinese, Korean, Russian, Japanese, Arabic, etc …)

User configurationUser configuration

Filtering levelFiltering level

Language-based rejectionLanguage-based rejection

EfficiencyEfficiency

Roughly 160 000 Incoming mails per day Spam filter detects from 25% to 35% as spam

EfficiencyEfficiency

False positives are very low Except for commercial lists (spam that you want) White lists at user level can be configured to prevent this

Good spam detection Statistics are hard to build Standard mailbox filtering statistics:

30 to 40 Spams filtered per day 1 or 2 Spams still go to the INBOX per week

Could still be improved with some optimization Not enough for some users with “public” email

address Old email address or published email address are more

targeted for Spam

Current evolutionCurrent evolution

Spammer techniques always follow anti-spam techniques

New detection mechanisms work only for a few months

Needs a full time work to have a constantly “up-to-date” filter

Only viable long term solution is to accept only mails from people you know: ICQ (and other messenger systems) already have this feature Accept only messages from people in my contact list Adding someone to the contact list requires validation

New feature (in test)New feature (in test) Good Mails not matching the

user’s white list are quarantined Mail is sent to sender requiring

action to validate himself

Once validated, sender is added to white list, mails are moved back to Inbox

Move to Inbox.Quarantine

Quarantine level

Inbox

Move to Cern Spam

Delete

Spam Filter level

Delete if evident spam level

Mail to sender for validation

What’s next ?What’s next ? Join forces against Spam

Share rules, regular expressions patterns and Bayesian statistics dictionary with other organizations

Central antispam configuration with Live Update like antivirus definitions is the solution. Therefore …

Long term goal: use a commercial product Like for antivirus products, only a full time working team

will provide up-to-date filters

In addition …In addition …

Within Exchange, mail is authenticated Not possible to forge To: or From: fields Delivery and Read receipts are reliable A platform for workflow application

Extend this towards the internet Mail messages digitally signed with guaranteed origin

and dates (See my presentation on PKI this Thursday)

ConclusionConclusion

Users are profiting from the new collaborative services Shared calendar (already used by 1500 accounts) Tasks, workflow Web and webdav interfaces

Spam is a serious issue Towards accepting only authenticated/verified mail There is a future for commercial products in this area