confused johnny: when automatic encryption leads to confusion and mistakes caleb stepanian, cindy...
TRANSCRIPT
Confused Johnny: When Automatic Encryption Leads to Confusion and MistakesCALEB STEPANIAN, CINDY ROGERS, NILESH PATEL
Background
Problem
Solution
Results
Conclusion
Outline1. Background Information
a) Who is Johnny?b) What is usability?
2. What is Johnny's problem?
3. How can we fix it?
4. Results
5. Conclusions
Background
Problem
Solution
Results
Conclusion
What is usability? Security software is usable if the people using it:
o Know the security tasks they need to performo Are able to figure out how to perform themo Don’t make dangerous errorso Are comfortable enough to continue use
Background
Problem
Solution
Results
Conclusion
Problem StatementJohnny finds it confusing to encrypt his emails.
Background
Problem
Solution
Results
Conclusion
Hypothesis
Johnny doesn’t encrypt because current solutions are not transparent enough.
Background
Problem
Solution
Results
Conclusion
Transparency To be considered transparent: 1. Cannot require too much effort 2. Must solve chicken-and-egg problem for keys 3. Handle encryption automatically, hiding cipher text
Background
Problem
Solution
Results
Conclusion
Experiment
Have Johnny try transparent encryption and opaque encryption methods to determine his preferences.
Background
Problem
Solution
Results
Conclusion
Experimental Methodology1. Find a transparent system that meets criteria2. Find other more opaque solutions3. Run one user study for each other solution
comparing it to the transparent one4. Find System Usability Scale (SUS) score for each5. Draw conclusions
Background
Problem
Solution
Results
Conclusion
System Usability ScaleA set of ten questions that allows one to access the usability of a system on a sliding scale.
Background
Problem
Solution
Results
Conclusion
Experimental SetupTransparent: Pwm (Private Webmail) Browser extension that overlays automatic encryption over web mail
Opaque: MP (Message Protector) Manual encryption with external program
Background
Problem
Solution
Results
Conclusion
Other Methods TestedDepot Base: Voltage(Voltage SecureMail Cloud) Sign up for an account and verify it.
Generic: Encrypt.it (Bookmarklet) Allows you to encrypt the text in any field with a password.
Background
Problem
Solution
Results
Conclusion
Results Because people did not see the encryption happening, 10% of users didn’t encrypt their emails correctly & some users didn’t innately trust the system. Manual encryption (copy pasting while seeing the ciphertext) and clear separation gave users more confidence in the system.
Background
Problem
Solution
Results
Conclusion
Comparison Results: PWM v. MP28 users tried both MP and Pwm
*Correctly identifying who would be able to read encrypted messages
Metric Percent users Pwm Study
Percent users MP study
Successful Decryption 86% 93%
Successful Encryption 83% 97%
Comprehension* 76% 93%
Intuitively decrypt 72% 100%
Preferred System 41% 41%
Background
Problem
Solution
Results
Conclusion
Study ResultsPwm Usability Study PWM v. Voltage Preference
Metric Successful Users out of 25
Setup Pwm 24
Successful Decryption
24
Reply with Encrypted Message
23
Send Encrypted Message Direct
22
•44% users reported Voltage was cumbersome to encrypt and decrypt a message•19% preferred Voltage
Background
Problem
Solution
Results
Conclusion
MP vs. Encipher.it Task 1: Install the given system
Task 2: Open Gmail and send encrypted message, decrypt response
Task 3: Open Facebook and send encrypted message, then decrypt reply
System Task 2 Task 3
MP 89% 96%
Encipher.it 57% / 50% 82% / 61%
Background
Problem
Solution
Results
Conclusion
Conclusions of MP vs. Encipher.itMP had a SUS score of 72.23Encipher.it had a SUS score of 61.25MP qualifies as “acceptable”Encipher.it ranks as “low marginal”
Background
Problem
Solution
Results
Conclusion
Conclusion Encryption needs to be somewhat manual so that users feel secure and know the difference between encryption and plaintext
Background
Problem
Solution
Results
Conclusion
Limitations1. User studies were short term lab studies2. First SUS question was “I think that I would like to use
this system frequently”.3. First MP study assumed secrets were already shared
Second MP study assumed Pwm was installed
PGP (Pretty Good Privacy) public and private keypairs
private key needed to sign and decrypt
public key needed to encrypt and verify signature
A user needs to generate a keypair and share their public key before an encrypted message can be sent to them
Key escrow server Trusted third party that generates and stores key material for users
Has ability to read all messages and masquerade as any user
Background
Problem
Solution
Results
Conclusion
Example SUS Survey Choose from 1 (strongly disagree) to 5 (strongly agree).
1. I think that I would like to use this system frequently
2. I found the system unnecessarily complex
3. I thought the system was easy to use
4. I think that I would need the support of a technical person to be able to use this system
5. I found the various functions in this system were well integrated
6. I thought there was too much inconsistency in this system
7. I found the system very cumbersome to use
8. I would imagine that most people would learn to use this system very quickly
9. I felt very confident using the system
10. I needed to learn a lot of things before I could get going with this system