a personal cloud storage app written in python ... · cloud based file sync local file system...
TRANSCRIPT
James Abel – PyBay Aug 2017
Latus®A Personal Cloud Storage App written in Python
www.github.com/jamesabel/latus
James Abel
Aug 12, 2017
@jamesabel
www.abel.co
Latus is a registered trademark of James Abel
James Abel – PyBay Aug 2017
Intro
• James Abel
• HW/SW consultant
• Former Intel Principal Engineer
• Some Python for several years (otherwise C/C++, ASM, etc.)
• Python ‘go to’ programming language for the last few years
James Abel – PyBay Aug 2017
Agenda• What is latus?• Why should I care?• Cloud based File Synchronization
• File system events• Event database• What’s in the cloud• Synchronization
• Encryption• Task Bar Application• Using latus• Next Steps• Summary
latus is on github at www.github.com/jamesabel/latus
James Abel – PyBay Aug 2017
What is latus?
• File storage and synchronization across your computers• AKA cloud storage• Automatically mirrors files across your computer and to the cloud• Make files accessible• Inherent backup
• App that runs in the background (‘client sync’)
• Open Source (GPLv3)
• “Zero Knowledge” Encryption
• Inherent versioning
• Python 3.6
• Utilizes lots of awesome packages!
Latus is a cloud file sync app written in Python
James Abel – PyBay Aug 2017
Why should I care?• You like cloud storage file sync, but you want ..
• Open Source• Python• Zero Knowledge Encryption to the cloud• Control over exactly where files are stored in the cloud (e.g. for regulatory, compliance
and/or business requirements)• Control over versioning history• Control over what takes up space in the cloud in the long term• Custom integrated applications• ‘Independent Web’
• Provide a customized branded (‘white label’) cloud storage/sync offering
• Or just example uses of several awesome packages and capabilities
Open Source File Sync written in Python!
James Abel – PyBay Aug 2017
What is file sync?
Computer A Computer B
The Interneta.txt
“latus” folder “latus” folder
James Abel – PyBay Aug 2017
What is file sync?
Computer A Computer B
The Interneta.txt a.txt
“latus” folder “latus” folder
James Abel – PyBay Aug 2017
Cloud Based File SyncFile System Events Database Cloud Synchronization
Local File System (PC/Mac)
WatchdogEvent Database(AWS DynamoDB + local SQLite cache)
Events
Sync Engine
File Cloud Storage (AWS S3)
Files
Computer A
Cloud
Computer B
Local File System (PC/Mac)
James Abel – PyBay Aug 2017
Cloud Based File Sync
Local File System (PC/Mac)
WatchdogEvent Database(AWS DynamoDB + local SQLite cache)
Events
Sync Engine
File Cloud Storage (AWS S3)
Files(encrypted)
Computer A
Cloud
Computer B
Local File System (PC/Mac)
Step 1: File “a.txt” created
James Abel – PyBay Aug 2017
Cloud Based File Sync
Local File System (PC/Mac)
WatchdogEvent Database(AWS DynamoDB + local SQLite cache)
Events
Sync Engine
File Cloud Storage (AWS S3)
Computer A
Cloud
Computer B
Local File System (PC/Mac)
Step 2.a.:“a.txt Created” Event
Step 2.b.:“a.txt” to S3
Files(encrypted)
James Abel – PyBay Aug 2017
Step 3:“a.txt” mirrored
Cloud Based File Sync
Local File System (PC/Mac)
WatchdogEvent Database(AWS DynamoDB + local SQLite cache)
Events
Sync Engine
File Cloud Storage (AWS S3)
Computer A
Cloud
Computer B
Local File System (PC/Mac)
Files(encrypted)
Events
Files(decrypted)
James Abel – PyBay Aug 2017
mivui – Monotonically Increasing Value• Events are strictly sequenced
• Monotonically Increasing Value in Micro-Seconds (uS) since Epoch as an Integer
• Equivalent to: int(round(time.time()*1E6))
• Integer allows reliable comparison and DB indexing/sort key
• Server provides mivui• Optionally can be locally created (but is then not absolutely guaranteed to be monotonically
increasing)
• http://api.abel.co/miv
{
"mivui": 1502435085769804,
"toc": 0,
"mivf": 1502435085.769804,
"dur": 0.0018029212951660156,
"success": true,
"mivs": "1502435085.769804“
}
James Abel – PyBay Aug 2017
File System Event Database Items
• originator – node (computer) where this event was detected (normally a UUID)
• mivui – monotonically increasing integer in micro-seconds from epoch (from server)
• detection – how was event detected – initial scan, watchdog, periodic poll
• event_type – create, delete, move, modify
• file_hash – SHA512 of file
• file_path – file path in the latus folder
• mtime – mod time of the file
• size – file size (in bytes)
• src_path – source in the case of moves (otherwise unused)
James Abel – PyBay Aug 2017
Cloud and Local event databases
Cloud database is ‘the’ database – local is a cache
Cloud(AWS DynamoDB)
Local Computer Nodes(SQLite cache)a b
AWS
SQLite
AWS DynamoDB
James Abel – PyBay Aug 2017
Synchronization
• Sync is hard
• Compute (algorithm) on the client
• All nodes see the events on all other nodes and use the same algorithm
• ‘winner’ is based on ordered events• Default algorithm is that the most recent event (across all nodes) wins
• Other algorithms are possible
• Files are available in the cloud to provide them locally• Encrypted in the cloud
James Abel – PyBay Aug 2017
Sync example$vi a.txt
$vi b.txt
$mv a.txt c.txt
$rm b.txt
DynamoDB
S3
class DetectionSource(IntEnum):
unknown = 0
initial_scan = 1
watchdog = 2
periodic_poll = 3
class LatusFileSystemEvent(IntEnum):
created = 1
modified = 2
deleted = 3
moved = 4
Winner!
James Abel – PyBay Aug 2017
Encryption• Keep unencrypted files off the internet/cloud• Uses the most excellent Python cryptography package
• Key exists on client nodes (computers)• Local computer is assumed secure
• In the cloud file contents are stored as AWS S3 objects• S3 object names are SHA512 hashes of the files contents ‘salted’ with encryption key• Thwarts dictionary lookup attacks
• For security, the encryption key should not be transferred over the internet• Keep the key ‘out of band’• Once generated, a key can be exported to a file (simple JSON format)• USB stick works well (put it in a safe place!)
• On first run:• Makes a new encryption key if no one exists
or• Asks the user for an existing key to be imported
James Abel – PyBay Aug 2017
boto3• Amazon’s Python library to access Amazon Web Services (AWS)• AWS credentials stored on the local file system or passed in explicitly to
boto3 APIs• ~/.aws/credentials
[default]
aws_access_key_id = YOUR_ACCESS_KEY
aws_secret_access_key = YOUR_SECRET_KEY
region=us-west-1 # means Northern California
• Access AWS
import boto3
s3 = boto3.resource('s3’)
# Upload a new file
data = open('test.jpg', 'rb')
s3.Bucket('my-bucket').put_object(Key='test.jpg', Body=data)
James Abel – PyBay Aug 2017
AWS S3 (with encryption)
‘a’
This object’s contents contains the bytes corresponding to a file with contents ‘b’
latusstorage S3 bucket
James Abel – Pyninsula Mar 2017
latus is a Task Bar App …
James Abel – Pyninsula Mar 2017
What is a Task Bar Application?
• A taskbar is an element of a graphical user interface which has various purposes. It typically shows which programs or applications are running on the device, as well as provide links or shortcuts to other programs or places, such as a start menu, notification area, and clock.
https://en.wikipedia.org/wiki/Taskbar
• AKA System Tray
James Abel – Pyninsula Mar 2017
PyQt
• Qt (/kjuːt/ "cute"[7][8][9]) is a cross-platform application framework that is used for developing application software that can be run on various software and hardware platforms with little or no change in the underlying codebase, while still being a native application with native capabilities and speed.• https://en.wikipedia.org/wiki/Qt_(software)
• What is PyQt?• PyQt is a set of Python v2 and v3 bindings for The Qt Company's Qt application
framework and runs on all platforms supported by Qt including Windows, OS X, Linux, iOS and Android.
• https://riverbankcomputing.com/software/pyqt/intro
• Currently using PyQt5
James Abel – Pyninsula Mar 2017
Hello World Task Bar App
class HelloWorldSystemTray(QSystemTrayIcon):
def __init__(self):
import icons
icon = QIcon(QPixmap(':icon.png'))
super().__init__(icon)
menu = QMenu()
menu.addAction("About").triggered.connect(self.about)
menu.addAction("Exit").triggered.connect(self.exit)
self.setContextMenu(menu)
def about(self):
about_box = QDialog()
layout = QGridLayout(about_box)
layout.addWidget(QLabel('hello world'))
about_box.setLayout(layout)
about_box.show()
about_box.exec_()
def exit(self):
QApplication.exit()
app = QApplication(sys.argv)
app.setQuitOnLastWindowClosed(False) # so popup dialogs don't close the system tray icon
system_tray = HelloWorldSystemTray()
system_tray.show()
app.exec_()
James Abel – PyBay Aug 2017
• Latus Task Bar App
class LatusSystemTrayIcon(QSystemTrayIcon):
def __init__(self, app, latus_appdata_folder):
self.app = app
self.latus_appdata_folder = latus_appdata_folder
menu = QMenu(parent)
menu.addAction("Open Latus Folder").triggered.connect(self.open_latus_folder)
menu.addAction("Preferences").triggered.connect(self.preferences)
menu.addAction("Export Latus Key").triggered.connect(self.export_latus_key)
menu.addAction("About").triggered.connect(self.about)
menu.addAction("Nodes").triggered.connect(self.nodes)
menu.addAction("Exit").triggered.connect(self.exit)
self.setContextMenu(menu)
def start_latus(self):
self.sync = latus.aws.sync_aws.Sync(self.latus_appdata_folder)
def show(self):
QSystemTrayIcon.show(self)
def exit(self):
self.sync.request_exit()
QApplication.exit()
app = QApplication(sys.argv)
system_tray = LatusSystemTrayIcon(app, latus_appdata_folder)
system_tray.start_latus()
system_tray.show()
app.exec_()
James Abel – PyBay Aug 2017
Preferences
• Local preferences kept in preferences.db• SQLite DB
• Latus crypto key
• Latus folder path
• AWS configuration (credentials, region)
• Accessible via icon pulldown
James Abel – PyBay Aug 2017
Using latus• Can be cloned and executed from source
• www.github.com/jamesabel/latus
• latus as a native app• latus uses many packages – it can be a challenge to freeze/install• cryptography and PyQt can be problematic to freeze
• Embedded Python interpreter tends to work best
• Freezers/Installers• Windows: pynsist, osnap using the ‘embedded’ Python interpreter
• MacOS: osnap currently
latus can be run for source or as a native app for Windows and MacOS
James Abel – PyBay Aug 2017
Hacking latus
• Fork from• www.github.com/jamesabel/latus
• Run regression tests• Uses py.test, localstack
• Has create, delete, move, etc.
• But can use more tests. ☺
• Pull requests
• File issues
James Abel – PyBay Aug 2017
Help – get a cool T Shirt!
* While supplies last** If you help ☺
James Abel – PyBay Aug 2017
Some Next Steps …
•Add AWS pubsub to get file system events instantly (AWS SNS/SQS)
•Cloud storage efficiency enhancements• Select folders for ‘offline only’
•Node management• Filter out special files (e.g. “.*”)•Code review ☺
Try out latus! Its on github at www.github.com/jamesabel/latus
James Abel – PyBay Aug 2017
Summary and Thank You
• latus is an Open Source Cloud Storage app in Python 3.6
• Hack on latus at SF Python Project Night, chat at Pyninsula or Bay Piggies
• Thank you• Glyph• Thomas Kluyver
Try out latus! Its on github at www.github.com/jamesabel/latus