Download - DIFFSYNC A file synchronization program CLASS PROJECT – EEL6883 BY ADAM RADWAN GARGI CHIPALKATTI
DIFFSYNCA file synchronization program
CLASS PROJECT – EEL6883
BY
ADAM RADWAN
GARGI CHIPALKATTI
FILE SYNCHRONIZATION
Process of making sure that two or more locations contain the same up-to-date files
One-way (Mirroring) or two-way file synchronization Useful for mobile workers, multiple computer users;
used for backup purposes Version control tools and File synchronization tools Various existing solutions, none of which were found
by OpenPublic Networks, LLC to be robust enough for production use in a distributed environment
GOALS
Develop a file synchronization program which is capable of working over a diverse range of geographical placement
Must be capable of running securely over common public network backbone, such as the Internet
Capable of two-way synchronization of files for changes to served web pages
Ability for a non-functional server to gracefully degrade while other servers detect this and recover appropriately to keep services running
RSYNC
Open Source software for Unix systems Mirroring takes place with only one transmission in
each direction Used to synchronize local directories or directories
via a remote shell such as RSH or SSH Two checksums: a weak “rolling“ 32-bit checksum
and a strong 128-bit MD4 checksum
UNISON
Runs on Unix-like operating systems (Linux, Mac OS X, and Solaris), and on Windows
Can synchronize files across platforms Detects 'conflicts' where a file has been modified on
both sources, and displays these to the user Uses the rsync algorithm Designed to be robust in the event of a program or
system crash or a communication failure
DIFFSYNC
Uses open source software – rsync, inotify Uses PHP scripting language Built on top of a MySQL 5.1 triple-master replicated
database system running in chrooted jail for security Custom SSH-based VPN for three independent point-
to-point TUN links between the servers Apache-based web server for verification running in
chrooted jail for security phpMyAdmin used for database management
DIFFSYNC FOR THREE SERVERS
Akimbo
Location: Orlando, FLIP: 72.189.195.130CPU: Pentium III 800 MhzStorage: 20GB, 128 MBOS: Gentoo Linux Philomena
Location: Ft. Lauderdale, FLIP: 65.111.166.49CPU: AMD 2000+Storage: 40GB, 256 MBOS: Gentoo Linux
Podice
Location: Fayetteville, NCIP: 216.117.160.235CPU: Pentium 4 2.8 GhzStorage: 80GB, 512 MBOS: Gentoo Linux
CREATION OF DIRECTORY & FILE~/TestArea $ mkdir 1 Array( [0] => Array ( [dir] => ./TestArea/ [event] => CREATE,ISDIR [file] => 1 ) [1] => Array ( [dir] => ./TestArea/ [event] => OPEN,ISDIR [file] => 1 ) [2] => Array ( [dir] => ./TestArea/ [event] => CLOSE_NOWRITE,CLOSE,ISDIR [file] => 1 ))
~/TestArea $ nano ./1/X Array( [0] => Array ( [dir] => ./TestArea/1/ [event] => CREATE [file] => X ) [1] => Array ( [dir] => ./TestArea/1/ [event] => OPEN [file] => X ) [2] => Array ( [dir] => ./TestArea/1/ [event] => MODIFY [file] => X ) [3] => Array ( [dir] => ./TestArea/1/ [event] => CLOSE_WRITE,CLOSE [file] => X ))
MOVE DIRECTORY & FILE
~/TestArea $ mv ./1/X ./1/Y Array( [0] => Array ( [dir] => ./TestArea/1/ [event] => MOVED_FROM [file] => X ) [1] => Array ( [dir] => ./TestArea/1/ [event] => MOVED_TO [file] => Y ) )
~/TestArea $ mv ./1 ./2 Array( [0] => Array ( [dir] => ./TestArea/ [event] => MOVED_FROM [file] => 1 ) [1] => Array ( [dir] => ./TestArea/ [event] => MOVED_TO [file] => 2 ) )
ATTRIB DIRECTORY & FILE
~/TestArea $ chmod 777 ./1 Array( [0] => Array ( [dir] => ./TestArea/ [event] => ATTRIB [file] => 1 ))
~/TestArea $ chmod 777 ./1/X Array( [0] => Array ( [dir] => ./TestArea/1/ [event] => ATTRIB [file] => X ))
DELETE DIRECTORY [4] => Array ( [dir] => ./TestArea/1/ [event] => CLOSE_NOWRITE,CLOSE,ISDIR [file] => ) [5] => Array ( [dir] => ./TestArea/ [event] => DELETE,ISDIR [file] => 1 ) [6] => Array ( [dir] => ./TestArea/1/ [event] => DELETE_SELF [file] => ) [7] => Array ( [dir] => ./TestArea/1/ [event] => IGNORED [file] => ))
~/TestArea $ rm –r ./1
Array( [0] => Array ( [dir] => ./TestArea/ [event] => OPEN,ISDIR [file] => 1 ) [1] => Array ( [dir] => ./TestArea/1/ [event] => OPEN,ISDIR [file] => ) [2] => Array ( [dir] => ./TestArea/1/ [event] => DELETE [file] => X ) [3] => Array ( [dir] => ./TestArea/ [event] => CLOSE_NOWRITE,CLOSE,ISDIR [file] => 1 )
DELETE FILE
~/TestArea $ rm ./1/X Array( [0] => Array ( [dir] => ./TestArea/1/ [event] => DELETE [file] => X ) )
EDIT FILE
~/TestArea $ nano ./1/X Array( [0] => Array ( [dir] => ./TestArea/1/ [event] => OPEN [file] => X ) [1] => Array ( [dir] => ./TestArea/1/ [event] => ACCESS [file] => X ) [2] => Array ( [dir] => ./TestArea/1/ [event] => CLOSE_NOWRITE,CLOSE [file] => X ))
Array( [0] => Array ( [dir] => ./TestArea/1/ [event] => MODIFY [file] => X ) [1] => Array ( [dir] => ./TestArea/1/ [event] => OPEN [file] => X ) [2] => Array ( [dir] => ./TestArea/1/ [event] => CLOSE_WRITE,CLOSE [file] => X ))
COPY FILE~/TestArea $ cp ./1/X ./1/Y Array( [0] => Array ( [dir] => ./TestArea/1/ [event] => OPEN [file] => X ) [1] => Array ( [dir] => ./TestArea/1/ [event] => CREATE [file] => Y ) [2] => Array ( [dir] => ./TestArea/1/ [event] => OPEN [file] => Y ))
Array( [3] => Array ( [dir] => ./TestArea/1/ [event] => ACCESS [file] => X ) [4] => Array ( [dir] => ./TestArea/1/ [event] => MODIFY [file] => Y ) [5] => Array ( [dir] => ./TestArea/1/ [event] => CLOSE_WRITE,CLOSE [file] => Y ) [6] => Array ( [dir] => ./TestArea/1/ [event] => CLOSE_NOWRITE,CLOSE [file] => X ))
DIFFICULTIES ENCOUNTERED
Must have a kernel module installed on the host server for inotify events to be recorded
Problems detecting changes made to newly-created directories within the specified inotify watch point
MySQL replication is sensitive to binary log changes when one node is disconnected
Accurate prediction and parsing of inotify events for complete analysis of changes made to filesystem
Not enough time in the semester to complete integration of all features into program
CONCLUSION
Fundamentals of DiffSync program prove that file synchronization is possible for servers in remote locations
Additional work will be completed to make this program ready for a production environment
Any questions?
THE END
THANK YOU!