transferring changes between perforce servers
DESCRIPTION
Transferring changes between two unrelated Perforce Servers can be a challenge. This talk explains how this can be done using a Python tool called PerforceTransfer. The concepts of this tool, its internal workings as well as its applications and limitations are explained.TRANSCRIPT
![Page 1: Transferring Changes Between Perforce Servers](https://reader033.vdocuments.us/reader033/viewer/2022061118/5469b9d9af7959653c8b4cdc/html5/thumbnails/1.jpg)
#
Sven Erik Technical Marketing
Transferring Changes Between Perforce Servers
Robert CowhamProfessional Services
![Page 2: Transferring Changes Between Perforce Servers](https://reader033.vdocuments.us/reader033/viewer/2022061118/5469b9d9af7959653c8b4cdc/html5/thumbnails/2.jpg)
#
• Background• Algorithm• Configuration
• Adapting for continuous transfer• Customer Case Study
Agenda
![Page 3: Transferring Changes Between Perforce Servers](https://reader033.vdocuments.us/reader033/viewer/2022061118/5469b9d9af7959653c8b4cdc/html5/thumbnails/3.jpg)
#
Sven Erik KnopTechnical MarketingPerforce Software
Author of P4Python.
Started as Lead Consultant at Perforce. Regular presenter at Perforce and other conferences.
Robert CowhamProfessional ServicesPerforce Software
API experience includes P4OFC (P4COM) and P4Python. Author of 'Learning Perforce SCM' by PACKT Publishing, Sep 2013.
![Page 4: Transferring Changes Between Perforce Servers](https://reader033.vdocuments.us/reader033/viewer/2022061118/5469b9d9af7959653c8b4cdc/html5/thumbnails/4.jpg)
#
Background
![Page 5: Transferring Changes Between Perforce Servers](https://reader033.vdocuments.us/reader033/viewer/2022061118/5469b9d9af7959653c8b4cdc/html5/thumbnails/5.jpg)
#
• One project in two separate Perforce Servers?– e.g. Public Depot and Master Depot
Background
Source P4D Target P4D
?
![Page 6: Transferring Changes Between Perforce Servers](https://reader033.vdocuments.us/reader033/viewer/2022061118/5469b9d9af7959653c8b4cdc/html5/thumbnails/6.jpg)
#
• Git? SHAs do not match.• Sandbox? Can only talk to one server.• Replication? Not practical.
• Reconcile? Changes? Integrations?
• Replay changes one by one ... ?
Solutions?
![Page 7: Transferring Changes Between Perforce Servers](https://reader033.vdocuments.us/reader033/viewer/2022061118/5469b9d9af7959653c8b4cdc/html5/thumbnails/7.jpg)
#
Idea: two overlapping workspaces• Transfer workspace sharing the same root
transfer
Source Target
![Page 8: Transferring Changes Between Perforce Servers](https://reader033.vdocuments.us/reader033/viewer/2022061118/5469b9d9af7959653c8b4cdc/html5/thumbnails/8.jpg)
#
• One workspace on each server with shared root• Client views have to match
– Depot views don’t have to match– Filter projects you want to transfer
• Need to set options to allwrite
Workspace details
Client: src-transRoot: /Users/sknop/perforce/transferOptions: allwriteView:
//source/project1/... //src-trans/...
Client: target-transRoot: /Users/sknop/perforce/transferOptions: allwriteView:
//target/external/... //target-trans/...
![Page 9: Transferring Changes Between Perforce Servers](https://reader033.vdocuments.us/reader033/viewer/2022061118/5469b9d9af7959653c8b4cdc/html5/thumbnails/9.jpg)
#
• Written in Python (2.6+ and 3.3+) using P4Python
• Makes use of P4::Map and P4::Resolver– Use Map instead of “p4 where”– Resolver allows scripting of resolve results
Code details
![Page 10: Transferring Changes Between Perforce Servers](https://reader033.vdocuments.us/reader033/viewer/2022061118/5469b9d9af7959653c8b4cdc/html5/thumbnails/10.jpg)
#
Algorithm (roughly)counter = getCounter()
changes = p4 –p source changes ...@counter,#head
for change in changes:
if add: p4 –p target add
if delete: p4 –p target delete -v
if edit: p4 –p target sync –k ; p4 –p target edit
integrate?
p4 –p source sync
if move: p4 –p target sync –f old; p4 –p target edit old; p4 –p target move old new
filterChangeByClientView(change)
![Page 11: Transferring Changes Between Perforce Servers](https://reader033.vdocuments.us/reader033/viewer/2022061118/5469b9d9af7959653c8b4cdc/html5/thumbnails/11.jpg)
#
• Integrations within workspace view are transferred– Resolve records match source records (details later)
• Integrations from outside the view are downgraded– integrate add/edit/delete
• Use P4::Resolver to preserve resolve outcome
Transferring integrations
![Page 12: Transferring Changes Between Perforce Servers](https://reader033.vdocuments.us/reader033/viewer/2022061118/5469b9d9af7959653c8b4cdc/html5/thumbnails/12.jpg)
#
Transferring integrations (cont)
Client: src-transView:
//depot/inside/... //src-trans/...
Client: target-transView:
//target/... //target-trans/...
![Page 13: Transferring Changes Between Perforce Servers](https://reader033.vdocuments.us/reader033/viewer/2022061118/5469b9d9af7959653c8b4cdc/html5/thumbnails/13.jpg)
#
• usage: P4Transfer.py [-h] [-n] [-c CONFIG] [-m MAXIMUM] [-k] [-p] [-r] [-s]
• [--sample_config] [-i]
• P4Transfer
• optional arguments:
• -h, --help show this help message and exit
• -n, --preview Preview only, no transfer
• -c CONFIG, --config CONFIG Default is transfer.cfg
• -m MAXIMUM, --maximum MAXIMUM Maximum number of changes to transfer
• -k, --nokeywords Do not expand keywords and remove +k from filetype
• -p, --preflight Run a sanity check first to ensure target is empty
• -r, --repeat Repeat transfer in a loop - for continuous transfer
• -s, --stoponerror Stop on any error even if --repeat has been specified
• --sample_config Print an example config file and exit
• -i, --ignore Treat integrations as adds and edits
Usage
![Page 14: Transferring Changes Between Perforce Servers](https://reader033.vdocuments.us/reader033/viewer/2022061118/5469b9d9af7959653c8b4cdc/html5/thumbnails/14.jpg)
#
• [general]
• counter_name = p4transfer_counter
• [source]
• p4port = source.perforce.com:1666
• p4user = sknop
• p4client = source_transfer
• [target]
• p4port = target.perforce.com:1777
• p4user = sknop
• p4client = target_transfer
Configuration file
P4Transfer.py --sample_config
![Page 15: Transferring Changes Between Perforce Servers](https://reader033.vdocuments.us/reader033/viewer/2022061118/5469b9d9af7959653c8b4cdc/html5/thumbnails/15.jpg)
#
• How do we test P4Transfer with two servers?• Use the “rsh trick”
– Spawn a local p4d but without consuming portP4PORT=rsh:/bin/p4d -r /path/to/target.root –i
• Tests:– Inject changes into source– Transfer– Verify result in target
Test Harness
![Page 16: Transferring Changes Between Perforce Servers](https://reader033.vdocuments.us/reader033/viewer/2022061118/5469b9d9af7959653c8b4cdc/html5/thumbnails/16.jpg)
#
Continuous Transfer
![Page 17: Transferring Changes Between Perforce Servers](https://reader033.vdocuments.us/reader033/viewer/2022061118/5469b9d9af7959653c8b4cdc/html5/thumbnails/17.jpg)
#
• Experience based on source code migrations– which can take days
• We want to keep an eye on things– But have a life at the same time
• Reliability/robustness in the face of (communication) errors
• Status / Error reporting to support the above
Background – Long Processes
![Page 18: Transferring Changes Between Perforce Servers](https://reader033.vdocuments.us/reader033/viewer/2022061118/5469b9d9af7959653c8b4cdc/html5/thumbnails/18.jpg)
#
• Customer working with multiple third parties• Provide some level of backup/DR• Limited access to remote repositories
– Replicas considered but rejected– Basically only read-only access
• Data size/connectivity issues
Case Study (Background)
![Page 19: Transferring Changes Between Perforce Servers](https://reader033.vdocuments.us/reader033/viewer/2022061118/5469b9d9af7959653c8b4cdc/html5/thumbnails/19.jpg)
#
• Volume / connectivity bandwidth– 280 Gb of data
• Seeding via offline backup would have been ideal
– 1Gb per hour transfer rate– Single (early) changelist ~120Gb!
• We wanted– Handle/report VPN disconnects– Peace of mind…!
Data & Connectivity Requirements
![Page 20: Transferring Changes Between Perforce Servers](https://reader033.vdocuments.us/reader033/viewer/2022061118/5469b9d9af7959653c8b4cdc/html5/thumbnails/20.jpg)
#
• Use standard Python logging module – Subclass for custom behavior– Beware of Python 2.x/3.x compatibility issues
• Custom logging enhancements– Use circular buffer to remember last 50 lines– Automatically notify users via email, including those
lines
Status/Logging Changes
![Page 21: Transferring Changes Between Perforce Servers](https://reader033.vdocuments.us/reader033/viewer/2022061118/5469b9d9af7959653c8b4cdc/html5/thumbnails/21.jpg)
#
Transferring 16 changes
Syncing 16 changes, files 5375, size 11.2 GB
Processing : 87370 "Removed unused physx files from filelist."
source = 87370 : target = 460
Processing : 87377 "Copying //depot/stable to live (//depot/live) various hotfixes"
source = 87377 : target = 461
Synced 8/16 changes, files 676/5375 (12.6 %), size 482.8 MB/11.2 GB (4.3 %)
Synced 8/16 changes, files 940/5375 (17.5 %), size 1.2 GB/11.2 GB (10.7 %)
Sync Status Emails
![Page 22: Transferring Changes Between Perforce Servers](https://reader033.vdocuments.us/reader033/viewer/2022061118/5469b9d9af7959653c8b4cdc/html5/thumbnails/22.jpg)
#
• P4API / P4Python has callback options# Report status per change
self.progress.ReportChangeSync()
# Create a custom callback object (next slide)
mycallback = SyncOutput(self.progress)
# Pass it to the P4 sync command
self.p4.run('sync', '...@{},{}'.format(change, change), handler=mycallback )
Sync Status Details
![Page 23: Transferring Changes Between Perforce Servers](https://reader033.vdocuments.us/reader033/viewer/2022061118/5469b9d9af7959653c8b4cdc/html5/thumbnails/23.jpg)
#
class SyncOutput(P4.OutputHandler):
def __init__(self, progress):
P4.OutputHandler.__init__(self)
self.progress = progress # Save reporting object
def outputStat(self, stat): # Function called by P4API
if 'fileSize' in stat:
# Report how much data synced for current file
self.progress.ReportFileSync(int(stat['fileSize']))
return P4.OutputHandler.HANDLED
Sync Callback Handler
![Page 24: Transferring Changes Between Perforce Servers](https://reader033.vdocuments.us/reader033/viewer/2022061118/5469b9d9af7959653c8b4cdc/html5/thumbnails/24.jpg)
#
• What happens when the Dragon King is:– Der Drachenkönig.jpg
• Windows servers (not in Unicode mode) handle (some) Unicode filenames happily– no P4CHARSET required, but…
• On Mac/Unix - UTF8 does the job• On Windows:
– Python 2.7 fine / Python 3.x requires work…
Unicode
![Page 25: Transferring Changes Between Perforce Servers](https://reader033.vdocuments.us/reader033/viewer/2022061118/5469b9d9af7959653c8b4cdc/html5/thumbnails/25.jpg)
#
• On Windows run as a service– Install: srvcinst.exe– Run service: srvany.exe (or other equivalents)
Running Continuously
![Page 26: Transferring Changes Between Perforce Servers](https://reader033.vdocuments.us/reader033/viewer/2022061118/5469b9d9af7959653c8b4cdc/html5/thumbnails/26.jpg)
#
• Basic algorithm unchanged– Fixed a couple of bugs– TDD => Test Harness
• Added repeat/polling options• Enhanced Robustness
– Logging / Status reporting– Error handling
Customization Summary
![Page 28: Transferring Changes Between Perforce Servers](https://reader033.vdocuments.us/reader033/viewer/2022061118/5469b9d9af7959653c8b4cdc/html5/thumbnails/28.jpg)
#
RESOURCES
Public Depot:swarm.workshop.perforce.com/projects/perforce-software-p4transfer/