syncing with-upstream
TRANSCRIPT
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Syncing with UpstreamDarragh Bailey/Tokyo/Oct, 2015
© Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.2
Bio – who am I?•Work for HP since 2005•Mostly focused on source control/CI/release over the 10
years• Involved in HP Public cloud and recently Helion 2.0 (just
released)
•OpenStack projects mostly involved with:– jenkins-job-builder, python-jenkins, git-upstream
• IRC– electrofelix (was pelix) in #openstack-infra on freenode
•GitHub– electrofelix (recent work around vagrant plugins)
© Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.3
What does ‘Syncing with upstream’ mean?•Direct consumer & contributor to OpenStack This talk
– Managing local backports/fixes where/when required– Encouraging upstreaming of fixes and automatic discarding
of backports– Workflow used by development of HP Helion to sync with
upstream
•Consuming distribution packages?– Still mention a few interesting tools that might be useful
© Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Why?Why do we need to align local patches with upstream?And why do we need git-upstream to handle this?
© Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.5
Why would anyone want a tool to align patches?• Syncing daily• Manually it could take 5-30
minutes (ave 15)
• For 25+ projects and growing?
• Worth spending ~90 engineer days per year to have this take <1 minute on average
• Currently just works >90%
https://xkcd.com/1205/
© Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.6
Ideally “Contribute upstream and consume”, sounds great but is that possible?
Why are we carrying local patches?
Assumptions:• All bugs can be debugged and
fixed within a virtual environment• Acceptance upstream is always
quick• There is no risk of new bugs being
added• Issues never impact a production
environment or stop teams from working– The only customers are end users
Get latest
upstream
Find issues(stress test)
Develop Fix
Submit upstream
Release
© Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.7
Probably more likely…
Get latest upstream
Develop Fix
Submit upstream
Release*
Apply/Drop local fixes Apply
locally
Find issues(stress test)
None found (ideal)
Bugs
© Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.8
Types of changes
• Packaging files (including pinning against older python packages)
• Compatibility hacks for older libraries/applications for migration/upgrades
• Backports where can’t roll forward, but need a recent change
• Dumb fixes breaks for other environments or configurations but allows progress for local configurations until proper solution
• Scale testing proving a fix really works; can’t always test upstream at sufficient scale
Anything that needs to be used in an environment besides your personal devstack area.
© Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.9
Where git-upstream fits
Get latest upstream
Develop Fix
Submit upstream
Release
Apply/Drop local fixes Apply
locally
Try to automate apply/drop of
fixes in an submit upstream
compatible manner
Find issues(stress test)
© Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.10
Is there a benefit to upstream if we carry?•Verify change actually solves the problem•Run debug patches on a live system
– Provide data for discussion on fix upstream•Allow time for review and improvements
– Avoid unnecessary pressure for changes to be accepted
© Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.11
So we wrote a tool to support a work-flow
•To clone: https://github.com/openstack/git-upstream•PyPI: https://pypi.python.org/pypi/git-upstream• IRC: #git-upstream
•Basic description:– git-upstream is an open source Python application that can
be used to keep in sync with upstream open source projects, mainly OpenStack.
© Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
How?How do we go about staying in sync?
© Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.13
Two phases to the work-flow
Syncing
• Fetch upstream changes• Push to namespace in
local repo
• master upstream/master
JJB example: contrib/jjb/mirror.yaml
Importing/Re-aligning
• Clone local project repo• Run ‘git upstream
import’….• Profit
Tends to be a little more…
© Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.14
The basics
Git-upstream expects to find upstream refs under ‘upstream/*’, i.e. upstream/stable/kilo, upstream/master
To sync all patches on your master branch against upstream/master:• git upstream importMore complete version• git upstream import --into master upstream/master
Followed by upload to internal gerrit
© Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.15
Quiet
By default git-upstream is quiet:
© Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.16
Verbose
But you can increase the verbosity (spam debug messages to console)
© Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.17
Separate logging
Or request a separate log file for debug:
© Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.18
git-upstream commands available
© Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
How does this work?Under the hood.
© Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.20
Two main approaches
Merge requests• Proven for Linux (kernel)• Well supported by
GitHub/GitLab
• Requires main maintainer in house for each project repo
• Conflicts likely unavoidable unless patch accepted upstream identical
• Doesn’t map well to OpenStack review workflow???
Patches• Proven for OpenStack• Well supported by Gerrit
• Possibly needs patch owner to help resolve conflicts
• Duplicate changes will cause conflicts
• Adding a set of patches to a repo as separate files will not be fun to review
© Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.21
Tools in this area already
•Patch queue• guilt (quilt in git)
• Branches• stgit• topgit
•Packaging• git-dpm
None appeared to play well with Gerrit workflow
https://xkcd.com/927/
© Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.22
So to the requirements we worked out…
•Wanted all developers to use same workflow as upstream (local Gerrit)
•Avoid reviewing commits containing patches (diff of a diff)•Automated discard of patches landed upstream
•A workflow that encourages us to upstream any local changes
© Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.23
Git is already great at managing patches as commitsWhat did we want?
previous import
next desired import
So can we get from old to new state without causing issues?
current
desired
landed upstream so drop next time
© Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.24
But how do we avoid this?
© Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.25
The secret sauce git merge -s ours --no-commit <desired>git read-tree -u --reset <desired>git commit --no-edit
• We use ‘ours’ to merge from desired to current and then replace the contents to make things easier for devs to follow, not because it’s necessary. Could use ‘ours’ and merge from current into desired.
landed upstream so drop next time
previous import
next desired import
current
desired
Outstanding fixes can be merged by Gerrit or cleanly rebased if conflicting with upstream
© Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Usage in HP Helion and status
© Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.27
Example drift
Currently worked out by a custom script, like to have git-upstream report the raw information.
© Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.28
Current state of affairsGit-upstream is still in a alpha/beta phase
Lack of reporting• Difficulty in comparing between upstream and local• Difficulty in understanding what changes are being carriedManual interaction is poor• Conflicts require manual finishing after resolving• No resume + checking of locally carried changesLack of syncing• Notes used for supersede must be explicitly synced using git
commands• No ‘git-upstream status’ to let user know
© Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.29
Git-upstream roadmap
List of main changes to be targeted• https://etherpad.openstack.org/p/git-upstream
Project available on stackforge• https://github.com/openstack/git-upstream• https://review.openstack.org/#/q/status:open+project:openstack/git
-upstream,n,z
Any help greatly appreciated!
© Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
FinTo clone: https://github.com/openstack/git-upstreamPyPI: https://pypi.python.org/pypi/git-upstreamIRC: #git-upstream (look for electrofelix/pelix)