Subversion Best Practices: Merging and Merge Tracking

CollabNet Webinar March 18, 2010

Agenda
Introductions The Essentials Bumps in the Road From 1.5.0 to 1.6.9 Merge Tracking in 1.7.0 and Beyond Best Practices Questions & Answers

Prerequisites
You should be a user of Subversion that routinely needs to merge. You should understand what merging is in general and how it is utilized in Subversion. You should understand what the merge tracking feature is that was added in Subversion 1.5.0. You should be ready to dive deeper technically into merge tracking.

The Essentials: svn:mergeinfo

Merge tracking is basically the process by which Subversion decides what revisions have been merged from a given source to a given target and/or what revisions are eligible to be merged. The key user-visible part of merge tracking is the versioned svn:merge-info property: e.g. /subversion/trunk:976,980-981 A record of merges made into a given directory or file Inheritable by default Mergeable Editable with 'svn propdel | propset | propedit' May be empty Inclusive ranges: '980-981' is '-r979:981' not '-r980:981'

The Essentials: Mergeinfo Inheritance

Explicit Mergeinfo: A path with the svn:mergeinfo property set on it. Inherited Mergeinfo: If a path doesn't have explicit mergeinfo, then it inherits the mergeinfo from its nearest parent with explicit mergeinfo. A path with explicit mergeinfo never inherits mergeinfo it's one or the other (or neither). Non-Inheritable Mergeinfo: Ranges marked with a '*' apply only to the path which the svn:mergeinfo property is set, e.g.: '/subversion/trunk:976,980-981,1005*' Mixed-revision working copies can thwart inheritance.

The Essentials: Direct vs. Indirect

Like any other versioned property, svn:mergeinfo can be copied. So just because a path has mergeinfo for a given merge source and revision doesnt necessarily mean a merge was literally done from that source to that path.

The Essentials: Implicit Mergeinfo

Regardless of whether a path has explicit or inherited mergeinfo, it always has implicit mergeinfo (a.k.a. natural history or just history). Say we have a branch created from trunk like this:

The implicit mergeinfo on /branches/B1 includes both its common history with trunk and its own history. If we think of this as merge info it would be:
'/trunk:420-713' '/branches/B1:714-1011'

While this mergeinfo isnt actually present in B1s explicit (or inherited) mergeinfo, Subversion acts as if it is for the purposes of determining what has been merged to the B1 branch.

The Essentials: When It is Not Used

There are times when mergeinfo is neither considered when deciding what has been merged, nor recorded to describe a merge:
--ignore-ancestry merges: Just applies the raw diff, i.e. that generated by svn diff. Foreign repository merges (i.e. merging difference from repository 'A' to a working copy for repository 'B') The diff you are attempting to merge is between two unrelated URLs. Using a 1.5.0+ client against a 1.4.x or older server. When reverse merging part of a target's natural history, mergeinfo is considered when deciding what to merge, but is not recorded:
>svn merge -c -40528 ^/trunk trunk --- Reverse-merging r40528 into 'trunk': U trunk\tools\buildbot\master\master.cfg >svn st M trunk\tools\buildbot\master\master.cfg ^

The Essentials: Subtree Mergeinfo

Subtree Mergeinfo is explicit mergeinfo found under the root of a branch. The subtree can be a directory or a file. A subtree merge is a merge targeting some path under the root of a branch. It is one way to create subtree mergeinfo. Subtree merges targeting a file are often referred to as file merges.

The Essentials: Missing Subtrees

In a perfect world a working copy merge target represents a complete repository subtree at a single revision, but merge tracking allows you to merge into a target which has missing subtrees. A subtree can be missing because it is:
Switched The working copy is shallow (i.e. checked out at depth = empty | immediates | files) Not present because you are not authorized to read it. There are also cases where a subtree is present in the target but is effectively treated as missing:
Merge is at an operational depth other than infinity. You are authorized to the subtree on the target, but not the corresponding source.

Subversion handles missing subtrees using non-inheritable mergeinfo and subtree mergeinfo.

Merge Tracking: Bumps in the Road

The original merge tracking design was partially inspired by svnmerge.py. One of the key differences is merge tracking's support of merges that don't target or apply to an entire branch (Hello subtree mergeinfo!). Subtree mergeinfo is usually at the root of most merge tracking bugs and any confusion about the feature. The biggest complaints:
WC-to-WC copies/moves created mergeinfo on the destination when none existed on the source. All subtree mergeinfo, regardless of whether the subtree was affected by the merge, is updated when recording mergeinfo to describe a merge. My merge changed one file, but I see hundreds of mergeinfo changes! Poor performance with large numbers of subtrees with mergeinfo. $%#@! this is taking a long time! Missing subtrees can spawn subtree mergeinfo. Where did that come from?

Merge Tracking: Bumps in the Road, Continued

Mea Culpa: We didn't foresee the amount of subtree mergeinfo some users would generate (1000's in some cases) nor the confusion, performance, and bug issues related to this. The bad news: Addressing the major problems wasn't simple. Performance and correctness concerns made some otherwise straightforward improvements a bit tricky. The good news: There have been many performance improvements, bug fixes, and enhancements starting with 1.5.1 and continuing right up to the current release (1.6.9). The better news: Some of the worst offenders are gone in 1.7.

Merge Tracking: From 1.5.0 to 1.6.9

or Why You REALLY Want a 1.6.9 Client
While hardly a comprehensive list, the following are some of the major improvements to merge tracking since its initial release: Note that all of these fixes are for the Subversion client. 1.5.1 1.5.4 Several fixes that prevent outright merge failures in some common use cases. Improvements to merge performance. 1.5.5 Prevents shallow merges from creating invalid mergeinfo that will break subsequent merges and log -g. Allow merge sources with gaps in their natural history. Don't create mergeinfo during WC-to-WC copies/moves when the source has no explicit mergeinfo. 1.5.6 Prevent colons in path names from breaking merges and log -g. Several fixes that address incorrectly formatted mergeinfo introduced by svnmergemigrate-history.py

Merge Tracking: From 1.5.0 to 1.6.9, Continued

1.6.1 Prevent foreign repository merges from setting mergeinfo. 1.6.2 Avoid abort on merge with servers prior to 1.6.0. Avoid segfault in some cases when merge target has subtree mergeinfo. Massive performance improvement when merge target has many subtrees with mergeinfo. 1.6.3 Fix incorrect 'svn mergeinfo show-revs eligible' results when dealing with copies. Fix segfault with noop file merges. 1.6.4 Another massive performance improvement when merge target has many subtrees with mergeinfo. 1.6.9 Prevent 'svnadmin load --parent-dir PATH' from creating mergeinfo with relative merge source paths. Also has fixes to tolerate this type of mergeinfo if you already have it (as we do in the Subversion repository).

Merge Tracking from 1.5.0 to 1.6.9 Summary

Did I mention you really want a 1.6.9 client?

Merge Tracking: Whats New in 1.7

1.7 (Coming to a site near you sometime in 2010) Only subtrees with explicit mergeinfo which are affected by the merge have their mergeinfo updated to describe the merge. --record-only merges apply svn:mergeinfo diffs in addition to recording mergeinfo as before. Allow 'svn mergeinfo' subcommand to account for non-inheritable mergeinfo and consider subtree mergeinfo. Produce notifications and headers when a merge records mergeinfo describing a merge or elides mergeinfo.

Merge Tracking: A Quick 1.7 Example

Given a branch working copy with some subtree mergeinfo:
1.7-dev-client>svn pg svn:mergeinfo -vR Properties on '.': svn:mergeinfo /branches/feature-branch-1:40527-40534 /branches/feature-branch-4:40562-40574 Properties on

'subversion\bindings\javahl\src\org\tigris\subversion\javahl\SVNClientInterface.java'

svn:mergeinfo /branches/feature-branch1/subversion/bindings/javahl/src/org/tigris/subversion/ javahl/SVNClientInterface.java:40527-40534 /branches/feature-branch4/subversion/bindings/javahl/src/org/tigris/subversion/ javahl/SVNClientInterface.java:40562-40574 /trunk/subversion/bindings/javahl/src/org/tigris/subversion/ javahl/SVNClientInterface.java:24421-24423

We see that three revisions are eligible to be merged:

1.7-dev-client>svn mergeinfo --show-revs eligible ^/trunk r40577 r40578 r40579

Merge Tracking: A Quick 1.7 Example, Continued

We merge one of the eligible revisions, which changes single file (not the subtree with mergeinfo). Note that unlike 1.5.x-1.6.x, the recording of mergeinfo describing the merge produces a notification:
1.7-dev-client>svn merge ^/trunk . -c40577 --- Merging r40577 into '.': U subversion\libsvn_client\merge.c --- Recording mergeinfo for merge of r40577 into '.': U .

More importantly, unlike earlier versions, only the subtrees affected by the merge have their mergeinfo updated:
1.7-dev-client>svn st M . M subversion\libsvn_client\merge.c 1.7-dev-client>svn diff --depth empty Property changes on: . ___________________________________________________________________ Modified: svn:mergeinfo Merged /trunk:r40577

None of this is a big deal in this simple example...but imagine merging a diff affecting 100's of paths in a working copy with 100's of subtrees with mergeinfo.

Merge Tracking: What Can Be Improved?

Cyclic/reflective merges need to work better when not using -reintegrate.
The fundamental problem is that the smallest merged unit is a revision. So when we merge rX from 'trunk' to 'branch', resolve conflicts on the merge, and commit the whole thing as rY, then rY contains parts of trunk's history and part of branch's history. If we merge rY back to trunk we need to differentiate what is already part of trunk's history and what is part of branch's history.

Renames on either side of a merge prior to 1.6.x didn't work well (skips or deletes of newer paths). Things got a bit better in 1.6.x as a tree conflict is raised in some cases, but ideally merge should automatically follow renames on either side of the merge and do the right thing. Can't automatically differentiate a real merge vs. a --record-only merge. Performance It can always be better.

Merge Tracking: More Information

The Mergeinfo Manifesto http://www.collab.net/community/subversion/articles/mergeinfo.html Foreign repository merges http://blogs.open.collab.net/svn/2008/03/do-not-post-mer.html How subtree mergeinfo arises http://blogs.open.collab.net/svn/2009/11/where-did-that-mergeinfocome-from.html --reintegrate merges http://blogs.open.collab.net/svn/2008/07/subversion-merg.html

Merge Best Practices

The improvements and bug fixes are primarily on the client-side so embrace the latest client. USE 1.6.9! Reduce the creation of subtree mergeinfo by:
Merging to the root of your branches wherever possible. Avoiding shallow merges. Not merging to working copies with missing subtrees.

Working copy should be at a single revision

Avoid fixed revisions or switched children in the working copy.

DO NOT hand edit or remove svn:mergeinfo properties unless you are sure you know what you are doing (and recheck yourself) Commit all svn:mergeinfo changes

Merge Best Practices, Continued

Do not have local, uncommitted changes in your working copy prior to a merge action. Point your working copy at HEAD for the line of development to which you are merging. Identify the source path in your commit message (optionally the revisions though a diff of svn:mergeinfo values will give you that); if youre doing a subtree merge or a merge that creates noninheritable mergeinfo, explain why/what you are doing this After merging, use status and diff for sanity check. Build and run unit tests before committing.

CollabNet ALM Platform for Distributed Teams

Programming language, development tool, methodology agnostic. One integrated platform built for collaboration. Onsite or SaaS deployment options. Optimized for Subversion.

Summary
There are three main types of mergeinfo: explicit, inherited, and non-inheritable. Subtree merges (merges below the branch level) are the main culprit of issues with merge tracking. A lot of progress has been made in the implementation of merge tracking since 1.5.0s release and today people should be using 1.6.9. Familiarize yourself with merging best practices, use them, and consistently refer to them before merging.

Question & Answer

CollabNet, Inc. info@collab.net (888)778-9793

Learn More:
Subversion resources: http://www.open.collab.net/products/subversion/whatsnew.html. Join openCollabNet technical community and download CollabNet Subversion: www.collab.net. CollabNet TeamForge resources: http://www.open.collab.net/products/ctf. Virtual Conference, April 15th Agile ALM for Distributed Development Register at: http://www.open.collab.net/news/events/virtualConf2010/.