changelist-design.txt

The Problem We're Solving
-------------------------

Subversion users typically edit sets of files in their working copy.
If a working copy contains a set of edited files which represents a
single logical change, then commands like 'svn diff', 'svn status',
'svn revert' and 'svn commit' automatically discover the edited files
and act on them.

A common problem, however, is that users often work on more than one
set of logical changes at a time.  The user is required to remember
which edited file belongs to which set, and carefully run 'diff',
'revert', or 'commit' commands only on lists of files which belong
together.

One workaround for this problem is to checkout multiple working
copies, and have one task per working copy.  Of course, this uses a
lot of disk space, and it's sometimes inconvenient to move around
between working copies.

The simple solution we're proposing here is to teach the svn client
(and working copy) do some simple management of local, human-named
sets of files, known as 'changesets'.  The goal is to allow users to
create, view, and manipulate sets of files in a working copy by
referring to them by name.


Doesn't Perforce Do This?
-------------------------

Perforce performs changelist management, and it's a large motivation
for this new feature.  But there's no way to emulate Perforce's
feature exactly; it has a different network model than Subversion.  So
instead, we'll examine the use-cases that Perforce enables, and
discuss how to solve those same use-cases in Subversion.


Non-Problems
-------------

Here are problems/features that are NOT in our list of goals:

  * Server management of changesets

    Subversion prides itself on being disconnected;  that's why it
    scales so well.  A changeset is an ephemeral thing created by a
    single user in a single working copy, whose only purpose is to
    make it easier to manipulate a change-in-progress.  It's not a
    "named revision", or a long-lived object in the repository.
    That's what global revision numbers are for.

    Some people aren't happy with the way tags work in Subversion, and
    have asked for the ability to identify repository revisions by
    human name.  While everybody wants to see the ability to search
    over revprops (for many reasons!), that whole issue is out of
    scope for this changelist feature.  It's been suggested that when
    a changelist gets committed, it become a searchable revprop;
    sounds fine, but lets get changelists and searchable revprops
    implemented independently first!

  * Enforcement of groupings

    Changesets don't exist as a prescriptive SCM process.  Some have
    suggested that the client not allow people to commit individual
    files in a changelist, or to do some side of server-side process
    enforcement revolving around changelists.  This is definitely not
    in the Subversion spirit, which allows teams to create whatever
    policies they wish.  The only purpose of changelists is to do
    provide some convenient bookkeeping to the user.

  * Overlapping changelists

    A number of people ask "but what if two different changes within a
    single file belong to different logical changes?"  My reply is:
    either "tough luck" or "don't do that" or "checkout a separate
    working copy".  My feeling is that trying to create a UI to
    manipulate individual diff-hunks within a file is a HUGE can of
    worms, probably best suited for a GUI.  While I wouldn't rule it
    out as a future *enhancement* to a changelist feature, it's
    certainly not worth the initial effort in the first draft of
    changelist management.  Overlapping changelists do occasionally
    happen, but they're rare enough that's it's not worth spending 90%
    of our time on a 10% case -- at least not in the beginning.

  * "Shelving" of changes

    Distributed version control systems don't have this sort of
    problem;  one could just do a 'local commit' of each
    changeset-in-progress, create local branches, and magically swap
    patches in and out as needed.  To that end, many have talked about
    making subversion working copies into "deep" objects containing
    some degree of history, or to write a nice 'svn patch' command to
    read custom 'svn diff' output.  My response is:  nice ideas, and
    those sort of really advanced designs are certainly things that
    simple changelist management can grow to take advantage of, but
    aren't prerequisites for tackling this problem.



Use-Cases
---------

A. Define a changelist by explicitly adding/removing paths to it.

B. See all existing changelist names (and their member paths)

C. Destroy a changelist definition all at once.

D. Examine all edits within a changelist (svn diff)

E. Revert all edits within a changelist (svn revert)

F. Commit all edits within a changelist (svn commit)

G. Receive server changes only paths within a changelist (svn update)

H. See the history of all paths within a changelst (svn log)

I. Fetch or set props on every path within a changelist (svn pl/ps/pe/pg/pd)



How Perforce Tackles the Use-Cases
----------------------------------

A.  Defining changelists

The Perforce server tracks each and every working copy, as well as
every changelist within every working copy.  All working copy files
are read-only until the user declares the intent to edit ('p4 edit')
one.  The server then makes the file read-write and places it into a
changelist with the name 'default'.

Users aren't allowed to invent their own names for changelists, as
this might lead to namespace overlaps.  (This is a side effect of
having the server track all changelists.)  'p4 change' creates a new
changelist by prompting the user for a log message, at which point the
server yanks the 'next' global global revision number and assigns it
as a name for the changelist.  The server not only tracks the
changelist via some number, but also tracks the
log-message-in-progress for the list.  ('p4 describe' can show the log
message attached to a changelist.)

B.  Viewing changelists

At any time, the 'p4 open' command shows all files that are being
edited, and which changelists they belong to.  It's quite similar to
the 'svn status' command, except that the output is somewhat harder to
read, due to non-aligned columns.

The response time is also quite fast, since p4 doesn't need to crawl
the working copy to discover edited files.  On the other hand, p4
doesn't scale so well when the server tries to track thousands of
users.

C.  Destroying changelists

'p4 change -d' will delete a changelist, but only if the edited files
within the changelist have been reverted.

D.  Viewing edits in a changelist

'p4 diff' shows contextual diffs for all edited files.  This is
actually a bit weak, as it shows diffs for *all* changelists in a
working copy.  Subversion should improve on this by allowing one to
'diff' just a single changelist.

E.  Reverting a changelist

'p4 revert -c NNN' reverts all edited files within changelist #NNN.
Note that it's also possible to revert single files ('p4 revert
foo.c').  If a single file within a changelist is reverted, its path
is removed from the changelist.

F.  Committing a changelist

'p4 submit -c NNN' atomically commits changelist #NNN to the
repository.  If the commit succeeds, a *new* global revision number is
assigned to the final commit, and the old 'NNN' number is discarded.
(This means that p4 actually burns through global revnums at twice the
speed as subversion!)  After the commit, the working copy no longer
has any record of the changelist.

G.  Updating a changelist

'p4 sync' is equivalent to 'svn up'.  Like subversion, 'p4 sync' can
be restricted to specific path targets, but amazingly not restricted
to a set of paths that make up a changelist.  This may be something
subversion can improve upon.

H.  Examining the history of changelist members

'p4 changes' is the closest thing to 'svn log'.  With no arguments, it
shows all changelists ever submitted.  With specific path arguments,
it limits the response to showing only changelists that affected those
paths.  Again, a changelist number cannot be supplied, which is
surprising.

I.  Propgets/sets on a changelist

Perforce has no versioned metadata.




Proposal for Subversion's Tackling of Use-Cases
-----------------------------------------------

A.  Defining changelists

Subversion's changelist feature will be entirely client-side
bookkeeping.  The purpose is to allow users to 'talk about' a set of
local paths via a convenient name, often restricting subcommands to
operate only on those paths.

The 'svn changelist' command allows a user to define a changelist with
an arbitrary UTF-8 name, as well as add member paths.  (At the moment,
a --remove flag is used to remove member paths.)  Unversioned items may
not be added to changelists.

$ svn changelist MYCHANGE foo.c bar.c
Path 'foo.c' is now part of changelist 'mychange'.
Path 'bar.c' is now part of changelist 'mychange'.

$ svn changelist bar.c --remove
Path 'bar.c' is no longer associated with a changelist.


### Open question: should we add a UI which allows the working copy to
    manage a log-message-in-progress for each changelist, the way p4
    does?  This could be something stored in ~/.subversion/ area.


B.  Viewing changelists

'svn status' currently shows changelist definitions by crawling the
working copy.  Output is much more readable than perforce, because
we're still preserving column alignment.

$ svn st
?      1.2-backports.txt
M      notes/wc-improvements

--- Changelist 'status-cleanup':
M      subversion/svn/main.c
       subversion/svn/revert-cmd.c
M      subversion/svn/info-cmd.c

--- Changelist 'status-printing':
M      subversion/svn/status-cmd.c



Note that unlike perforce, changelist membership is orthogonal to
whether or not the file has local modifications.  So it's possible for
'svn status' to show a changelist containing unmodified files.
Conversely, it's possible for a file to be modified, but unassociated
with any changelist.

'svn status' considers changelist membership to be inherently
"interesting enough" to justify displaying a path, regardless of
whether it's modified.

Note that merely upgrading subversion won't break scripts that parse
'svn status' output.  Such scripts might break *only* if users begin
to use the new changelist feature.  This is a good balance between
allowing subversion's development to progress, while not automatically
punishing users for upgrading.  (Either way, the "---" characters
should prevent scripts from accidentally detecting conflicts with "^C"
regular expressions.)

### Open question:  at the moment, changelists are implemented by
    simply storing a new attribute in the .svn/entries file.  Rather
    than having the svn client crawl and 'discover' changelists,
    should we take a hint from p4 and have them centrally managed in
    the ~/.subversion/ area?

     Pros:
       - much faster than crawling
       - whole changelist definition available, regardless of CWD

     Cons:
       - breaks the 'portable WC' ideal.  (If WC moves to another box,
         changelist definition is lost.)


### Open question:  should 'svn status' be able to restrict its output
    to a single changelist, a la 'svn status --changelist mychange'?


C.  Destroying changelists

Commands can be restricted to operate only on changelist members by
specifying the "--changelist NAME" flag.  (Perhaps it can be shortened
to '--cl' also?)

To destroy a changelist, one would need to remove all member-paths
from it.  There's no good UI for this yet, other than to use 'svn
changelist --remove path1 path2 path3 ...'.  ### Improve this?


D.  Viewing edits in a changelist

Improve on perforce by allowing 'svn diff' to restrict its output to
only members of a certain changelist:

$ svn diff --changelist mychange
[...]


E.  Reverting a changelist

Allow 'svn revert' to restrict its effect just to members of a
changelist:

$ svn revert --changelist mychange
[...]

Again, note that this won't destroy the changelist.  The changelist
would now contain just a set of unmodified paths, and 'svn status'
would continue to display them.  (This differs from perforce, whereby
local-edits are intimately tied to changelist membership.)


F.  Committing a changelist

'svn commit' should be able to commit only changelist members, just as
if the paths had been typed on the commandline individually:

$ svn commit --changelist mychange
Modifying foo.c
Adding bar.c
[...]
Committed revision YYY.


After the commit succeeds, the committed files are NO LONGER
associated with the changelist, and so the changelist definition
ceases to exist.  (Note: we probably want to have a switch to
'preserve the changelist' after a commit, similar to the way in which
the '--no-unlock' switch preserves locks after a commit.)

If the user chooses to commit just a single member of a changelist,
that member is removed from the changelist after the commit.


G.  Updating a changelist

### Open question:  is this a useful use-case?  Perforce doesn't have
    it, and I've never missed it.  I always want to update the entire
    working copy, not just some small set of files.


H.  Examining the history of changelist members

'svn log' should be able to restrict its history retrieval to only
revisions which affected members of the changelist.  So running

$ svn log --changelist mychange

...should produce output equivalent to

$ svn log member1 member2 member3 ...


'svn log' already knows not to print log messages more than once
(i.e. it prints the union of all revisions).

Note that this feature would be an improvement over perforce, which
allows multiple targets on the commandline, but no changelist
shorthand for them.


I.  Propgets/sets on a changelist

'svn proplist', 'svn propget', 'svn propset', 'svn propdel' should all
work with the --changelist switch as well, so that a user can quickly
perform metadata operations on a whole set of files.

 ### Open question: should we also allow 'svn lock/unlock' to operate
     on changelists?  It might be just as convenient in certain
     scenarios.



----------------

### Open UI question:

   If one's CWD is deep within a working copy, how should

        $ svn subcommand --changlist mychange

    ...behave?  Should it operate on *all* members of the changelist,
    or only those members within the CWD (and recursively "below")?

   --> malcolmr and dlr believe that it's perfectly fine to use only
       parts of changelists 'below' the target path.


--------------------

==> Finished items:

    * svn changelist [--remove]

    * svn status shows grouped changelists
        -  'svn status --changelist' works too
    * 'svn info' shows changelists

    * svn commit --changelist
    * svn revert --changelist
    * svn log --changelist
    * svn diff --changelist  (wc-wc and wc-repos cases)
    * svn update --changelist
    * svn lock/unlock --changelist
    * svn propget/propset --changelist
###    * svn proplist/propdel --changelist



==> TO-DO:

* make --cl the same as --changelist, for convenience?

* questions about commits:  

    - how does 'svn ci --changelist' interact with nonrecursive commits?
    - how does it interact with a list of specific targets?
    - how does it deal with a schedule-delete folder?


----------------------------

Commandline UI use-cases:


1. add path(s) to a CL:

     svn cl CLNAME foo.c bar.c baz.c

2. remove path(s) from whatever CLs they each belong to.

     svn cl --remove foo.c bar.c baz.c

3. move path(s) from CL1 to CL2.

     svn cl CL2 foo.c

4. undefine a CL all at once (by removing all members)

     svn cl --remove --changelist CLNAME

5. rename a CL

     svn cl NEWNAME --changelist OLDNAME


==================================================================

Feature Revamp:  sussman and cmpilato.


Goal:  changelists should be treated as 'filters' everywhere, not as a
way to just add targets to a commandline.


The basic syntax of commands will be:

  svn subcommand target1 target2 ... targetN \
                 --changelist foo1 --changelist foo2 ... --changelist fooM

The CLI parses the targets as usual: possibly inserting an implicit
'.' target, canonicalizing the list, etc.

The CLI now passes a list of changelist-names down into each
svn_client_subcommand() routine as a "bunch of filters" to apply while
working.  If svn_client_subcommand() decides to process a target --
either one it got explicitly, or one it discovered through recursion
-- it first checks that the target is a member of one of the
changelists.  If not, it skips the target and keeps going.

(This is the way 'svn commit' currently works:  harvest_committables()
only harvests things that are committable *and* a member of the
passed-in changelist.)

This means that the UI use-cases listed above change slightly:


   4. undefine a CL all at once (by removing all members)
   
        svn cl TARGET --remove --changelist CLNAME
   
        (TARGET might be implicit '.' or not, and depth is empty by
        default; use --depth to override.)
   
   5. rename a CL
   
        svn cl NEWNAME TARGET --changelist OLDNAME
   
        (TARGET might be implicit '.' or not, and depth is empty by
        default; use --depth to override.)


TO-DO list:

  [X] allow multiple --changelist args
  [X] svn status should display grouped changelists
  [X] 'svn info' should display a target's changelist field
  [X] rename --keep-changelist option to --keep-changelists
  [X] fix --changelist and allow multiple changelists in subcommands:

      No-problem subcommands:
      [X] svn changelist --changelist
      [X] svn commit --changelist
      [X] svn diff --changelist (only wc-wc and wc-repos cases)
      [X] svn info --changelist
      [X] svn propget --changelist
      [X] svn proplist --changelist
      [X] svn propset --changelist
      [X] svn propdel --changelist
      [X] svn revert --changelist
      [X] svn status --changelist

      Problem subcommands (see below):
      [X] svn update --changelist
      [X] svn lock --changelist              ### removed changelist support
      [X] svn log --changelist               ### removed changelist support
      [X] svn unlock --changelist            ### removed changelist support

  [ ] ensure that the bindings implementations of these APIs are up to snuff
  [X] write tests!

Problem Subcommands:

  Using a definition of --changelist as a filter means that
  subcommands which are, by default, non-recursive in nature, have a
  somewhat odd interface.  For example, 'svn info --changelist FOO'
  (which ultimately translates to 'svn info . --depth empty
  --changelist FOO') will either return exactly one info result, or
  exactly none, depending on whether or not the current working
  directory is in changelist FOO.  This is trivially worked around by
  deepening the invocation: 'svn info -R --changelist FOO'.  But what
  about subcommands for which there is no --depth support, such as
  'lock', 'log', 'unlock'?  Do we lose the changelist support, or grow
  some sort of depth-crawling ability for these things?  [RESOLUTION:
  We've removed changelist support from 'lock', 'unlock', and 'log'.]

  'svn update' presents an interesting challenge, too.  The public
  svn_client_update3() API takes a list of paths, and returns a list
  of revision numbers to which those paths were updated.  Each path is
  treated as, effectively, a separate update -- complete with output
  line that notes the updated-to revision.  So, if we do changelist
  expansion outside the API, we might turn a single-target operation
  into a multi-target one, and the user sees N full updates processes
  happen.  If we push 'changelists' down into the API, we can fake a
  single update with notification tricks.  But that starts to get
  nasty when we look at non-file changelist support later and the
  interactions with externals and such.  And if we push 'changelists'
  all the way down into the update editor, then we've got a mess of a
  whole 'nuther type, downloading tons of server data we won't use,
  and so on.  [RESOLUTION: Let the command-line client do the
  changelist path expansion.]