experiments: provide access to experiment plots

[pmrowla]

While thinking about #4448 it occurred to me that there's currently no way to use dvc plots with experiment revisions. Implementation-wise, we should only need a dvc exp plots ... subcommand that runs regular plots in the experiments workspace, the same way that show/diff work for experiments.

One concern would be how to handle situations where someone wants to save an experiments plot - do we need functionality to git add & commit plots output into the internal experiment branches? Or can we just say for experiments, generated plots should always be considered temporary, and not be committed anywhere?

@dmpetrov @pared

[dmpetrov]

@pmrowla, great catch!

What about metrics? Does dvc metrics diff 123456 abcdef where abcdef is a ref to an experiment?

One concern would be how to handle situations where someone wants to save an experiments plot

@pmrowla could you please clarify? Are you talking about the output of the dvc plot diff command? If so then, yes, it is a temporary file.

[pmrowla]

@dmpetrov dvc metrics diff will not work with experiment refs, but dvc exp diff ... can be used to compare regular commits with experiment refs.

And yes, I was referring to the html output files generated by plot diff and show.

[pared]

I think that we should not allow persistence of experiment plots, if the change is worth committing, it should be committed to the repo.

[pmrowla]

So after starting an implementation of dvc exp plots ... subcommands, I've got a couple thoughts:

Rather than having a separate dvc exp plots show <experiment_rev> ... command, it may make more sense to just use regular dvc plots show. To show plots for a specific (regular) commit, you currently need to do:

$ git checkout <rev>
$ dvc plots show ...

For experiments, you can already get the same behavior, by doing:

$ dvc exp checkout <experiment_rev>
$ dvc plots show

So I'm not sure it makes sense for us to have a dedicated dvc exp plots show command.

For plots diff, we do accept specific revisions, but rather than have a dedicated dvc exp plots diff ... subcommand, we could just add a -e/--experiment flag to the existing dvc plots diff to denote that experiment revisions will be included in the list of revisions to diff.

Likewise, for #4455 I think all we would need is the new parallel coordinates template to do dvc plots diff -t <new_template_name> [-e] [revisions...] (and could be used with both regular and experiment revisions)

So in summary there's 2 questions here:

1. Would dvc exp checkout + dvc plots show work for us? (It seems to me that it would work)
2. Would we prefer dvc plots diff -e/--experiment ... or a separate dvc exp plots diff ... (using -e rather than an entire new command seems cleaner to me)
   • This question can also be extended to metrics/params, do we want to keep the dedicated dvc exp diff ... command, or should we just have -e for metrics diff and params diff?

@dmpetrov @pared

[pared]

@pmrowla I agree that separate command might be overkill.
-e option for diff would match nicely with run -e.
Another option would be to provide a special constant that would mark the current workspace for plots and metrics.
For now, it is only possible to plot current, uncommitted workspace only against one other revision (eg dvc plots diff HEAD)
Maybe we should just allow something like dvc plots diff workspace HEAD. That way we could handle experiments, and would extend functionality of plot.

Ah, now I see that workspace approach would prevent us from comparing experiments, which is not good. I guess -e is the way to go.