adjustments to borg2 output?

[jdchristensen]

Since borg2 is a breaking release, I have some suggestions for minor adjustments to the output.

First, when pruning, I'd like to be able to save in my log files a list of which archives were pruned, without seeing which ones were kept. With the --list option, you see both, like this:

Keeping archive (rule: secondly #1):     host2test2                           Sat, 2023-01-14 19:37:52 -0500 [7de1304bdc05c4649ebba407f237668e25b6ad698487068dfc5737eba403456e]
Keeping archive (rule: secondly #2):     host2test1                           Sat, 2023-01-14 19:36:49 -0500 [38ccf10bbdd667d0058ebdfca385f33bb427874744ebf1710200e625b17f45e0]
Pruning archive (1/1):                   test1                                Sat, 2023-01-14 19:35:33 -0500 [043650568d4845cf51db71d912eeb793c1726a2d7267ffaec44204b4570f1278]

Maybe we could change it so that the kept archives are only shown if --debug is given (or at least, if -v is given)?

When creating an archive, if you are connecting to a repo from a different host than the last one that used it, the chunks cache needs to be synchronized. Currently, without -v, there is no output related to this, and with -v you see something like:

Synchronizing chunks cache...
Archives: 3, w/ cached Idx: 1, w/ outdated Idx: 1, w/o cached Idx: 3.
Fetching and building archive index for host1test2 ...
Merging into master chunks index ...
Fetching and building archive index for host2test2 ...
Merging into master chunks index ...
Fetching and building archive index for host2test3 ...
Merging into master chunks index ...
Done.

I like using -v to have more info in the logs, but this seems too verbose to me. How about something like:

Synchronizing chunks cache:
Archives: 3, w/ cached Idx: 1, w/ outdated Idx: 1, w/o cached Idx: 3.
Fetching and building archive index for host1test2
Fetching and building archive index for host2test2
Fetching and building archive index for host2test3

with the other lines only shown with --debug?

Also, I've always found the line

Archives: 3, w/ cached Idx: 1, w/ outdated Idx: 1, w/o cached Idx: 3.

confusing. It sounds like there are 3 archives, 1 of which is cached and 3 of which are not, but 1+3 != 3. Moreover, you only need to give three numbers to specify the situation. How about something like:

1 fresh cached archive, 1 stale cached archive; 3 need fetching.

That's len(cached_ids & archive_ids), len(cached_ids - archive_ids), and len(archive_ids - cached_ids). Alternatively, we could have:

2 cached archives (1 of which is stale); 3 need fetching.

That's len(cached_ids), len(cached_ids - archive_ids), and len(archive_ids - cached_ids).

If any of these changes sound good, I'm willing to make pull requests.

[jdchristensen]

I see that I in fact converted the line

Archives: 3, w/ cached Idx: 1, w/ outdated Idx: 1, w/o cached Idx: 3.

incorrectly, as further evidence that it is confusing. This would translate to

0 fresh cached archives, 1 stale cached archive; 3 need fetching.

If we want to avoid the issue with plurals (archive vs archives), we could use:

Fresh cached archives: 0. Stale cached archives: 1. 3 need fetching.

Or maybe we just want to give the essential information, namely the number that need fetching. Then we could combine

Synchronizing chunks cache:
Archives: 3, w/ cached Idx: 1, w/ outdated Idx: 1, w/o cached Idx: 3.

into one line:

Synchronizing chunks cache for 3 archives.

[ThomasWaldmann]

borg prune: I rather want to show both kept/pruned archives by default (== as it is now). When specifying the prune policy via cli args, users tell what they want to keep. So it is a good thing when borg shows that it actually followed the policy, giving details why it kept a specific archives.

In the past (before borg 1.2, IIRC), we frequently got bug reports because users suspected borg was not pruning correctly (but it was always due to some misunderstanding / wrong expectations). Showing why some specific archive was kept makes it easier for users to understand that borg is working correctly.

So, guess we could implement the shorter output you want via some additional option, like --list-only-pruned?

Or you could use grep. :-)

[jdchristensen]

I think I'd prefer to leave the prune output the same rather than create an additional command line option. My idea was that --list would only show things pruned, but --list -v would show both kept and pruned. But I don't mind leaving it as is. It's just that one typically has many that are kept and few that are pruned, so it's hard to see the ones that are pruned. (I do currently use grep -v ...)

[ThomasWaldmann]

Merging into master chunks index ...

Yes, guess that should be DEBUG loglevel only.

[ThomasWaldmann]

This is the related code:

            logger.info(
                "Archives: %d, w/ cached Idx: %d, w/ outdated Idx: %d, w/o cached Idx: %d.",
                len(archive_ids),
                len(cached_ids),
                len(cached_ids - archive_ids),
                len(archive_ids - cached_ids),
            )

So, it is computing these set sizes:

• number of archives
• indexes we have in the index cache
• outdated indexes (their corresponding archive does not exist any more in the repo)
• indexes not in the cache / indexes we have to fetch

I find that quite informative.

[jdchristensen]

Note that any three of those let you compute the fourth, so there is some redundancy. (1st - 4th = 2nd - 3rd.)

Also, the phrasing is very confusing. It looks like it is saying that there are N archives, and then saying how many have various properties, but the three remaining numbers don't add up to the first number. And the two conditions "w/ cached Idx" and "w/o cached Idx" sound like they cover all possibilities: either an archive has a cached index or it doesn't, right? But the 2nd and 4th also don't add up to the first. So I doubt many people understand what the current output is saying.

If you want all of the information, a format like

Fresh cached archives: 0. Stale cached archives: 1. 3 need fetching.

tells you everything there is to know, but seems to me to be much clearer.

[ThomasWaldmann]

How about:

Cached archive chunk indexes: 3 fresh, 1 stale, 3 need fetching.

[jdchristensen]

I like that even better. I can make a PR. Should I only target borg2, in case people are parsing the output of borg1?

[ThomasWaldmann]

Yes, guess it would be enough for master branch.

If it bothers you in 1.2.x, you can also make a backport later for that, if you like. I don't think it is probable that someone parses that part of the output.