History: capacity

[GeorgeMac]

We should implement some configurable bounds around history / reflog capacity.

Some thoughts on how to represent the limits:

1. Per pipeline / phase count threshold (phase.max)

Put a per phase threshold on the number of versions retained.
This will at-least bound history to a constant factor of the number of phases.

Downside: lots of phases could bloat history

• Per pipeline / phase duration threshold (phase.max_age)

Put a timestamp threshold on the number of versions.
Anything older gets deleted.
Could be combined with a minimum to preserve at-least N entries.

Downside: busy phases could produce a lot of history

• Global count (max log capacity for all events across all phases) (global.max)

We could have a global count threshold, where it discards oldest entries first.
This would give a solid upper bound on the size of history.
It could lead to situations where some phases have no history if the limit is reached and they haven't had much.

Downside: Busy phases could saturate capacity (pushing out quiet phases)

• Combination (hard, soft and min limits)

We could have some combined hard, soft and min limits, with time being the function used to decide where to prune.
This tries to be a best effort cap on all history (given that number of phases * min limit < hard), where exceeding hard triggers the deletion process. It retains a minimum amount of history per phase, so there is always something to revert back to. Soft limit is used to auto-prune history per-phase, so that busy phases don't steal all the capacity.

Hard limit would enforce a global count to trigger when to start pruning.
Soft limit would be on the phase to set a maximum depth.
Min would ensure at-least N entries are preserved per phase.

Sensible defaults for these would hopefully get most folks a long way.