Metrics Collection

[aluzzardi]

We should have a way to do metrics collection, either by supporting a solution and implementing it by documenting it, or shipping with a default.

A possible solution would be to deploy a prometheus and run collectors on every node.

[aluzzardi]

/cc @docker/fiesta-cucaracha-maintainers @ehazlett

[ehazlett]

+1 for builtin metrics.

I like the idea of the collector as it keeps things very light on the node side as they just expose their current status and no storage is needed. The collector can then scrape on whatever interval they desire and the processing and storage is on the collector.

[ehazlett]

Oh wait I think I misunderstood. Would the collector be on each node or external? If on the node, what would it be collecting?

[dongluochen]

+1 on metric collection. I think it great to have a default and swappable monitoring framework. It enables cluster state reporting, utilization monitoring, alerting, auto-scaling,etc. We may investigate how to deploy it using V2 declarative API.

[aluzzardi]

@ehazlett @dongluochen

This is in the same spirit as #248, summary from the other conversation (with s/logging/metrics/):

We do not support distributed metrics in the product - we just make sure distributed metrics can be deployed on the platform and engines configured to leverage it. Maybe someday we'll auto deploy it, but in the meantime we'll just document it as a recipe.

Makes sense?

[aluzzardi]

Labeling this as a docs issue - we'd need to come up with a recipe and make sure it's well supported

/cc @sfsmithcha @mgoelzer