[Serve] Allow scaling of replicas based on response time

[JGSweets]

Currently, replicas scale based on QPS. It would also be beneficial to be able to scale based on response time. This can help ensure a maximum response time when a system gets overloaded due to long running queries as opposed to the number of queries.

[WesleyYue]

More generally, it would be useful we can can plugin our own scaling logic so we can tailor it to our needs (for example, target prompt tokens per second)

[github-actions]

This issue is stale because it has been open 120 days with no activity. Remove stale label or comment or this will be closed in 10 days.

[github-actions]

This issue was closed because it has been stalled for 10 days with no activity.