ui: avoid risk of 32-bit int overflow in VNC buffer check

For very large framebuffers, it is theoretically possible for the result of 'vs->throttle_output_offset * VNC_THROTTLE_OUTPUT_LIMIT_SCALE' to exceed the size of a 32-bit int. For this to happen in practice, the video RAM would have to be set to a large enough value, which is not likely today. None the less we can be paranoid against future growth by using division instead of multiplication when checking the limits. Reported-by: Laszlo Ersek <[email protected]> Signed-off-by: Daniel P. Berrangé <[email protected]> Reviewed-by: Laszlo Ersek <[email protected]> Reviewed-by: Philippe Mathieu-Daudé <[email protected]> Message-id: [email protected] Signed-off-by: Gerd Hoffmann <[email protected]>
byxk · Feb 16, 2018 · dffa1de · dffa1de
1 parent 8dfa306
commit dffa1de
Showing 1 changed file with 2 additions and 2 deletions.
diff --git a/ui/vnc.c b/ui/vnc.c
@@ -1579,8 +1579,8 @@ void vnc_write(VncState *vs, const void *data, size_t len)
      * handshake, or from the job thread's VncState clone
      */
     if (vs->throttle_output_offset != 0 &&
-        vs->output.offset > (vs->throttle_output_offset *
-                             VNC_THROTTLE_OUTPUT_LIMIT_SCALE)) {
+        (vs->output.offset / VNC_THROTTLE_OUTPUT_LIMIT_SCALE) >
+        vs->throttle_output_offset) {
         trace_vnc_client_output_limit(vs, vs->ioc, vs->output.offset,
                                       vs->throttle_output_offset);
         vnc_disconnect_start(vs);