Xi/randr: fix handling of PropModeAppend/Prepend

The handling of appending/prepending properties was incorrect, with at
least two bugs: the property length was set to the length of the new
part only, i.e. appending or prepending N elements to a property with P
existing elements always resulted in the property having N elements
instead of N + P.

Second, when pre-pending a value to a property, the offset for the old
values was incorrect, leaving the new property with potentially
uninitalized values and/or resulting in OOB memory writes.
For example, prepending a 3 element value to a 5 element property would
result in this 8 value array:
  [N, N, N, ?, ?, P, P, P ] P, P
                            ^OOB write

The XI2 code is a copy/paste of the RandR code, so the bug exists in
both.

CVE-2023-5367, ZDI-CAN-22153

This vulnerability was discovered by:
Jan-Niklas Sohn working with Trend Micro Zero Day Initiative

Signed-off-by: Peter Hutterer <[email protected]>
(cherry picked from commit 541ab2ecd41d4d8689e71855d93e492bc554719a)

unix/Xvnc/programs/Xserver/Xi/xiproperty.c

@@ -730,7 +730,7 @@ XIChangeDeviceProperty(DeviceIntPtr dev, Atom property, Atom type,
                 XIDestroyDeviceProperty(prop);
             return BadAlloc;
         }
-        new_value.size = len;
+        new_value.size = total_len;
         new_value.type = type;
         new_value.format = format;
 
@@ -747,7 +747,7 @@ XIChangeDeviceProperty(DeviceIntPtr dev, Atom property, Atom type,
         case PropModePrepend:
             new_data = new_value.data;
             old_data = (void *) (((char *) new_value.data) +
-                                  (prop_value->size * size_in_bytes));
+                                  (len * size_in_bytes));
             break;
         }
         if (new_data)

unix/Xvnc/programs/Xserver/randr/rrproperty.c

@@ -209,7 +209,7 @@ RRChangeOutputProperty(RROutputPtr output, Atom property, Atom type,
                 RRDestroyOutputProperty(prop);
             return BadAlloc;
         }
-        new_value.size = len;
+        new_value.size = total_len;
         new_value.type = type;
         new_value.format = format;
 
@@ -226,7 +226,7 @@ RRChangeOutputProperty(RROutputPtr output, Atom property, Atom type,
         case PropModePrepend:
             new_data = new_value.data;
             old_data = (void *) (((char *) new_value.data) +
-                                  (prop_value->size * size_in_bytes));
+                                  (len * size_in_bytes));
             break;
         }
         if (new_data)