Bug 1754509 - Reject non-reflectable calls early in nsXPCWrappedJS::C…

…allMethod(). r=nika

We need to reject calls to non-reflectable methods before we do anything (like
AutoEntryScript) that might cause a GC, because these methods have not been
annotated with JS_HAZ_CAN_RUN_SCRIPT.

Differential Revision: https://phabricator.services.mozilla.com/D138351

js/xpconnect/src/XPCWrappedJSClass.cpp

@@ -764,6 +764,14 @@ nsXPCWrappedJS::CallMethod(uint16_t methodIndex, const nsXPTMethodInfo* info,
     return NS_ERROR_UNEXPECTED;
   }
 
+  // We need to reject an attempt to call a non-reflectable method before
+  // we do anything like AutoEntryScript which might allocate in the JS engine,
+  // because the method isn't marked with JS_HAZ_CAN_RUN_SCRIPT, and we want
+  // to be able to take advantage of that in the GC hazard analysis.
+  if (!info->IsReflectable()) {
+    return NS_ERROR_FAILURE;
+  }
+
   Value* sp = nullptr;
   Value* argv = nullptr;
   uint8_t i;
@@ -790,7 +798,7 @@ nsXPCWrappedJS::CallMethod(uint16_t methodIndex, const nsXPTMethodInfo* info,
 
   JSContext* cx = ccx.GetJSContext();
 
-  if (!cx || !info->IsReflectable()) {
+  if (!cx) {
     return NS_ERROR_FAILURE;
   }