Bug 1871186 - Make simulated alloc failure when marking less frequent…

… r=jandem

Something I found when testing the original testcase attached to the bug was
that OOM tests involving GC can take a very long time (and sometimes time out)
because of the very frequent simulated OOM when pushing things onto the mark
stack. I added this originally to improve test coverage here but this may have
been over-enthusiastic. The patch changes this to rely on simulated alloc
failure whew the mark stack vector is resized and starts this off with a
smaller size in debug builds to ensure coverage.

Depends on D197911

Differential Revision: https://phabricator.services.mozilla.com/D197912

js/src/gc/GCMarker.h

@@ -24,7 +24,12 @@ class GCMarker;
 class SliceBudget;
 class WeakMapBase;
 
+#ifdef DEBUG
+// Force stack resizing to ensure OOM test coverage in debug builds.
+static const size_t MARK_STACK_BASE_CAPACITY = 4;
+#else
 static const size_t MARK_STACK_BASE_CAPACITY = 4096;
+#endif
 
 enum class SlotsOrElementsKind {
   Unused = 0,  // Must match SlotsOrElementsRangeTag

js/src/gc/Marking.cpp

@@ -1924,7 +1924,7 @@ inline MarkStack::SlotsOrElementsRange MarkStack::popSlotsOrElementsRange() {
 
 inline bool MarkStack::ensureSpace(size_t count) {
   if (MOZ_LIKELY((topIndex_ + count) <= capacity())) {
-    return !js::oom::ShouldFailWithOOM();
+    return true;
   }
 
   return enlarge(count);