Bug 1470558: Distinguish yields and awaits in completion values. r=jo…

…rendorff

This patch introduces a new type to the debugger, `js::Completion`, describing
how a given JavaScript evaluation completed. It's a wrapper around a `Variant`
with alternatives:
- Return
- Throw
- Terminate
- InitialYield (the initial yield of a generator, returning the generator object)
- Yield (subsequent yields of a generator)
- Await (both initial and subsequent)

We can construct a `Completion` in two ways:

- From any JavaScript operation's result (a success value and a context carrying
  an exception value and stack). This only distinguishes between Return, Throw,
  and Terminate.

- From a stack frame that's about to be popped. This allows us to identify
  yields and awaits.

Given a `Completion` we can construct Debugger API 'completion values' to pass
to hooks, as well as the resumption/value/context states that tell the engine
how to continue execution. Within Debugger itself, `Completion` values are a
convenient place to gather up various bits of logic: identifying suspensions,
chaining resumption values from multiple Debugger hooks, and so on.

Although `Completion` should be used throughout Debugger, this patch only uses
it for the `onPop` hook. Subsequent patches in the series will apply it to other
cases where Debugger can invoke JavaScript.

Differential Revision: https://phabricator.services.mozilla.com/D24997

--HG--
extra : moz-landing-system : lando

js/src/doc/Debugger/Conventions.md

@@ -66,11 +66,9 @@ Similarly:
 
 ## Completion Values
 
-When a debuggee stack frame completes its execution, or when some sort
-of debuggee call initiated by the debugger finishes, the `Debugger`
-interface provides a value describing how the code completed; these are
-called *completion values*. A completion value has one of the
-following forms:
+The `Debugger` API often needs to convey the result of running some JS code. For example, suppose you get a `frame.onPop` callback telling you that a method in the debuggee just finished. Did it return successfully? Did it throw? What did it return? The debugger passes the `onPop` handler a *completion value* that tells what happened.
+
+A completion value is one of these:
 
 <code>{ return: <i>value</i> }</code>
 :   The code completed normally, returning <i>value</i>. <i>Value</i> is a
@@ -82,12 +80,51 @@ following forms:
     the value was thrown, and may be missing.
 
 `null`
-:   The code was terminated, as if by the "slow script" dialog box.
+:   The code was terminated, as if by the "slow script" ribbon.
+
+Generators and async functions add a wrinkle: they can suspend themselves (with `yield` or `await`), which removes their frame from the stack. Later, the generator or async frame might be returned to the stack and continue running where it left off. Does it count as "completion" when a generator suspends itself?
+
+The `Debugger` API says yes. `yield` and `await` do trigger the `frame.onPop` handler, passing a completion value that explains why the frame is being suspended. The completion value gets an extra `.yield` or `.await` property, to distinguish this kind of completion from a normal `return`.
+
+<pre>
+{ return: *value*, yield: true }
+</pre>
+
+where *value* is a debuggee value for the iterator result object, like `{ value: 1, done: false }`, for the yield.
+
+When a generator function is called, it first evaluates any default argument
+expressions and destructures its arguments. Then its frame is suspended, and the
+new generator object is returned to the caller. This initial suspension is reported
+to any `onPop` handlers as a completion value of the form:
+
+<pre>
+{ return: *generatorObject*, yield: true, initial: true }
+</pre>
+
+where *generatorObject* is a debuggee value for the generator object being
+returned to the caller.
+
+When an async function awaits a promise, its suspension is reported to any
+`onPop` handlers as a completion value of the form:
+
+<pre>
+{ return: *promise*, await: true }
+</pre>
+
+where *promise* is a debuggee value for the promise being returned to the
+caller.
+
+The first time a call to an async function awaits, returns, or throws, a promise
+of its result is returned to the caller. Subsequent resumptions of the async
+call, if any, are initiated directly from the job queue's event loop, with no
+calling frame on the stack. Thus, if needed, an `onPop` handler can distinguish
+an async call's initial suspension, which returns the promise, from any
+subsequent suspensions by checking the `Debugger.Frame`'s `older` property: if
+that is `null`, the call was resumed directly from the event loop.
 
-If control reaches the end of a generator frame, the completion value is
-<code>{ throw: <i>stop</i> }</code> where <i>stop</i> is a
-`Debugger.Object` object representing the `StopIteration` object being
-thrown.
+Async generators are a combination of async functions and generators that can
+use both `yield` and `await` expressions. Suspensions of async generator frames
+are reported using any combination of the completion values above.
 
 
 ## Resumption Values

js/src/doc/Debugger/Debugger.Frame.md

@@ -117,6 +117,10 @@ the `Debugger.onEnterFrame` handler is called each time a frame is
 resumed. (This means these events can fire multiple times for the same
 `Frame` object, which is odd, but accurately conveys what's happening.)
 
+The [completion value][cv] passed to the `frame.onPop` handler for a suspension
+contains additional properties to clarify what's going on. See the documentation
+for completion values for details.
+
 
 ## Stepping Into Generators: The "Initial Yield"
 
@@ -279,19 +283,19 @@ the compartment to which the handler method belongs.
     frames.
 
 `onPop`
-:   This property must be either `undefined` or a function. If it is a
-    function, SpiderMonkey calls it just before this frame is popped,
-    passing a [completion value][cv] indicating how this frame's execution
-    completed, and providing this `Debugger.Frame` instance as the `this`
-    value. The function should return a [resumption value][rv] indicating
-    how execution should proceed. On newly created frames, this property's
-    value is `undefined`.
+:   This property must be either `undefined` or a function. If it is a function,
+    SpiderMonkey calls it just before this frame is popped or suspended, passing
+    a [completion value][cv] indicating the reason, and providing this
+    `Debugger.Frame` instance as the `this` value. The function should return a
+    [resumption value][rv] indicating how execution should proceed. On newly
+    created frames, this property's value is `undefined`.
 
     When this handler is called, this frame's current execution location, as
     reflected in its `offset` and `environment` properties, is the operation
-    which caused it to be unwound. In frames returning or throwing an
-    exception, the location is often a return or a throw statement. In frames
-    propagating exceptions, the location is a call.
+    which caused it to be unwound. In frames returning or throwing an exception,
+    the location is often a return or a throw statement. In frames propagating
+    exceptions, the location is a call. In generator or async function frames,
+    the location may be a `yield` or `await` expression.
 
     When an `onPop` call reports the completion of a construction call
     (that is, a function called via the `new` operator), the completion
@@ -320,15 +324,15 @@ the compartment to which the handler method belongs.
     resumption value each handler returns establishes the completion value
     reported to the next handler.
 
+    The `onPop` handler is typically called only once for a given
+    `Debugger.Frame`, after which the frame becomes inactive. However, in the
+    case of [generators and async functions](#suspended), `onPop` fires each
+    time the frame is suspended.
+
     This handler is not called on `"debugger"` frames. It is also not called
     when unwinding a frame due to an over-recursion or out-of-memory
     exception.
 
-    The `onPop` handler is typically called only once for a given frame,
-    after which the frame becomes inactive. However, in the case of
-    [generators and async functions](suspended), `onPop` fires each time
-    the frame is suspended.
-
 
 ## Function Properties of the Debugger.Frame Prototype Object
 

js/src/gc/Marking.cpp

@@ -431,6 +431,11 @@ JS_PUBLIC_API void JS::UnsafeTraceRoot(JSTracer* trc, T* thingp,
   js::TraceNullableRoot(trc, thingp, name);
 }
 
+namespace js {
+class SavedFrame;
+class AbstractGeneratorObject;
+}  // namespace js
+
 // Instantiate a copy of the Tracing templates for each public GC pointer type.
 #define INSTANTIATE_PUBLIC_TRACE_FUNCTIONS(type)                          \
   template JS_PUBLIC_API void JS::UnsafeTraceRoot<type>(JSTracer*, type*, \
@@ -441,6 +446,8 @@ JS_PUBLIC_API void JS::UnsafeTraceRoot(JSTracer* trc, T* thingp,
       JSTracer*, type*, const char*);
 FOR_EACH_PUBLIC_GC_POINTER_TYPE(INSTANTIATE_PUBLIC_TRACE_FUNCTIONS)
 FOR_EACH_PUBLIC_TAGGED_GC_POINTER_TYPE(INSTANTIATE_PUBLIC_TRACE_FUNCTIONS)
+INSTANTIATE_PUBLIC_TRACE_FUNCTIONS(SavedFrame*);
+INSTANTIATE_PUBLIC_TRACE_FUNCTIONS(AbstractGeneratorObject*);
 #undef INSTANTIATE_PUBLIC_TRACE_FUNCTIONS
 
 namespace js {

js/src/jit-test/lib/match-debugger.js

@@ -5,15 +5,44 @@ if (typeof Match !== 'function') {
 }
 
 class DebuggerObjectPattern extends Match.Pattern {
-  constructor(className) {
+  constructor(className, props) {
     super();
     this.className = className;
+    if (props) {
+      this.props = Match.Pattern.OBJECT_WITH_EXACTLY(props);
+    }
   }
 
   match(actual) {
-    if (!(actual instanceof Debugger.Object) || actual.class !== this.className) {
-      throw new Match.MatchError(`Expected Debugger.Object of class ${this.className}, got ${actual}`);
+    if (!(actual instanceof Debugger.Object)) {
+      throw new Match.MatchError(`Expected Debugger.Object, got ${actual}`);
+    }
+
+    if (actual.class !== this.className) {
+      throw new Match.MatchError(`Expected Debugger.Object of class ${this.className}, got Debugger.Object of class ${actual.class}`);
     }
+
+    if (this.props !== undefined) {
+      const lifted = {};
+      for (const name of actual.getOwnPropertyNames()) {
+        const desc = actual.getOwnPropertyDescriptor(name);
+        if (!('value' in desc)) {
+          throw new Match.MatchError(`Debugger.Object referent has non-value property ${uneval(name)}`);
+        }
+        lifted[name] = desc.value;
+      }
+
+      try {
+        this.props.match(lifted);
+      } catch (inner) {
+        if (!(inner instanceof Match.MatchError)) {
+          throw inner;
+        }
+        inner.message = `matching Debugger.Object referent properties:\n${inner.message}`;
+        throw inner;
+      }
+    }
+
     return true;
   }
 }

js/src/jit-test/tests/debug/Frame-onPop-async-generators-01.js

@@ -38,11 +38,11 @@ collect().then(value => {
   assertDeepEq(value, [2, 4]);
 
   Pattern([
-    EXACT({ return: new DebuggerObjectPattern("Promise") }),
-    EXACT({ return: 2 }),
-    EXACT({ return: new DebuggerObjectPattern("Promise") }),
-    EXACT({ return: 4 }),
-    EXACT({ return: "ok" }),
+    EXACT({ return: new DebuggerObjectPattern("Promise"), await: true }),
+    EXACT({ return: 2, yield: true }),
+    EXACT({ return: new DebuggerObjectPattern("Promise"), await: true }),
+    EXACT({ return: 4, yield: true }),
+    EXACT({ return: "ok", await: true }),
     EXACT({ return: "ok" }),
   ]).assert(log);
 

js/src/jit-test/tests/debug/Frame-onPop-generators-06.js

@@ -23,6 +23,6 @@ g.eval(`
 
 assertDeepEq([... g.f()], [3]);
 Pattern([
-  EXACT({ return: new DebuggerObjectPattern("Object") }),
-  EXACT({ return: new DebuggerObjectPattern("Object") }),
+  EXACT({ return: new DebuggerObjectPattern("Object", { value: 3, done: false }), yield: true }),
+  EXACT({ return: new DebuggerObjectPattern("Object", { value: "ok", done: true }) }),
 ]).assert(log);

js/src/jit-test/tests/debug/Frame-onPop-generators-07.js

@@ -0,0 +1,40 @@
+// Completion values for generators report yields and initial yields.
+
+load(libdir + "asserts.js");
+load(libdir + 'match.js');
+load(libdir + 'match-debugger.js');
+const { Pattern } = Match;
+const { OBJECT_WITH_EXACTLY: X } = Pattern;
+
+const g = newGlobal({ newCompartment: true });
+g.eval(`
+  function* f() {
+    yield "yielding";
+    return "returning";
+  }
+`);
+
+const dbg = new Debugger(g);
+const completions = [];
+dbg.onEnterFrame = frame => {
+  frame.onPop = completion => {
+    completions.push(completion);
+  };
+};
+
+assertDeepEq([... g.f()], ["yielding"]);
+print(uneval(completions));
+Pattern([
+  X({
+    return: new DebuggerObjectPattern("Generator", {}),
+    yield: true,
+    initial: true
+  }),
+  X({
+    return: new DebuggerObjectPattern("Object", { value: "yielding", done: false }),
+    yield: true
+  }),
+  X({
+    return: new DebuggerObjectPattern("Object", { value: "returning", done: true })
+  }),
+]).assert(completions);

js/src/vm/CommonPropertyNames.h

@@ -210,6 +210,7 @@
   MACRO(index, index, "index")                                                 \
   MACRO(infinity, infinity, "infinity")                                        \
   MACRO(Infinity, Infinity, "Infinity")                                        \
+  MACRO(initial, initial, "initial")                                           \
   MACRO(InitializeCollator, InitializeCollator, "InitializeCollator")          \
   MACRO(InitializeDateTimeFormat, InitializeDateTimeFormat,                    \
         "InitializeDateTimeFormat")                                            \