Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Collection Runner and Newman: a request that got no response does not abort the collection, no way to do so programmatically #1898

Open
jhardin-aptos opened this issue Feb 5, 2019 · 5 comments

Comments

@jhardin-aptos
Copy link

I have a fairly large test collection that I'm working on right now. This morning I fired it off in Newman with HTML output against my dev environment and... it just sat there.

Disabling HTML output showed pretty clearly that the webservice wasn't responding - oops, forgot to start it up. D'oh! :(

So I've been digging around in the Postman docs to see how to catch this and do postman.setNextRequest(null); to abort the collection, and... there's no programmatic way to capture "no response" - and getting no response to a request (obviously) does not automatically abort the collection run.

This isn't so bad in Collection Runner because that's interactive and the user can abort it easily enough, but Newman is probably running the tests unattended.

To Reproduce

  1. Open a multi-request test collection
  2. Point it at a webservice that will not respond
  3. Run it using Newman
  4. The first request gets "no response" (e.g. ECONNREFUSED)
  5. Subsequent requests still execute

Expected behavior
Either:

  1. The collection should abort (but I can see reasons to not do this automatically)
    or
  2. Expose some way to control the collection behavior on a request failure

Newman does have the -bail failure option, but that's too broad - I do want to run multiple tests and verify their results and see all the failed test conditions; bailing on the first test failure isn't desirable.

Perhaps -bail needs a request_failure option too?

Perhaps (also?) there needs to be an option to run a javascript snippet if the request fails. This would be more flexible, but would be a more-intrusive change to both Newman and Postman...

App information:
Newman 4.3.1 on Windows 10

@jhardin-aptos
Copy link
Author

I attempted this to abort the collection promptly if the service is not responding:

  1. Have the first request be a "warm the service" request with a simple lightweight request of the service
  2. In that request's pre-request script do postman.setNextRequest(null);
  3. In that request's tests script do postman.setNextRequest(2);

...but it doesn't work, the collection always terminates after the first request. Apparently only the first call to setNextRequest() is respected.

Can we please get some clear documentation of how setNextRequest() behaves? Details to clarify:

  1. When does it take effect? Immediately, or at the end of the processing of the current request? (I had to figure this out by trial-and-error...)
  2. The problem illustrated by the above: can you override a setNextRequest() with a subsequent one? Or is only the first one that is executed in a given request effective? (I had to figure this out by trial-and-error, too)

Note: answering those questions here in issue comment discussion is not documentation.

@jhardin-aptos jhardin-aptos changed the title Newman: a request that got no response does not abort the collection, no way to do so programmatically Collection Runner and Newman: a request that got no response does not abort the collection, no way to do so programmatically Feb 8, 2019
@shamasis shamasis transferred this issue from postmanlabs/postman-app-support Feb 18, 2019
@shamasis
Copy link
Member

Moved it to newman and would make part of release that tackles better execution control.

  • allowing failed requests to still run test script
  • option to bail on request error

@govind794
Copy link

I was able to break the flow using below code. Write it in the test script of first api, if api response returns success = false, it would break the flow.
pm.test("API response", function() {
var body = pm.response.json();
let identical = _.isEqual(body.success, true);
if (identical === false) {
postman.setNextRequest(null);
}
});

@spenceclark
Copy link

Is this issue still open?

I have a request very early on in my collection that checks to make sure a dependancy is up.

My test looks like this:

`// Check for success
pm.test("Status code is 200", function () {
pm.response.to.have.status(200);
});

// Parse response
var resp = pm.response.json();

pm.test("Maintenance mode = false", function () {
pm.expect(resp.inMaintenanceMode).to.be.eql(false);
});

if (resp.inMaintenanceMode === true)
postman.setNextRequest(null);`

When I run via Postman app - it stops at that point if the flag is true.
However the exact same pack run via Newman carries on to the next request.

@spenceclark
Copy link

Is this issue still open?

I have a request very early on in my collection that checks to make sure a dependancy is up.

My test looks like this:

`// Check for success
pm.test("Status code is 200", function () {
pm.response.to.have.status(200);
});

// Parse response
var resp = pm.response.json();

pm.test("Maintenance mode = false", function () {
pm.expect(resp.inMaintenanceMode).to.be.eql(false);
});

if (resp.inMaintenanceMode === true)
postman.setNextRequest(null);`

When I run via Postman app - it stops at that point if the flag is true.
However the exact same pack run via Newman carries on to the next request.

Actually no, this does seem to be working as expected in Newman when i tried again today; it must have been something I did wrong.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

5 participants