Add tags to individual interactions

[monch1962]

It'd be really handy to have a tags field added to individual Pacts, broadly as follows:

{ ...
  "request": ...,
  "response": ...,
  "tags": ["regression", "JIRA-123", "performance", ...],
  ...
}

A lot of the time I don't need to execute every Pact to test a change, and I want to reuse my Pacts for purposes other than integration testing. This would allow us to select specific subsets of Pacts to execute in certain circumstances. For example, if we want to retest JIRA-123, the above Pact would be one of potentially several selected; if we don't want a full regression run, we might want to exclude this test; if we want this test to be reused as a performance test case, then we can auto-generate JMeter/K6 tests from the Pact, and so on. For negative testing, I might want to tag a Pact as "fuzzable", so I can use security tools to fuzz different fields in the request and ensure that I always get a HTTP 4xx status being returned (rather than a 2xx or 5xx).
Tags themselves should be arbitrarily-named (i.e. not a predefined list), and support multiple tags for each Pact.
Broker could probably add support for this pretty easily, and it'd add a lot of flexibility to how people can use Pact. In particular, it'd let us adapt Pact to our ideal workflows, rather than adapt those workflows to support Pact

[bethesque]

Hi @monch1962 can you explain why the tags on the version resources don't support your current needs?

[bethesque]

Ooh, you mean, add tags to the interaction. (An interaction is a request/response pair, and a "pact" is a collection of interactions.)

Yes, that makes more sense. Should it specifically be tags? I'm wary of overloading the term. What about a user defined metadata hash? People could put anything they liked in there.

[mefellows]

This kind of relates to #72, but with a different purpose in mind. On the one hand, I can see the contract specification bloating for these sorts of things, but on the other absolutely recognise that both needs are valid.

What prior art could inspire us here (thinking JSONSchema, OAS, GraphQL etc)?

In a way, having a user-defined metadata section could be good, as long as you can allow querying / navigating nested items.

One complication of such an approach would be that you couldn't reliably depend on the properties for certain use cases such as making it render usefully in a Pact Broker or building generic tooling around.

For example, as powerful as tags on version resources are, they prevent us ever really "knowing" what the real contract is (because real is a tag defined by the user).

But pondering aside, I like the idea.

For negative testing, I might want to tag a Pact as "fuzzable", so I can use security tools to fuzz different fields in the request and ensure that I always get a HTTP 4xx status being returned (rather than a 2xx or 5xx).

On a side note, this and related use cases very much interests me. If you're keen on elaborating this offline (well, offline to this ticket in slack perhaps) I'd love to hear it. I can see an argument to help support an ecosystem around this.

[monch1962]

Ooh, you mean, add tags to the interaction. (An interaction is a request/response pair, and a "pact" is a collection of interactions.)

Yes, that makes more sense. Should it specifically be tags? I'm wary of overloading the term. What about a user defined metadata hash? People could put anything they liked in there.

Hi Beth, yeah the term "tag" is definitely overloaded but it sounds like you understand what I'm asking for. Since you've brought it up, I'm interested in making more use of "tag-like things" for both the individual interaction and the pact. More details in my response to Matt (see below).

[monch1962]

For negative testing, I might want to tag a Pact as "fuzzable", so I can use security tools to fuzz different fields in the request and ensure that I always get a HTTP 4xx status being returned (rather than a 2xx or 5xx).

On a side note, this and related use cases very much interests me. If you're keen on elaborating this offline (well, offline to this ticket in slack perhaps) I'd love to hear it. I can see an argument to help support an ecosystem around this.

Hi Matt, we've already developed a bunch of bash/jq and Powershell scripts to mutate existing API tests for security and general "solution robustness" testing. For example:

• adding in a simple extra header,

• adding in an additional huge header (e.g. 20k bytes),

• adding a header containing unprintable characters,

• changing a numeric header value to a string or vice-versa,

• adding extra fields to JSON payloads

and so on. The intent is that the system under test should produce sane responses in this situation, and "sane" is usually application-specific.

In addition to generating a set of negative tests for the app itself, this approach also assists in testing web server config, intrusion detection config, etc. via CI.

We already do quite a lot of automated testing in this space, but not (...yet...) with Pact and I can't share all the details here. As you might imagine, we find some fairly obvious problems this way, but also some pretty nasty security holes that may not get fixed in production... Happy to discuss in a not-so-public forum though.

It'd be very handy to be able to do this to pacts (or individual interactions as per Beth's comment above), but to do so we need the ability to identify which pacts or interactions are suitable for this sort of mutation. That's where being able to add user-defined tags would come in handy.

Just to be clear, I'm not suggesting building the capability to mutate tests in this fashion inside Pact itself - just add some way of identifying & select suitable tests so these mutations can be applied outside of Pact (via Powershell, jq etc.)

[mefellows]

Thanks for the detail @monch1962! It all makes sense to me.

[bethesque]

Maybe we could call them "interaction labels" to avoid confusion.

[uglyog]

I like labels, it indicates they don't have any behaviour associated with them. I've already implemented comments #45, we can now just add these to that structure.

[mefellows]

Is this item done Ron? The v4 tracking item suggests it's added to JVM and Rust, but the comment above suggests it's not.

Currently discussing filtering options with somebody (see https://pact-foundation.slack.com/archives/C5F4KFKR8/p1722598892948679) and this sprang to mind as a useful way to address their problem (as per the OP here).

[rholshausen]

No, it was not done. I don't know why.

[rholshausen]

Oh, the comments structure was proposed to be re-used instead of adding a new attribute.

[mefellows]

Are you suggesting something like this JUnit DSL in Test (adapted from your comment in #45):

      .given("good state")
      .comment("This is a comment")
      .label("label-x")
      .uponReceiving("V3 PactProviderTest test interaction")
      .path("/")
      .method("GET")
      .comment("Another comment")
      .willRespondWith()
      .status(200)
      .body("{\"responsetest\": true, \"version\": \"v3\"}")
      .comment("This is also a comment")
      .toPact(V4Pact.class);

Producing pact file contents:

{
 "interactions": [
    {
      "comments": {
        "testname": "runTest(au.com.dius.pact.consumer.junit.v4.V4HttpPactTest)",
        "labels": ["label-x"]
        "text": [
          "This is a comment",
          "Another comment",
          "This is also a comment"
        ]
      },
      "description": "V3 PactProviderTest test interaction"
}

[rholshausen]

Yes, that is one way to do it.