Specmatic backward-compatibility check does not detect changes in included files

[harikrishnan83]

Discussed in #1643

^{Originally posted by mickle-ak February 27, 2025}
Specmatic's "backward-compatibility-check" does not detect changes in included files if the "root" file is not modified.

=== changes in included files only

Given: The OAS file "OpenApi.yaml" includes references to other files (e.g., $ref: 'data.yaml#/schemas/Task'):

     ... 
     content:
       application/json:
         schema:
           $ref: 'data.yaml#/schemas/Task'
     ... 

When: A breaking change is made to the schema in the included file "data.yaml", while no changes are made in the "root" file "OpenApi.yaml".
And: The Specmatic "backward-compatibility-check" is executed:

java -jar specmatic.jar backward-compatibility-check

Then: Specmatic does not detect the changes and outputs the following:

Specmatic Version: 2.5.0
No specs were changed, skipping the check.
backwardCompatitbilityTest() has SUCCEEDED

=== changes in included files AND in main OAS file
same as above plus

When: Any change is made in the "root" file "OpenApi.yaml" (even adding a whitespace is sufficient).
And: The Specmatic "backward-compatibility-check" is executed:

java -jar specmatic.jar backward-compatibility-check

Then: Specmatic detects and reports the breaking change:

Specmatic Version: 2.5.0
Checking backward compatibility of the following specs:

• Specs that have changed:
  1. OpenApi.yaml
     ...
     Verdict for spec OpenApi.yaml:
     (INCOMPATIBLE) The changes to the spec are NOT backward compatible with the corresponding spec from HEAD
     ...

Suggestion:
Since it is quite common in real-world scenarios to split OAS into multiple files, I believe Specmatic should also detect changes in included files and validate the entire API for backward compatibility.

[harikrishnan83]

#1660 may be a related question