E2E tests

The purpose of this docs is to describe KICS' E2E test suite

Getting Started

There are several ways to execute the E2E tests.

TLDR

This steps will build the kics and then run the E2E using the built binary (placed by default under ${PWD}/bin/kics)

make test-e2e

Step by Step

These steps will build the kics and then run the E2E using the built binary.

go build -o ./bin/kics cmd/console/main.go

If you want to provide a version:

go build -o ./bin/kics -ldflags "-X github.com/Checkmarx/kics/internal/constants.Version=$(git rev-parse --short HEAD) cmd/console/main.go

and then:

E2E_KICS_BINARY=./bin/kics go test "github.com/Checkmarx/kics/e2e" -parallel 12 -v

Test Structure

Test case main structure

var tests = []struct {
	name          string // name of the test case
	args          args // args structure of the test
	wantStatus    int // expected output status code
	removeFiles []string // path for removing files created during the test
	validation    Validation // custom validation function for the test case (usually regex)
}

Test case args structure

type cmdArgs []string

type args struct {
	args            []cmdArgs // args to pass to kics binary
	expectedOut     []string  // path to file with expected output
	expectedPayload []string  // path to file with expected payload
	expectedResult  []string // path to file with expected result
	expectedLog     LogValidation // custom validation structure
}

type LogValidation struct {
	logFile        string // file name of the generated log
	validationFunc Validation // custom validation function to verify the log content
}

Each test can use more or less keys within its structure, depending on the complexity of the test.

A single test can contain verification based on:

• status code only (required for all tests)

• status code + custom regex verification

• status code + CLI output (file check)

• status code + payload content (file check)

• status code + result content (file check)

• status code + log content (file check with validation function)

• status code + result content (file check) + payload content (file check)

• status code + result content (file check) + payload content (file check) + log content (file check with validation function)

• status code + CLI output + payload content (file check) + result content (file check) + custom regex verification ....

Note: All Tests are running in parallel. To avoid errors when creating new E2E tests which contain flags that create files (payload, results, logs...) you should create this output file in a folder called "output" and the name of the generated file must contain the ID of the test case. This will guarantee that files generated by tests will not have the same name, avoiding read/write/remove errors when running several tests in parallel. The Example below (3 and 4) demonstrate E2E tests with file creation (payload/logs)

Test Scenarios

Example 1: A test case that executes a command and checks only the status code output

This type of test is simple and does not require the creation of files.

// E2E-CLI-020 - KICS scan with --exclude-queries flag
// should not run queries that was provided in this flag.
{
	name: "E2E-CLI-020",
	args: args{
		args: []cmdArgs{
			[]string{"scan", "--exclude-queries", "15ffbacc-fa42-4f6f-a57d-2feac7365caa", "-s",
				"-q", "../assets/queries", "-p", "fixtures/samples/terraform-single.tf"},
		},
	},
	wantStatus: []int{0},
}

Example 2: A test case that executes a command and checks:

• Status code output
• Custom validation function (regex)

The custom validation function must perform a validation based on the CLI output generated by the test. The function must return true if the validation is correct and false otherwise.

	// E2E-CLI-006 - KICS generate-id should exhibit
	// a valid UUID in the CLI and return exit code 0
	{
		name: "E2E-CLI-006",
		args: args{
			args: []cmdArgs{
				[]string{"generate-id"},
			},
		},
		wantStatus: 0,
		// When the validation function is used,
		// the expectedOut function can be suppressed

		// The validator function allows the creation of customized functions to
		// perform actions on the output generated by the test.
		validation: func(outputText string) bool {
			// Usually, a validation check for elements present in the output using regex
			uuidRegex := "[a-f0-9]{8}-[a-f0-9]{4}-4{1}[a-f0-9]{3}-[89ab]{1}[a-f0-9]{3}-[a-f0-9]{12}"
			match, _ := regexp.MatchString(uuidRegex, outputText)
			
			// After the custom check, the function should return
			// a boolean indicating if the test has passed or failed.
			return match
		},
	},
}

Example 3: A test case that executes a command and checks:

• Status code output
• Kics CLI output
• Generated payload file content

The Tests that check CLI output, payload or results, need a comparison file for each output you want to compare (except for the status code).

The example below contains 2 files for comparing the outputs of CLI and the payload file: "E2E_CLI_005" and "E2E_CLI_005_PAYLOAD.json". Files used for comparing outputs must have the same name as the test and must be added to the "fixture" folder.

In addition, it is necessary to remove the files that will be generated during the test by adding them to removeFiles (required only for test cases that generates files from results/payloads/logs). Files created by Kics during testing should always be created in the 'outputs' folder

// E2E-CLI-005 - KICS scan with -- payload-path flag should create a file with the
// passed name containing the payload of the files scanned
{
	name: "E2E-CLI-005",
	args: args{
		args: []cmdArgs{
			[]string{"scan", "--silent", "-q", "../assets/queries", "-p", "fixtures/samples/terraform.tf",
				"--payload-path", "output/E2E_CLI_005_PAYLOAD.json"},
		},
		expectedOut: []string{
			"E2E_CLI_005",
		},
		expectedPayload: []string{
			"E2E_CLI_005_PAYLOAD.json",
		},
	},
	wantStatus:  []int{50},
	removeFiles: []string{"E2E_CLI_005_PAYLOAD.json"},
}

Example 4: A test case that executes a command and checks:

• Status code output
• Generated log file content

In contrast to what is expected for results and payload, log files don't have a file to compare, instead KICS validates logs like it validates CLI's output. In this case, the validation function will receive the log file data as its input and you can use any regex or do text validations in the log file content.

The example below generates an output file called: "E2E_CLI_038_LOG" and creates a validation function expecting 3 matches in the generated log file content.

In addition, it is necessary to remove files generated during the test, adding them to removeFiles (required only for test cases that generates files from results/payloads/logs).

// E2E-CLI-038 - KICS scan command with --log-path
// should generate and save a log file for the scan
{
	name: "E2E-CLI-038",
	args: args{
		args: []cmdArgs{

			[]string{"scan", "--log-path", "output/E2E_CLI_038_LOG",
				"-q", "../assets/queries", "-p", "fixtures/samples/terraform-single.tf"},
		},

		expectedLog: LogValidation{
			logFile: "E2E_CLI_038_LOG",
			validationFunc: func(logText string) bool {
				match1, _ := regexp.MatchString("Scanning with Keeping Infrastructure as Code Secure", logText)
				match2, _ := regexp.MatchString(`Files scanned: \d+`, logText)
				match3, _ := regexp.MatchString(`Queries loaded: \d+`, logText)
				return match1 && match2 && match3
			},
		},
	},
	wantStatus:  []int{40},
	removeFiles: []string{"E2E_CLI_038_LOG"},
}

E2E tests are skiped in short mode:

func Test_E2E_CLI(t *testing.T) {
	kicsPath := getKICSBinaryPath("")

	if testing.Short() {
		t.Skip("skipping E2E tests in short mode.")
	}
//...
}

Test Functions (utils.go)

getKICSBinaryPath: This function gets the path of the kics executable.

runCommand: This function is intended to run kics with the input arguments sent by the test

readFixture & readFile: These functions reads a file (from its folder and name). The folders used are always: fixtures or output.

prepareExpected: This function prepares the file to be compared, avoiding errors in reading and formatting.

checkLine & checkJSONLog: These functions are used to check the Kics output generated in the CLI and compare it with the expectedOutput file.

fileCheck & setFields: TThese functions read and compare the expected files with the files provided during the test. This function compares files from expectedPayload and expectedResult.