Whitewater Detect Secrets

About

The purpose of the project is to detecting secrets within a code base. This is a fork of detect-secrets from yelp. This include more detection, some of which are unique for IBM. Additional features to help integrate with services IBM uses.

detect-secrets is an aptly named module for (surprise, surprise) detecting secrets within a code base.

However, unlike other similar packages that solely focus on finding secrets, this package is designed with the enterprise client in mind: providing a backwards compatible, systematic means of:

1. Preventing new secrets from entering the code base,
2. Detecting if such preventions are explicitly bypassed, and
3. Providing a checklist of secrets to roll, and migrate off to a more secure storage.

This way, you create a separation of concern: accepting that there may currently be secrets hiding in your large repository (this is what we refer to as a baseline), but preventing this issue from getting any larger, without dealing with the potentially gargantuous effort of moving existing secrets away.

It does this by running periodic diff outputs against heuristically crafted regex statements, to identify whether any new secret has been committed. This way, it avoids the overhead of digging through all git history, as well as the need to scan the entire repository every time.

For a look at recent changes, please see the changelog.

User Guide

If you are looking for information on how to use this project as an end user please refer to the user guide.

Contribution

Please read the CONTRIBUTING.md. Bellow is information on how setup the testing environment, and run the tests.

Testing

To run the tests you need install the dependencies described bellow.

You need to run the setup once or after you do a make clean. To run the setup run the following command:

make setup

To run the tests run:

make test

If you want to clean you environment, if you have a bad setup or tests, just run:

make clean

Testing Dependencies

This project is written in Python. Here are the dependencies needed to run the tests:

• python The version can be installed using an utility like pyenv ( instructions bellow ) or your os package manager
  • 2.7
  • 3.5
  • 3.6
  • pypy
• tox installed via pip or your os package manager
• make

Installing via pyenv

1. Install pyenv in your environment. Note: you need to add the environment to you .bashrc. You will most likely run into the common build problems listed here.
2. Install the environment listed above
3. Set the environment as global using the pyenv global $VERSION command
4. Install tox pip install tox

Running test in a docker image

If you don't want to figure out how to install it locally or don't want to spend the time you can use the development docker image. Install docker and docker-compose. Then run:

docker-compose build test && docker-compose run --rm test

Plugins

Each of the checks are developed as plugins in the detect_secrets/plugins directory. Each plugin represent a single test or a group of tests. The following is a list of the currently developed plugins:

The current heuristic searches we implement out of the box include:

• Base64HighEntropyString: checks for all strings matching the Base64 character set, and alerts if their Shannon entropy is above a certain limit.

• HexHighEntropyString: checks for all strings matching the Hex character set, and alerts if their Shannon entropy is above a certain limit.

• PrivateKeyDetector: checks to see if any private keys are committed.

• BasicAuthDetector: checks to see if BasicAuth is used e.g. https://username:[email protected]

• KeywordDetector: checks to see if certain keywords are being used e.g. password or secret

• ArtifactoryDetector: checks to see if Artifactory credentials are present.

• GheDetector: checks to see if GitHub credentials are present.

IBM versioning and rebase guide

• update.md