Let Bazelisk correctly separate releases and metadata from Bazel forks.

This fixes an issue where Bazelisk would accidentally reuse a cached binary or "latest releases" JSON from fork A, even when the version label refers fork B.

Also make the documentation a bit clearer and remove outdated info from it.

README.md

@@ -2,78 +2,76 @@
 
 **A user-friendly launcher for Bazel.**
 
-## News
-
-- 2018-01-20: Bazelisk is rewritten in Go. It has the same features as the Python version and both versions are tested against the same integration test suite. This version might be easier to use on Windows, because it can be compiled to a native executable that has no other dependencies.
-
 ## About Bazelisk
 
-Bazelisk is a wrapper for Bazel. It automatically picks a good version of Bazel given your current working directory, downloads it from the official server (if required) and then transparently passes through all command-line arguments to the real Bazel binary. You can call it just like you would call Bazel.
+Bazelisk is a wrapper for Bazel written in Go.
+It automatically picks a good version of Bazel given your current working directory, downloads it from the official server (if required) and then transparently passes through all command-line arguments to the real Bazel binary.
+You can call it just like you would call Bazel.
 
 Some ideas how to use it:
-- Install it as the `bazel` binary in your PATH (e.g. /usr/local/bin). Never worry about upgrading Bazel to the latest version again.
-- Check it into your repository and recommend users to build your software via `./bazelisk.py build //my:software`. That way, even someone who has never used Bazel or doesn't have it installed can build your software.
-- As a company using Bazel or as a project owner, add a `.bazelversion` file to your repository. This will tell Bazelisk to use the exact version specified in the file when running in your workspace. The fact that it's versioned inside your repository will then allow for atomic upgrades of Bazel including all necessary changes. If you install Bazelisk as `bazel` on your CI machines, too, you can even test Bazel upgrades via a normal presubmit / pull request. It will also ensure that users will not try to build your project with an incompatible version of Bazel, which is often a cause for frustration and failing builds.
-
-## How does Bazelisk know which fork and version to run?
+- Install it as the `bazel` binary in your PATH (e.g. /usr/local/bin).
+  Never worry about upgrading Bazel to the latest version again.
+- Check it into your repository and recommend users to build your software via `./bazelisk build //my:software`.
+  That way, even someone who has never used Bazel or doesn't have it installed can build your software.
+- As a company using Bazel or as a project owner, add a `.bazelversion` file to your repository.
+  This will tell Bazelisk to use the exact version specified in the file when running in your workspace.
+  The fact that it's versioned inside your repository will then allow for atomic upgrades of Bazel including all necessary changes.
+  If you install Bazelisk as `bazel` on your CI machines, too, you can even test Bazel upgrades via a normal presubmit / pull request.
+  It will also ensure that users will not try to build your project with an incompatible version of Bazel, which is often a cause for frustration and failing builds.
+
+Before Bazelisk was rewritten in Go, it was a Python script.
+This still works and has the advantage that you can run it on any platform that has a Python interpreter, but is currently unmaintained and it doesn't support as many features.
+The documentation below describes the newer Go version only.
+
+## How does Bazelisk know which Bazel version to run and where to get it from?
 
 It uses a simple algorithm:
-- If the environment variable `USE_BAZEL_VERSION` is set, it will use the fork and version specified in the value.
-- Otherwise, if a `.bazelversion` file exists in the current directory or recursively any parent directory, it will read the file and use the fork and version specified in it.
-- Otherwise it will use the official release from `bazelbuild/bazel` and check GitHub for the latest version of Bazel, cache the result for an hour and use that version.
+- If the environment variable `USE_BAZEL_VERSION` is set, it will use the version specified in the value.
+- Otherwise, if a `.bazelversion` file exists in the current directory or recursively any parent directory, it will read the file and use the version specified in it.
+- Otherwise it will use the official latest Bazel release.
 
-Bazelisk currently follows the release convention on `bazelbuild/bazel` to build the URL. The URL format looks like `https://github.com/<FORK>/bazel/releases/download/<VERSION>/<FILENAME>`.
-
-The fork and version should be separated by slash `<FORK>/<VERSION>`:
-- If the fork and version are `foobar/0.28.0` and the platform is `linux`, the URL will be `https://github.com/foobar/bazel/releases/download/0.28.0/bazel-0.28.0-linux-x86_64`.
-- If the version is not provided `foobar/`, it uses the latest version.
-- If the fork is not provided `/0.28.0`, it uses the official release from `bazelbuild/bazel`.
-- If no `/` is present `0.28.0`, the value is treated as version and it uses the official release from `bazelbuild/bazel`.
+A version can optionally be prefixed with a fork name.
+The fork and version should be separated by slash: `<FORK>/<VERSION>`.
+If you want to create a fork with your own releases, you have to follow the naming conventions that we use in `bazelbuild/bazel` for the binary file names.
+The URL format looks like `https://github.com/<FORK>/bazel/releases/download/<VERSION>/<FILENAME>`.
 
 Bazelisk currently understands the following formats for version labels:
-- `latest` means the latest stable version of Bazel as released on GitHub. Previous
-  releases can be specified via `latest-1`, `latest-2` etc.
-- A version number like `0.17.2` means that exact version of Bazel. It can also
-  be a release candidate version like `0.20.0rc3`.
-- `last_green` refers to the Bazel binary that was built at the most recent commit that passed [Bazel CI](https://buildkite.com/bazel/bazel-bazel). Ideally this binary should be very close to Bazel-at-head.
+- `latest` means the latest stable version of Bazel as released on GitHub.
+  Previous releases can be specified via `latest-1`, `latest-2` etc.
+- A version number like `0.17.2` means that exact version of Bazel.
+  It can also be a release candidate version like `0.20.0rc3`.
+
+Additionally, a few special version names are supported for our official releases only (these formats do not work when using a fork):
+- `last_green` refers to the Bazel binary that was built at the most recent commit that passed [Bazel CI](https://buildkite.com/bazel/bazel-bazel).
+  Ideally this binary should be very close to Bazel-at-head.
 - `last_downstream_green` points to the most recent Bazel binary that builds and tests all [downstream projects](https://buildkite.com/bazel/bazel-at-head-plus-downstream) successfully.
-- `last_rc` points to the most recent release candidate. If there is no active release candidate, Bazelisk uses the latest Bazel release instead. Currently only the Go version of Bazelisk supports this value.
-
-These formats are not supported on a fork.
-
-In the future we will add support for building Bazel from source at a given commit.
+- `last_rc` points to the most recent release candidate.
+  If there is no active release candidate, Bazelisk uses the latest Bazel release instead.
 
 ## Other features
 
 The Go version of Bazelisk offers two new flags.
 
-`--strict` expands to the set of incompatible flags which may be enabled for the
-given version of Bazel.
+`--strict` expands to the set of incompatible flags which may be enabled for the given version of Bazel.
 
 ```shell
 bazelisk --strict build //...
 ```
 
-`--migrate` will run Bazel multiple times to help you identify compatibility
-issues. If the code fails with `--strict`, the flag `--migrate` will run Bazel
-with each one of the flag separately, and print a report at the end. This will
-show you which flags can safely enabled, and which flags require a migration.
+`--migrate` will run Bazel multiple times to help you identify compatibility issues.
+If the code fails with `--strict`, the flag `--migrate` will run Bazel with each one of the flag separately, and print a report at the end.
+This will show you which flags can safely enabled, and which flags require a migration.
 
-You can set `BAZELISK_GITHUB_TOKEN` to set a GitHub access token to use for API
-requests to avoid rate limiting when on shared networks.
+You can set `BAZELISK_GITHUB_TOKEN` to set a GitHub access token to use for API requests to avoid rate limiting when on shared networks.
 
-You can set `BAZELISK_SHUTDOWN` to run `shutdown` between builds when
-migrating if you suspect this affects your results.
+You can set `BAZELISK_SHUTDOWN` to run `shutdown` between builds when migrating if you suspect this affects your results.
 
-You can set `BAZELISK_CLEAN` to run `clean --expunge` between builds when
-migrating if you suspect this affects your results.
+You can set `BAZELISK_CLEAN` to run `clean --expunge` between builds when migrating if you suspect this affects your results.
 
-If `tools/bazel` exists in your workspace root and is executable, Bazelisk will run this file,
-instead of the Bazel version it downloaded. It will set the environment variable `BAZEL_REAL` to
-the path of the downloaded Bazel binary. This can be useful, if you have a wrapper script that e.g.
-ensures that environment variables are set to known good values. This behavior can be disabled by
-setting the environment variable `BAZELISK_SKIP_WRAPPER` to any value (except the empty string)
-before launching Bazelisk.
+If `tools/bazel` exists in your workspace root and is executable, Bazelisk will run this file, instead of the Bazel version it downloaded.
+It will set the environment variable `BAZEL_REAL` to the path of the downloaded Bazel binary.
+This can be useful, if you have a wrapper script that e.g. ensures that environment variables are set to known good values.
+This behavior can be disabled by setting the environment variable `BAZELISK_SKIP_WRAPPER` to any value (except the empty string) before launching Bazelisk.
 
 ## Releases
 
@@ -83,7 +81,8 @@ Binary and source releases are provided on our [Releases](https://github.com/baz
 
 For ease of use, the Python version of Bazelisk is written to work with Python 2.7 and 3.x and only uses modules provided by the standard library.
 
-The Go version can be compiled to run natively on Linux, macOS and Windows. You need at least Go 1.11 to build Bazelisk, otherwise you'll run into errors like `undefined: os.UserCacheDir`.
+The Go version can be compiled to run natively on Linux, macOS and Windows.
+You need at least Go 1.11 to build Bazelisk, otherwise you'll run into errors like `undefined: os.UserCacheDir`.
 
 To install the Go version, type:
 
@@ -97,16 +96,17 @@ To add it to your PATH:
 export PATH=$PATH:$(go env GOPATH)/bin
 ```
 
-For more information, you may read about the [`GOPATH` environment
-variable](https://github.com/golang/go/wiki/SettingGOPATH).
+For more information, you may read about the [`GOPATH` environment variable](https://github.com/golang/go/wiki/SettingGOPATH).
 
 ## Ideas for the future
 
 - Add support for checked-in Bazel binaries.
 - When the version label is set to a commit hash, first download a matching binary version of Bazel, then build Bazel automatically at that commit and use the resulting binary.
-- Add support to automatically bisect a build failure to a culprit commit in Bazel. If you notice that you could successfully build your project using version X, but not using version X+1, then Bazelisk should be able to figure out the commit that caused the breakage and the Bazel team can easily fix the problem.
+- Add support to automatically bisect a build failure to a culprit commit in Bazel.
+  If you notice that you could successfully build your project using version X, but not using version X+1, then Bazelisk should be able to figure out the commit that caused the breakage and the Bazel team can easily fix the problem.
 
 ## FAQ
 
 ### Where does Bazelisk store the downloaded versions of Bazel?
-It creates a directory called "bazelisk" inside your [user cache directory](https://golang.org/pkg/os/#UserCacheDir) and will store them there. Feel free to delete this directory at any time, as it can be regenerated automatically when required.
+It creates a directory called "bazelisk" inside your [user cache directory](https://golang.org/pkg/os/#UserCacheDir) and will store them there.
+Feel free to delete this directory at any time, as it can be regenerated automatically when required.

bazelisk.go

@@ -41,7 +41,7 @@ const (
 	bazelReal      = "BAZEL_REAL"
 	skipWrapperEnv = "BAZELISK_SKIP_WRAPPER"
 	wrapperPath    = "./tools/bazel"
-	bazelUpstream  = "BAZEL_UPSTREAM"
+	bazelUpstream  = "bazelbuild"
 )
 
 var (
@@ -121,13 +121,6 @@ func parseBazelForkAndVersion(bazelForkAndVersion string) (string, string, error
 		return "", "", fmt.Errorf("invalid version \"%s\", could not parse version with more than one slash", bazelForkAndVersion)
 	}
 
-	if len(bazelFork) == 0 {
-		bazelFork = bazelUpstream
-	}
-	if len(bazelVersion) == 0 {
-		bazelVersion = "latest"
-	}
-
 	return bazelFork, bazelVersion, nil
 }
 
@@ -194,11 +187,11 @@ func maybeDownload(bazeliskHome, url, filename, description string) ([]byte, err
 	return body, nil
 }
 
-func resolveLatestVersion(bazeliskHome string, offset int) (string, error) {
-	url := "https://api.github.com/repos/bazelbuild/bazel/releases"
-	releasesJSON, err := maybeDownload(bazeliskHome, url, "releases.json", "list of Bazel releases from GitHub")
+func resolveLatestVersion(bazeliskHome, bazelFork string, offset int) (string, error) {
+	url := fmt.Sprintf("https://api.github.com/repos/%s/bazel/releases", bazelFork)
+	releasesJSON, err := maybeDownload(bazeliskHome, url, bazelFork+"-releases.json", "list of Bazel releases from github.com/"+bazelFork)
 	if err != nil {
-		return "", fmt.Errorf("could not get releases from GitHub: %v", err)
+		return "", fmt.Errorf("could not get releases from github.com/%s/bazel: %v", bazelFork, err)
 	}
 
 	var releases []release
@@ -303,24 +296,29 @@ func getHighestRcVersion(versions []string) (string, error) {
 	return fmt.Sprintf("%src%d", version, lastRc), nil
 }
 
-func resolveVersionLabel(bazeliskHome, bazelVersion string) (string, bool, error) {
-	// Returns three values:
-	// 1. The label of a Blaze release (if the label resolves to a release) or a commit (for unreleased binaries),
-	// 2. Whether the first value refers to a commit,
-	// 3. An error.
-	lastGreenCommitPathSuffixes := map[string]string{"last_green": "github.com/bazelbuild/bazel.git/bazel-bazel", "last_downstream_green": "downstream_pipeline"}
-	if pathSuffix, ok := lastGreenCommitPathSuffixes[bazelVersion]; ok {
-		commit, err := getLastGreenCommit(pathSuffix)
-		if err != nil {
-			return "", false, fmt.Errorf("cannot resolve last green commit: %v", err)
+func resolveVersionLabel(bazeliskHome, bazelFork, bazelVersion string) (string, bool, error) {
+	if bazelFork == bazelUpstream {
+		// Returns three values:
+		// 1. The label of a Blaze release (if the label resolves to a release) or a commit (for unreleased binaries),
+		// 2. Whether the first value refers to a commit,
+		// 3. An error.
+		lastGreenCommitPathSuffixes := map[string]string{
+			"last_green":            "github.com/bazelbuild/bazel.git/bazel-bazel",
+			"last_downstream_green": "downstream_pipeline",
 		}
+		if pathSuffix, ok := lastGreenCommitPathSuffixes[bazelVersion]; ok {
+			commit, err := getLastGreenCommit(pathSuffix)
+			if err != nil {
+				return "", false, fmt.Errorf("cannot resolve last green commit: %v", err)
+			}
 
-		return commit, true, nil
-	}
+			return commit, true, nil
+		}
 
-	if bazelVersion == "last_rc" {
-		version, err := resolveLatestRcVersion()
-		return version, false, err
+		if bazelVersion == "last_rc" {
+			version, err := resolveLatestRcVersion()
+			return version, false, err
+		}
 	}
 
 	r := regexp.MustCompile(`^latest(?:-(?P<offset>\d+))?$`)
@@ -335,7 +333,7 @@ func resolveVersionLabel(bazeliskHome, bazelVersion string) (string, bool, error
 				return "", false, fmt.Errorf("invalid version \"%s\", could not parse offset: %v", bazelVersion, err)
 			}
 		}
-		version, err := resolveLatestVersion(bazeliskHome, offset)
+		version, err := resolveLatestVersion(bazeliskHome, bazelFork, offset)
 		return version, false, err
 	}
 
@@ -395,9 +393,9 @@ func determineURL(fork string, version string, isCommit bool, filename string) s
 
 	if fork == bazelUpstream {
 		return fmt.Sprintf("https://releases.bazel.build/%s/%s/%s", version, kind, filename)
-	} else {
-		return fmt.Sprintf("https://github.com/%s/bazel/releases/download/%s/%s", fork, version, filename)
 	}
+
+	return fmt.Sprintf("https://github.com/%s/bazel/releases/download/%s/%s", fork, version, filename)
 }
 
 func downloadBazel(fork string, version string, isCommit bool, directory string) (string, error) {
@@ -698,12 +696,12 @@ func main() {
 		log.Fatalf("could not parse Bazel fork and version: %v", err)
 	}
 
-	resolvedBazelVersion, isCommit, err := resolveVersionLabel(bazeliskHome, bazelVersion)
+	resolvedBazelVersion, isCommit, err := resolveVersionLabel(bazeliskHome, bazelFork, bazelVersion)
 	if err != nil {
 		log.Fatalf("could not resolve the version '%s' to an actual version number: %v", bazelVersion, err)
 	}
 
-	bazelDirectory := filepath.Join(bazeliskHome, "bin")
+	bazelDirectory := filepath.Join(bazeliskHome, "bin", bazelFork)
 	err = os.MkdirAll(bazelDirectory, 0755)
 	if err != nil {
 		log.Fatalf("could not create directory %s: %v", bazelDirectory, err)

test.sh

@@ -0,0 +1,15 @@
+#!/bin/bash
+
+set -euxo pipefail
+
+rm -rf "$HOME/Library/Caches/bazelisk"
+env -u USE_BAZEL_VERSION ./bin/bazelisk-darwin-amd64 version
+USE_BAZEL_VERSION="latest" ./bin/bazelisk-darwin-amd64 version
+USE_BAZEL_VERSION="0.28.0" ./bin/bazelisk-darwin-amd64 version
+USE_BAZEL_VERSION="last_green" ./bin/bazelisk-darwin-amd64 version
+USE_BAZEL_VERSION="last_downstream_green" ./bin/bazelisk-darwin-amd64 version
+USE_BAZEL_VERSION="last_rc" ./bin/bazelisk-darwin-amd64 version
+USE_BAZEL_VERSION="bazelbuild/latest" ./bin/bazelisk-darwin-amd64 version
+USE_BAZEL_VERSION="bazelbuild/0.27.0" ./bin/bazelisk-darwin-amd64 version
+USE_BAZEL_VERSION="philwo/latest" ./bin/bazelisk-darwin-amd64 version
+USE_BAZEL_VERSION="philwo/0.25.0" ./bin/bazelisk-darwin-amd64 version