Skip to content

Commit

Permalink
feat: Optionally Limit the number of concurrent go routines via Sized…
Browse files Browse the repository at this point in the history 
…WaitGroup (gruntwork-io#38)
  • Loading branch information
@jphuynh
jphuynh authored May 10, 2021
1 parent d2d6079 commit e9ce453
Show file tree
Hide file tree
Showing 8 changed files with 32 additions and 14 deletions.
21 changes: 11 additions & 10 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -324,16 +324,17 @@ echo "gruntwork-io/terragrunt gruntwork-io/terratest" | git-xargs \
`git-xargs` exposes several flags that allow you to customize its behavior to better suit your needs. For the latest info on flags, you should run `git-xargs --help`. However, a couple of the flags are worth explaining more in depth here:
| Flag | Description | Type | Required |
| ---------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------- | -------- |
| `--branch-name` | You must specify the name of the branch to make your local and remote changes on. You can further control branching behavior via `--skip-pull-requests` as explained below | String | Yes |
| `--repos` | If you want to specify many repos and manage them in files (which makes batching and testing easier) then use this flag to pass the filepath to a repos file. See [the repos file format](#option-2-flat-file-of-repository-names) for more information | String | No |
| `--repo` | Use this flag to specify a single repo, e.g., `--repo gruntwork-io/cloud-nuke`. Can be passed multiple times to target several repos | String | No |
| `--github-org` | If you want to target every repo in a Github org that your GITHUB_OAUTH_TOKEN has access to, pass the name of the Organization with this flag, to page through every repo via the Github API and target it | String | No |
| `--commit-message` | The commit message to use when creating commits. If you supply this flag, but neither the optional `--pull-request-title` or `--pull-request-description` flags, then the commit message value will be used for all three. | String | No |
| `--skip-pull-requests` | If you don't want any pull requests opened, but would rather have your changes committed directly to your specified branch, pass this flag. Note that it won't work if your Github repo is configured with branch protections on the branch you're trying to commit directly to! | Boolean | No |
| `--skip-archived-repos`| If you want to exclude archived (read-only) repositories from the list of targeted repos, pass this flag. | Boolean | No |
| `--dry-run` | If you are in the process of testing out `git-xargs` or your intial set of targeted repos, but you don't want to make any changes via the Github API (pushing your local changes or opening pull requests) you can pass the dry-run branch. This is useful because the output report will still tell you which repos would have been affected, without actually making changes via the Github API to your remote repositories. | Boolean | No |
| Flag | Description | Type | Required |
| ------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------- | -------- |
| `--branch-name` | You must specify the name of the branch to make your local and remote changes on. You can further control branching behavior via `--skip-pull-requests` as explained below | String | Yes |
| `--repos` | If you want to specify many repos and manage them in files (which makes batching and testing easier) then use this flag to pass the filepath to a repos file. See [the repos file format](#option-2-flat-file-of-repository-names) for more information | String | No |
| `--repo` | Use this flag to specify a single repo, e.g., `--repo gruntwork-io/cloud-nuke`. Can be passed multiple times to target several repos | String | No |
| `--github-org` | If you want to target every repo in a Github org that your GITHUB_OAUTH_TOKEN has access to, pass the name of the Organization with this flag, to page through every repo via the Github API and target it | String | No |
| `--commit-message` | The commit message to use when creating commits. If you supply this flag, but neither the optional `--pull-request-title` or `--pull-request-description` flags, then the commit message value will be used for all three. | String | No |
| `--skip-pull-requests` | If you don't want any pull requests opened, but would rather have your changes committed directly to your specified branch, pass this flag. Note that it won't work if your Github repo is configured with branch protections on the branch you're trying to commit directly to! | Boolean | No |
| `--skip-archived-repos` | If you want to exclude archived (read-only) repositories from the list of targeted repos, pass this flag. | Boolean | No |
| `--dry-run` | If you are in the process of testing out `git-xargs` or your intial set of targeted repos, but you don't want to make any changes via the Github API (pushing your local changes or opening pull requests) you can pass the dry-run branch. This is useful because the output report will still tell you which repos would have been affected, without actually making changes via the Github API to your remote repositories. | Boolean | No |
| `--max-concurrent-repos` | Limits the number of concurrent processed repositories. This is only useful if you encounter issues and need throttling when running on a very large number of repos. Default is `0` (Unlimited) | Integer | No |
## Best practices, tips and tricks
Expand Down
1 change: 1 addition & 0 deletions cmd/git-xargs.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,7 @@ func parseGitXargsConfig(c *cli.Context) (*config.GitXargsConfig, error) {
config.ReposFile = c.String("repos")
config.GithubOrg = c.String("github-org")
config.RepoSlice = c.StringSlice("repo")
config.MaxConcurrentRepos = c.Int("max-concurrent-repos")
config.Args = c.Args()

shouldReadStdIn, err := dataBeingPipedToStdIn()
Expand Down
7 changes: 7 additions & 0 deletions common/common.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -13,9 +13,11 @@ const (
BranchFlagName = "branch-name"
PullRequestTitleFlagName = "pull-request-title"
PullRequestDescriptionFlagName = "pull-request-description"
MaxConcurrentReposFlagName = "max-concurrent-repos"
DefaultCommitMessage = "git-xargs programmatic commit"
DefaultPullRequestTitle = "git-xargs programmatic pull request"
DefaultPullRequestDescription = "git-xargs programmatic pull request"
DefaultMaxConcurrentRepos = 0
)

var (
Expand Down Expand Up @@ -62,4 +64,9 @@ var (
Usage: "The description to add to pull requests opened by git-xargs",
Value: DefaultPullRequestDescription,
}
GenericMaxConcurrentReposFlag = cli.IntFlag{
Name: MaxConcurrentReposFlagName,
Usage: "Limits the number of concurrent processed repositories. This is only useful if you encounter issues and need throttling when running on a very large number of repos. Default is 0 (Unlimited)",
Value: DefaultMaxConcurrentRepos,
}
)
2 changes: 2 additions & 0 deletions config/config.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@ type GitXargsConfig struct {
DryRun bool
SkipPullRequests bool
SkipArchivedRepos bool
MaxConcurrentRepos int
BranchName string
CommitMessage string
PullRequestTitle string
Expand All @@ -35,6 +36,7 @@ func NewGitXargsConfig() *GitXargsConfig {
DryRun: false,
SkipPullRequests: false,
SkipArchivedRepos: false,
MaxConcurrentRepos: 0,
BranchName: "",
CommitMessage: common.DefaultCommitMessage,
PullRequestTitle: common.DefaultPullRequestTitle,
Expand Down
1 change: 1 addition & 0 deletions go.mod
View file
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@ require (
github.com/kataras/tablewriter v0.0.0-20180708051242-e063d29b7c23
github.com/landoop/tableprinter v0.0.0-20200805134727-ea32388e35c1
github.com/mattn/go-runewidth v0.0.9 // indirect
github.com/remeh/sizedwaitgroup v1.0.0
github.com/sirupsen/logrus v1.7.0
github.com/stretchr/testify v1.7.0
github.com/urfave/cli v1.22.5
Expand Down
2 changes: 2 additions & 0 deletions go.sum
View file
Original file line number Diff line number Diff line change
Expand Up @@ -430,6 +430,8 @@ github.com/prometheus/procfs v0.0.0-20190507164030-5867b95ac084/go.mod h1:TjEm7z
github.com/prometheus/procfs v0.0.2/go.mod h1:TjEm7ze935MbeOT/UhFTIMYKhuLP4wbCsTZCD3I8kEA=
github.com/prometheus/procfs v0.0.3/go.mod h1:4A/X28fw3Fc593LaREMrKMqOKvUAntwMDaekg4FpcdQ=
github.com/prometheus/tsdb v0.7.1/go.mod h1:qhTCs0VvXwvX/y3TZrWD7rabWM+ijKTux40TwIPHuXU=
github.com/remeh/sizedwaitgroup v1.0.0 h1:VNGGFwNo/R5+MJBf6yrsr110p0m4/OX4S3DCy7Kyl5E=
github.com/remeh/sizedwaitgroup v1.0.0/go.mod h1:3j2R4OIe/SeS6YDhICBy22RWjJC5eNCJ1V+9+NVNYlo=
github.com/remyoudompheng/bigfft v0.0.0-20170806203942-52369c62f446/go.mod h1:uYEyJGbgTkfkS4+E/PavXkNJcbFIpEtjt2B0KDQ5+9M=
github.com/rogpeppe/fastuuid v0.0.0-20150106093220-6724a57986af/go.mod h1:XWv6SoW27p1b0cqNHllgS5HIMJraePCO15w5zCzIWYg=
github.com/rogpeppe/go-charset v0.0.0-20180617210344-2471d30d28b4/go.mod h1:qgYeAmZ5ZIpBWTGllZSQnw97Dj+woV0toclVaRGI8pc=
Expand Down
1 change: 1 addition & 0 deletions main.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -71,6 +71,7 @@ func setupApp() *cli.App {
common.GenericCommitMessageFlag,
common.GenericPullRequestTitleFlag,
common.GenericPullRequestDescriptionFlag,
common.GenericMaxConcurrentReposFlag,
}

app.Action = cmd.RunGitXargs
Expand Down
11 changes: 7 additions & 4 deletions repository/process.go
View file
Original file line number Diff line number Diff line change
@@ -1,20 +1,23 @@
package repository

import (
"sync"

"github.com/google/go-github/v32/github"
"github.com/gruntwork-io/git-xargs/config"
"github.com/gruntwork-io/go-commons/logging"
"github.com/remeh/sizedwaitgroup"
"github.com/sirupsen/logrus"
)

// Loop through every repo we've selected and use a WaitGroup so that the processing can happen in parallel
func ProcessRepos(gitxargsConfig *config.GitXargsConfig, repos []*github.Repository) error {
logger := logging.GetLogger("git-xargs")
var wg sync.WaitGroup

// Limit the number of concurrent goroutines using the MaxConcurrentRepos config value
// MaxConcurrentRepos == 0 will fallback to unlimited (previous default behavior)
wg := sizedwaitgroup.New(gitxargsConfig.MaxConcurrentRepos)

for _, repo := range repos {
wg.Add(1)
wg.Add()
go func(gitxargsConfig *config.GitXargsConfig, repo *github.Repository) error {
defer wg.Done()
// For each repo, run the supplied command against it and, if it succeeds without error,
Expand Down

0 comments on commit e9ce453

Please sign in to comment.