Skip to content

Commit

Permalink
[Build] Do a git shallow clone with fetch-depth of 25 in checkout (ap…
Browse files Browse the repository at this point in the history
…ache#8394)

### Motivation

There's a significant different in performance when using a shallow
clone. 125 seconds vs. 10 seconds to clone Pulsar on a 100Mbps internet
connection. 568MB vs. 16MB was transferred.

In Github Actions, the speed difference with fetch-depth: 25 compared to fetch-depth: 0 for Pulsar repository is 2-4 seconds compared to 70-75 seconds. about **15-35x faster.** 
Besides a fast checkout, 552MB less data was transferred and less disk space will be consumed.

example benchmark of cloning full vs. shallow (--depth=25):
```
$ time git clone https://github.com/apache/pulsar.git pp
Cloning into 'pp'...
remote: Enumerating objects: 601, done.
remote: Counting objects: 100% (601/601), done.
remote: Compressing objects: 100% (340/340), done.
remote: Total 1207163 (delta 473), reused 296 (delta 226), pack-reused 1206562
Receiving objects: 100% (1207163/1207163), 568.24 MiB | 5.66 MiB/s, done.
Resolving deltas: 100% (862029/862029), done.
git clone https://github.com/apache/pulsar.git pp  122,95s user 9,60s system 105% cpu 2:05,23 total
$ time git clone --depth=25 https://github.com/apache/pulsar.git pp2
Cloning into 'pp2'...
remote: Enumerating objects: 7699, done.
remote: Counting objects: 100% (7699/7699), done.
remote: Compressing objects: 100% (5239/5239), done.
remote: Total 7699 (delta 3120), reused 3707 (delta 1177), pack-reused 0
Receiving objects: 100% (7699/7699), 16.16 MiB | 2.85 MiB/s, done.
Resolving deltas: 100% (3120/3120), done.
git clone --depth=25 https://github.com/apache/pulsar.git pp2  2,24s user 0,64s system 31% cpu 9,152 total
```

Currently for the Pulsar Github workflows, it's only the [diff-only action](https://github.com/apache/pulsar-test-infra/blob/master/diff-only/entrypoint.sh) that depends on having all PR commits in the checked out local git repository. Therefore setting the value to 25 is reasonable. The 25th oldest commit would contain all changes so beyond
that point, the diff-only action produce the correct result. That's
a non critical critical issue since diff-only is used to detect if
the PR contains only documentation changes. It simply wouldn't be able
to do the detection if the PR has more than 25 commits. (the fetch-depth
limit could be increase if 25 isn't enough)

### Modifications

Change the fetch-depth of check action to 25 instead of 0 which does a full clone.
  • Loading branch information
lhotari authored Oct 28, 2020
1 parent 03f71d9 commit 01e7cf1
Show file tree
Hide file tree
Showing 23 changed files with 23 additions and 23 deletions.
2 changes: 1 addition & 1 deletion .github/workflows/ci-cpp.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,7 @@ jobs:
- name: checkout
uses: actions/checkout@v2
with:
fetch-depth: 0
fetch-depth: 25
ref: ${{ github.event.pull_request.head.sha }}

- name: Check if this pull request only changes documentation
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/ci-go-functions-style.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -48,7 +48,7 @@ jobs:
- name: Check out code into the Go module directory
uses: actions/checkout@v2
with:
fetch-depth: 0
fetch-depth: 25
ref: ${{ github.event.pull_request.head.sha }}

- name: Check if this pull request only changes documentation
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/ci-go-functions-test.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -50,7 +50,7 @@ jobs:
- name: Check out code into the Go module directory
uses: actions/checkout@v2
with:
fetch-depth: 0
fetch-depth: 25
ref: ${{ github.event.pull_request.head.sha }}

- name: Check if this pull request only changes documentation
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,7 @@ jobs:
- name: checkout
uses: actions/checkout@v2
with:
fetch-depth: 0
fetch-depth: 25
ref: ${{ github.event.pull_request.head.sha }}

- name: Cache local Maven repository
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/ci-integration-cli.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,7 @@ jobs:
- name: checkout
uses: actions/checkout@v2
with:
fetch-depth: 0
fetch-depth: 25
ref: ${{ github.event.pull_request.head.sha }}

- name: Cache local Maven repository
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/ci-integration-function-state.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,7 @@ jobs:
- name: checkout
uses: actions/checkout@v2
with:
fetch-depth: 0
fetch-depth: 25
ref: ${{ github.event.pull_request.head.sha }}

- name: Cache local Maven repository
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/ci-integration-messaging.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,7 @@ jobs:
- name: checkout
uses: actions/checkout@v2
with:
fetch-depth: 0
fetch-depth: 25
ref: ${{ github.event.pull_request.head.sha }}

- name: Cache local Maven repository
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/ci-integration-process.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,7 @@ jobs:
- name: checkout
uses: actions/checkout@v2
with:
fetch-depth: 0
fetch-depth: 25
ref: ${{ github.event.pull_request.head.sha }}

- name: Cache local Maven repository
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/ci-integration-schema.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,7 @@ jobs:
- name: checkout
uses: actions/checkout@v2
with:
fetch-depth: 0
fetch-depth: 25
ref: ${{ github.event.pull_request.head.sha }}

- name: Cache local Maven repository
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/ci-integration-sql.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,7 @@ jobs:
- name: checkout
uses: actions/checkout@v2
with:
fetch-depth: 0
fetch-depth: 25
ref: ${{ github.event.pull_request.head.sha }}

- name: Cache local Maven repository
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/ci-integration-standalone.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,7 @@ jobs:
- name: checkout
uses: actions/checkout@v2
with:
fetch-depth: 0
fetch-depth: 25
ref: ${{ github.event.pull_request.head.sha }}

- name: Cache local Maven repository
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/ci-integration-thread.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,7 @@ jobs:
- name: checkout
uses: actions/checkout@v2
with:
fetch-depth: 0
fetch-depth: 25
ref: ${{ github.event.pull_request.head.sha }}

- name: Cache local Maven repository
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/ci-integration-tiered-filesystem.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,7 @@ jobs:
- name: checkout
uses: actions/checkout@v2
with:
fetch-depth: 0
fetch-depth: 25
ref: ${{ github.event.pull_request.head.sha }}

- name: Cache local Maven repository
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/ci-integration-tiered-jcloud.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,7 @@ jobs:
- name: checkout
uses: actions/checkout@v2
with:
fetch-depth: 0
fetch-depth: 25
ref: ${{ github.event.pull_request.head.sha }}

- name: Cache local Maven repository
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/ci-license.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,7 @@ jobs:
- name: checkout
uses: actions/checkout@v2
with:
fetch-depth: 0
fetch-depth: 25
ref: ${{ github.event.pull_request.head.sha }}

- name: Check if this pull request only changes documentation
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/ci-pulsarbot.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,7 @@ jobs:
- name: checkout
uses: actions/checkout@v2
with:
fetch-depth: 0
fetch-depth: 25
ref: ${{ github.event.pull_request.head.sha }}

- name: Execute pulsarbot command
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/ci-unit-broker-broker-gp1.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,7 @@ jobs:
- name: checkout
uses: actions/checkout@v2
with:
fetch-depth: 0
fetch-depth: 25
ref: ${{ github.event.pull_request.head.sha }}

- name: Check if this pull request only changes documentation
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/ci-unit-broker-broker-gp2.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,7 @@ jobs:
- name: checkout
uses: actions/checkout@v2
with:
fetch-depth: 0
fetch-depth: 25
ref: ${{ github.event.pull_request.head.sha }}

- name: Check if this pull request only changes documentation
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/ci-unit-broker-client-api.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,7 @@ jobs:
- name: checkout
uses: actions/checkout@v2
with:
fetch-depth: 0
fetch-depth: 25
ref: ${{ github.event.pull_request.head.sha }}

- name: Check if this pull request only changes documentation
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/ci-unit-broker-client-impl.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,7 @@ jobs:
- name: checkout
uses: actions/checkout@v2
with:
fetch-depth: 0
fetch-depth: 25
ref: ${{ github.event.pull_request.head.sha }}

- name: Check if this pull request only changes documentation
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/ci-unit-broker-other.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,7 @@ jobs:
- name: checkout
uses: actions/checkout@v2
with:
fetch-depth: 0
fetch-depth: 25
ref: ${{ github.event.pull_request.head.sha }}

- name: Check if this pull request only changes documentation
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/ci-unit-proxy.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,7 @@ jobs:
- name: checkout
uses: actions/checkout@v2
with:
fetch-depth: 0
fetch-depth: 25
ref: ${{ github.event.pull_request.head.sha }}

- name: Check if this pull request only changes documentation
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/ci-unit.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,7 @@ jobs:
- name: checkout
uses: actions/checkout@v2
with:
fetch-depth: 0
fetch-depth: 25
ref: ${{ github.event.pull_request.head.sha }}

- name: Check if this pull request only changes documentation
Expand Down

0 comments on commit 01e7cf1

Please sign in to comment.