Skip to content

Commit ed2b33b

Browse files
committed
Subdir supports glob patterns
This also adds documentation to the README! This modifies the subdir argument (after the `//` in any URL) to support globbing. Since the path after `//` is always resolved _after_ any downloader and decompressor finishes, we have the full filesystem paths local on disk. Therefore, we can glob the paths to support unknown subdirs. This is specifically done so we can support "unwrapping" archives that are known to be rooted with a parent directory but the directory name is unknown. This is common practice with tar.gz, tar.bz, etc. files. The following would now work: ``` http://example.com/my.tar.gz//* ``` This would unwrap the first directory. If there are multiple directories in the archive (specifically if multiple glob matches are found) then it is a download error. Existing tests pass, new tests added.
1 parent 99c9021 commit ed2b33b

File tree

5 files changed

+84
-1
lines changed

5 files changed

+84
-1
lines changed

README.md

+31
Original file line numberDiff line numberDiff line change
@@ -119,6 +119,37 @@ The protocol-specific options are documented below the URL format
119119
section. But because they are part of the URL, we point it out here so
120120
you know they exist.
121121

122+
### Subdirectories
123+
124+
If you want to download only a specific subdirectory from a downloaded
125+
directory, you can specify a subdirectory after a double-slash `//`.
126+
go-getter will first download the URL specified _before_ the double-slash
127+
(as if you didn't specify a double-slash), but will then copy the
128+
path after the double slash into the target directory.
129+
130+
For example, if you're downloading this GitHub repository, but you only
131+
want to download the `test-fixtures` directory, you can do the following:
132+
133+
```
134+
https://github.com/hashicorp/go-getter.git//test-fixtures
135+
```
136+
137+
If you downloaded this to the `/tmp` directory, then the file
138+
`/tmp/archive.gz` would exist. Notice that this file is in the `test-fixtures`
139+
directory in this repository, but because we specified a subdirectory,
140+
go-getter automatically copied only that directory contents.
141+
142+
Subdirectory paths may contain may also use filesystem glob patterns.
143+
The path must match _exactly one_ entry or go-getter will return an error.
144+
This is useful if you're not sure the exact directory name but it follows
145+
a predictable naming structure.
146+
147+
For example, the following URL would also work:
148+
149+
```
150+
https://github.com/hashicorp/go-getter.git//test-*
151+
```
152+
122153
### Checksumming
123154

124155
For file downloads of any protocol, go-getter can automatically verify

client.go

+15-1
Original file line numberDiff line numberDiff line change
@@ -305,7 +305,21 @@ func (c *Client) Get() error {
305305
return err
306306
}
307307

308-
return copyDir(realDst, filepath.Join(dst, subDir), false)
308+
// Subdir matching supports globs. This allows matching unknown
309+
// subdirectory structures but where we know the general path.
310+
// For example, for archives that have a root folder containing
311+
// all the items, you can download it with the getter URL of:
312+
// 'path//*'.
313+
matches, err := filepath.Glob(filepath.Join(dst, subDir))
314+
if err != nil {
315+
return err
316+
}
317+
if len(matches) > 1 {
318+
return fmt.Errorf("subdir %q matches multiple paths", subDir)
319+
}
320+
subDir = matches[0]
321+
322+
return copyDir(realDst, subDir, false)
309323
}
310324

311325
return nil

get_test.go

+38
Original file line numberDiff line numberDiff line change
@@ -145,6 +145,44 @@ func TestGetAny_archive(t *testing.T) {
145145
}
146146
}
147147

148+
func TestGet_archiveRooted(t *testing.T) {
149+
dst := tempDir(t)
150+
u := testModule("archive-rooted/archive.tar.gz")
151+
if err := Get(dst, u); err != nil {
152+
t.Fatalf("err: %s", err)
153+
}
154+
155+
mainPath := filepath.Join(dst, "root", "hello.txt")
156+
if _, err := os.Stat(mainPath); err != nil {
157+
t.Fatalf("err: %s", err)
158+
}
159+
}
160+
161+
func TestGet_archiveSubdirWild(t *testing.T) {
162+
dst := tempDir(t)
163+
u := testModule("archive-rooted/archive.tar.gz")
164+
u += "//*"
165+
if err := Get(dst, u); err != nil {
166+
t.Fatalf("err: %s", err)
167+
}
168+
169+
mainPath := filepath.Join(dst, "hello.txt")
170+
if _, err := os.Stat(mainPath); err != nil {
171+
t.Fatalf("err: %s", err)
172+
}
173+
}
174+
175+
func TestGet_archiveSubdirWildMultiMatch(t *testing.T) {
176+
dst := tempDir(t)
177+
u := testModule("archive-rooted-multi/archive.tar.gz")
178+
u += "//*"
179+
if err := Get(dst, u); err == nil {
180+
t.Fatal("should error")
181+
} else if !strings.Contains(err.Error(), "multiple") {
182+
t.Fatalf("err: %s", err)
183+
}
184+
}
185+
148186
func TestGetAny_file(t *testing.T) {
149187
dst := tempDir(t)
150188
u := testModule("basic-file/foo.txt")
210 Bytes
Binary file not shown.
173 Bytes
Binary file not shown.

0 commit comments

Comments
 (0)