Skip to content

Commit

Permalink
Subdir supports glob patterns
Browse files Browse the repository at this point in the history
This also adds documentation to the README!

This modifies the subdir argument (after the `//` in any URL) to support
globbing. Since the path after `//` is always resolved _after_ any
downloader and decompressor finishes, we have the full filesystem paths
local on disk. Therefore, we can glob the paths to support unknown
subdirs.

This is specifically done so we can support "unwrapping" archives that
are known to be rooted with a parent directory but the directory name is
unknown. This is common practice with tar.gz, tar.bz, etc. files.

The following would now work:

```
http://example.com/my.tar.gz//*
```

This would unwrap the first directory. If there are multiple directories
in the archive (specifically if multiple glob matches are found) then it
is a download error.

Existing tests pass, new tests added.
  • Loading branch information
mitchellh committed Sep 2, 2017
1 parent 99c9021 commit ed2b33b
Show file tree
Hide file tree
Showing 5 changed files with 84 additions and 1 deletion.
31 changes: 31 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -119,6 +119,37 @@ The protocol-specific options are documented below the URL format
section. But because they are part of the URL, we point it out here so
you know they exist.

### Subdirectories

If you want to download only a specific subdirectory from a downloaded
directory, you can specify a subdirectory after a double-slash `//`.
go-getter will first download the URL specified _before_ the double-slash
(as if you didn't specify a double-slash), but will then copy the
path after the double slash into the target directory.

For example, if you're downloading this GitHub repository, but you only
want to download the `test-fixtures` directory, you can do the following:

```
https://github.com/hashicorp/go-getter.git//test-fixtures
```

If you downloaded this to the `/tmp` directory, then the file
`/tmp/archive.gz` would exist. Notice that this file is in the `test-fixtures`
directory in this repository, but because we specified a subdirectory,
go-getter automatically copied only that directory contents.

Subdirectory paths may contain may also use filesystem glob patterns.
The path must match _exactly one_ entry or go-getter will return an error.
This is useful if you're not sure the exact directory name but it follows
a predictable naming structure.

For example, the following URL would also work:

```
https://github.com/hashicorp/go-getter.git//test-*
```

### Checksumming

For file downloads of any protocol, go-getter can automatically verify
Expand Down
16 changes: 15 additions & 1 deletion client.go
Original file line number Diff line number Diff line change
Expand Up @@ -305,7 +305,21 @@ func (c *Client) Get() error {
return err
}

return copyDir(realDst, filepath.Join(dst, subDir), false)
// Subdir matching supports globs. This allows matching unknown
// subdirectory structures but where we know the general path.
// For example, for archives that have a root folder containing
// all the items, you can download it with the getter URL of:
// 'path//*'.
matches, err := filepath.Glob(filepath.Join(dst, subDir))
if err != nil {
return err
}
if len(matches) > 1 {
return fmt.Errorf("subdir %q matches multiple paths", subDir)
}
subDir = matches[0]

return copyDir(realDst, subDir, false)
}

return nil
Expand Down
38 changes: 38 additions & 0 deletions get_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -145,6 +145,44 @@ func TestGetAny_archive(t *testing.T) {
}
}

func TestGet_archiveRooted(t *testing.T) {
dst := tempDir(t)
u := testModule("archive-rooted/archive.tar.gz")
if err := Get(dst, u); err != nil {
t.Fatalf("err: %s", err)
}

mainPath := filepath.Join(dst, "root", "hello.txt")
if _, err := os.Stat(mainPath); err != nil {
t.Fatalf("err: %s", err)
}
}

func TestGet_archiveSubdirWild(t *testing.T) {
dst := tempDir(t)
u := testModule("archive-rooted/archive.tar.gz")
u += "//*"
if err := Get(dst, u); err != nil {
t.Fatalf("err: %s", err)
}

mainPath := filepath.Join(dst, "hello.txt")
if _, err := os.Stat(mainPath); err != nil {
t.Fatalf("err: %s", err)
}
}

func TestGet_archiveSubdirWildMultiMatch(t *testing.T) {
dst := tempDir(t)
u := testModule("archive-rooted-multi/archive.tar.gz")
u += "//*"
if err := Get(dst, u); err == nil {
t.Fatal("should error")
} else if !strings.Contains(err.Error(), "multiple") {
t.Fatalf("err: %s", err)
}
}

func TestGetAny_file(t *testing.T) {
dst := tempDir(t)
u := testModule("basic-file/foo.txt")
Expand Down
Binary file added test-fixtures/archive-rooted-multi/archive.tar.gz
Binary file not shown.
Binary file added test-fixtures/archive-rooted/archive.tar.gz
Binary file not shown.

0 comments on commit ed2b33b

Please sign in to comment.