Godep breaks when go get fails to fetch a non-existing package in an existing repo

[fkautz]

In the github.com/docker/distribution project, godep is setting an import package as:

github.com/MSOpenTech/azure-sdk-for-go/clients/storage

instead of

github.com/MSOpenTech/azure-sdk-for-go

The result is

 godep restore
package github.com/MSOpenTech/azure-sdk-for-go/clients/storage
    imports github.com/MSOpenTech/azure-sdk-for-go/clients/storage
    imports github.com/MSOpenTech/azure-sdk-for-go/clients/storage: cannot find package "github.com/MSOpenTech/azure-sdk-for-go/clients/storage" in any of:
    /home/fkautz/opt/go1.4.1/src/github.com/MSOpenTech/azure-sdk-for-go/clients/storage (from $GOROOT)
    /home/fkautz/gopath/src/github.com/MSOpenTech/azure-sdk-for-go/clients/storage (from $GOPATH)

[fkautz]

Further analysis, godep uses go get.

github.com/MSOpenTech/azure-sdk-for-go/clients/storage

results in

go get github.com/MSOpenTech/azure-sdk-for-go/clients/storage

which fails because clients/storage is not in master.

However, the repo is properly downloaded. All that remains is the vcs revision being set to the correct version. (DL succeeds thanks to https://github.com/golang/go/blob/master/src/cmd/go/vcs.go#L697-L701)

One potential fix would be to detect "cannot find package" in the error, traverse to the repo location (as identified in vcs.go or ending in .vcs), and performing the vcs revision from the root of the repo. A go get without -u should result in a successful package download at that point to verify it worked.

Do you have any recommendations? I can help implement a fix for this.

[kr]

This is reasonable for a start. Another possibility would be to copy code from the go tool to download all packages in process. This would give much more control over this process, and let us fix one or two other issues as well. However, that would be a lot more work.

For checking the error output of go get, there are a couple of things to consider:

1. If the godep restore is eventually successful, it should produce be no output.
2. If it is not successful, each error should be printed exactly once.

[fkautz]

I am ok with copying code over. I'll put together an initial attempt.

[fkautz]

@kr: Here is a working restore. I have given you access to github.com/fkautz/godep so we can collaborate on this more efficiently.

@ahmetalpbalkan: pinging you since you were involved in earlier priv convo on this

https://github.com/fkautz/godep/tree/restore-uses-inproc-with-fixed-repo-root

There are currently two commits. The first is a naive copy from cmd/go to github.com/tools/godep/govcs. The second is a working restore. Once we get to a nice state, we can squash.

A few things need to be done before this is ready for a PR and merge.

1: get needs to be modified to use govcs
2: govcs needs to be cleaned out of everything that is unnecessary
3: run tests to make sure we don't break others.

I also tried a few manual tests.

1. Remove a repo
2. Repo exists and set to current master (verified rewind)
3. Repo doesn't exist (or private), got http login requests (as expected from go get)

[fkautz]

Also, this cmd/go was from 1.4.1. It is likely to fail in previous versions. If backwards compatibility is important, we should work our way backwards after pruning anything we don't want in the govcs package.

[ahmetb]

@fkautz have you found a way to update this one yet? 😄

[fkautz]

The branch listed above technically works, but needs some cleanup. We replicated cmd/go, publicly exposed some private fields/methods, and are able to pull and store properly.

I'll get back onto this.

[karlkfi]

Looks like the proposed solution branch vendors go code, which is probably a non-viable solution to this problem due to to having to keep it compatible with the installed go version (ignoring potential licensing issues).

Would it be possible to instead pop off dir names when a cannot find package error occurs on go get and retry?

[karlkfi]

Looks like that change is non-trivial due to the way godep restore delegates to go get, go list, and git cat-file before checking out the specific revision. You could do a go get on path.Join(path.Dir(dep.ImportPath), "...") but you'd still have to find out what dir your in, and if the parent dir doesn't have any go files then go list will error. You can get parse the json and get the dir even when it fails, but not with the current LoadPackages impl. And then vcs.exists will fail if pkg.Dir doesn't exist. So you'd have to make the package first...

Kind of a mess. Might be easier to add an option to go get to get a specific revision.

[fkautz]

@karlkfi that branch was the result of this comment above: #186 (comment)

Here's some additional info:

Premise: godep uses go get to check out the repo and git to rewind

You can classify every repo into two categories.

1. golang doesn't know the root. e.g. it doesn't know how to resolve git.example.com/docker/docker/hack to any repo

Fortunately, golang provides a way to resolve the second issue by allowing the user to specify a repo. So, we need to solve the first problem.

1. golang knows the root. e.g. github.com/docker/docker/hack/ will be resolved to https://github.com/docker/docker.git

So, godep calls go get. go get fails because it can check out the repo and then verifies the directory within the repo exists. So, the resulting fix I proposed basically downloads the repo, drops the directory not found exception, then rewinds to the correct version.

Another way to fix this would be to take the hardcoded list and just make calls to git directly.

[freeformz]

This should now be fixed in v37. Please re-open if not. Thanks!

[ironcladlou]

Using godep @ 03595ab, I'm still observing the bug:

# in some clean GOPATH...
go get github.com/openshift/origin
go get github.com/tools/godep
cd src/github.com/openshift/origin
$GOPATH/bin/godep restore -v

Failures will eventually occur:

...

godep: error downloading dep (github.com/aws/aws-sdk-go/internal/endpoints): cannot find package "github.com/aws/aws-sdk-go/internal/endpoints" in any of:
        /usr/lib/golang/src/github.com/aws/aws-sdk-go/internal/endpoints (from $GOROOT)
        /home/vagrant/tmp/subbug/src/github.com/aws/aws-sdk-go/internal/endpoints (from $GOPATH)
godep: Downloading dependency (if needed): github.com/aws/aws-sdk-go/internal/protocol/ec2query
godep: error downloading dep (github.com/aws/aws-sdk-go/internal/protocol/ec2query): cannot find package "github.com/aws/aws-sdk-go/internal/protocol/ec2query" in any of:
        /usr/lib/golang/src/github.com/aws/aws-sdk-go/internal/protocol/ec2query (from $GOROOT)
        /home/vagrant/tmp/subbug/src/github.com/aws/aws-sdk-go/internal/protocol/ec2query (from $GOPATH)

...

The Godeps.json file contains entries that reference import paths which only exist at the specified revisions. For example:

{
  "ImportPath": "github.com/aws/aws-sdk-go/internal/endpoints",
  "Comment": "v0.9.9",
  "Rev": "c4ae871ffc03691a7b039fa751a1e7afee56e920"
}

Here, github.com/aws/aws-sdk-go/internal/endpoints only exists in Rev, and it seems like godep is still trying to generate a call like go get github.com/aws/aws-sdk-go/internal/endpoints (from HEAD) rather than downloading the github.com/aws/aws-sdk-go repo and then rewinding in Git (for example).

[eparis]

Docker just remove timeutils: moby/moby#18685

So this block will now refuse to godep restore

                {
                        "ImportPath": "github.com/docker/docker/pkg/timeutils",
                        "Comment": "v1.4.1-4045-g2b27fe1",
                        "Rev": "2b27fe17a1b3fb8472fde96d768fa70996adf201"
                },