-
Notifications
You must be signed in to change notification settings - Fork 67
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
garbage collection seems to be broken #198
Comments
Thank you for submitting your first issue to this repository! A maintainer will be here shortly to triage and review.
Finally, remember to use https://discuss.ipfs.io if you just need general support. |
It looks like the query logic isn't properly parsing keys. The query logic is known to have a bunch of issues, fixed in #39. However, that implements naive query logic (in some edge cases) which make the test-cases take way too long. |
The |
I've successfully merged release v0.8.0 into the #39 and made adjustments to certain code logic to ensure the query functions correctly. The changes made are as follows:
Here's the update I applied: diff --git a/s3.go b/s3.go
index 6205220..02f6269 100644
--- a/s3.go
+++ b/s3.go
@@ -215,6 +215,18 @@ func (s *S3Bucket) Query(ctx context.Context, q dsq.Query) (dsq.Results, error)
// Normalize the path and strip the leading / as S3 stores values
// without the leading /.
prefix := ds.NewKey(q.Prefix).String()[1:]
+ prefix = s.s3Path(prefix)
+ if strings.HasPrefix(prefix, "/") {
+ prefix = prefix[1:]
+ }
+ dirPrefix := s.s3Path("") + "/"
+ if strings.HasPrefix(dirPrefix, "/") {
+ dirPrefix = dirPrefix[1:]
+ }
+ dirPrefixLen := len(dirPrefix)
+ if len(prefix) < dirPrefixLen {
+ prefix = dirPrefix
+ }
sent := 0
queryLimit := func() int64 {
@@ -226,7 +238,7 @@ func (s *S3Bucket) Query(ctx context.Context, q dsq.Query) (dsq.Results, error)
resp, err := s.S3.ListObjectsV2WithContext(ctx, &s3.ListObjectsV2Input{
Bucket: aws.String(s.Bucket),
- Prefix: aws.String(s.s3Path(prefix)),
+ Prefix: aws.String(prefix),
Delimiter: aws.String("/"),
MaxKeys: aws.Int64(queryLimit()),
})
@@ -250,7 +262,7 @@ func (s *S3Bucket) Query(ctx context.Context, q dsq.Query) (dsq.Results, error)
resp, err = s.S3.ListObjectsV2WithContext(ctx, &s3.ListObjectsV2Input{
Bucket: aws.String(s.Bucket),
- Prefix: aws.String(s.s3Path(prefix)),
+ Prefix: aws.String(prefix),
Delimiter: aws.String("/"),
MaxKeys: aws.Int64(queryLimit()),
ContinuationToken: resp.NextContinuationToken,
@@ -260,8 +272,9 @@ func (s *S3Bucket) Query(ctx context.Context, q dsq.Query) (dsq.Results, error)
}
}
+ key := (*resp.Contents[index].Key)[dirPrefixLen:]
entry := dsq.Entry{
- Key: ds.NewKey(*resp.Contents[index].Key).String(),
+ Key: ds.NewKey(key).String(),
Size: int(*resp.Contents[index].Size),
}
if !q.KeysOnly { This change has enabled both |
Garbage collection appears to be broken with this plugin.
I have a working implementation of things where I can add / pin content, list content, get the size of the repo etc.
However running
ipfs repo gc
returns nothing and shows this in the daemon logs:No content is deleted from the s3 bucket when this happens.
Oddly I can run
ipfs block rm CID
just fine and the content is removed from the s3 bucketThe text was updated successfully, but these errors were encountered: