lakefs-rclone-export: Support S3 compatible storages

[jameshod5]

What happened?

What actually happened, including error codes if applies.

• Trying to export my LakeFS repo using the Docker command that is provided in the documentation, but I am met with an error:

S3 bucket mast path test/lakefs: error reading destination root directory: InvalidAccessKeyId: The AWS Access Key Id you provided does not exist in our records.

Using the exact same id/key with s5cmd works perfectly, so it is not an error with the key itself.

Steps to Reproduce:

1. Create a LakeFS repo with the S3 endpoint set as s3://mast/test/lakefs/
2. Upload some data to the repo.
3. Try and export that data to a different directory within the same S3 bucket s3://mast/test/lakefs/export (not sure if this is the reason for failure, does it need to be a different S3 bucket entirely?). Using this command:

docker run \              
    --network=dbnetwork \
    -e AWS_ENDPOINT_URL=xxx \
    -e LAKEFS_ACCESS_KEY_ID=xxx -e LAKEFS_SECRET_ACCESS_KEY=xxx \
    -e LAKEFS_ENDPOINT=http://lakefs:8000/ \
    -e AWS_ACCESS_KEY_ID=xxx -e AWS_SECRET_ACCESS_KEY=xxx \
    treeverse/lakefs-rclone-export:latest \
    example-repo \
    s3://mast/test/lakefs/export/ \
    --branch="main"

Expected behavior

I expect the data within my LakeFS repo to be exported to s3://mast/test/lakefs/export

lakeFS version

1.44.0

How lakeFS is installed

Docker

Affected clients

No response

Relevant log output

ERROR : S3 bucket mast path test/lakefs: error reading destination root directory: InvalidAccessKeyId: The AWS Access Key Id you provided does not exist in our records.
        status code: 403, request id: 3W8AJK4R2769N8WY, host id: 39ECKd4koxNTl2n7r0jqvpcuxCRersV3qHjvUYGiXZMB60WeLVMxd5QumHWibiwCkfA10g7w0io=

Contact details

[email protected]

[itaiad200]

Hey @jameshod5,

It seems that the process is trying to interact with S3 directly instead of the S3-compatible store you configured using AWS_ENDPOINT_URL.

From a quick look at the code, it appears that lakefs-rclone-export doesn't currently support that option, though it should be relatively simple to implement.

If you're interested in contributing, you can follow the relevant code here. It should be pretty straightforward.