Support more errors for retries

[benjdezi]

In lib/ex_aws/request.ex, a retry is only attempted when we get ProvisionedThroughputExceededException or ThrottlingException. However, in some cases it would be nice to be able to do that for other types of errors (like NotFound for S3 GET).

I would be convenient if config allowed to specify a list a combinations of http methods and error types we want to attempt retries for.

[benwilson512]

This is not quite correct. Any 5XX status code will result in a retry. The Throughput and Throttling errors also permit retrying for 4XX errors.

However, that chunk of the code could definitely receive some refactoring. Really this ought to be handled at an operation or service level because it's a bit odd to have custom logic for basically just DynamoDB there and not in a module more oriented towards DynamoDB support.

That said, I don't see why I would want to retry on a S3 GET. If an item is not there, there's no reason to expect that there will be in the next 15 or so seconds that you get out of a retry. A particular application might have that expectation, but if so that's the responsibility of the application to handle.

[CrowdHailer]

Hopefully this is the right issue to add my comment too. I've just started using ex_aws in conjunction with arc, So it might be an arc issue. I am trying to save an object to S3 and am getting an unmatched case within the ex_aws code. It looks like it does not handle any 3** response code. In my opinion it should automatically retry at the new location.

This is the offending case clause https://github.com/CargoSense/ex_aws/blob/master/lib/ex_aws/request.ex#L34

and my error

** (CaseClauseError) no case clause matching: {:ok, %{body: "\nPermanentRedirectThe bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint.useful-music-tmpuseful-music-tmp.s3.amazonaws.com5C76365D4EE13821GXrpbJ+dX0HAjdNbaL6Q1L2ZMp/EKxPJGugU+2SJw3rvbPvkXZE0uWQiqyTRyiohz/SteZAbAs4=", headers: [{"x-amz-bucket-region", "us-west-2"}, {"x-amz-request-id", "5C76365D4EE13821"}, {"x-amz-id-2", "GXrpbJ+dX0HAjdNbaL6Q1L2ZMp/EKxPJGugU+2SJw3rvbPvkXZE0uWQiqyTRyiohz/SteZAbAs4="}, {"Content-Type", "application/xml"}, {"Transfer-Encoding", "chunked"}, {"Date", "Fri, 16 Dec 2016 09:46:50 GMT"}, {"Connection", "close"}, {"Server", "AmazonS3"}], status_code: 301}}
(ex_aws) lib/ex_aws/request.ex:34: ExAws.Request.request_and_retry/7
(ex_aws) lib/ex_aws/operation/s3.ex:44: ExAws.Operation.ExAws.Operation.S3.perform/2
(arc) lib/arc/storage/s3.ex:39: Arc.Storage.S3.do_put/3
(elixir) lib/task/supervised.ex:94: Task.Supervised.do_apply/2
(elixir) lib/task/supervised.ex:45: Task.Supervised.reply/5
(stdlib) proc_lib.erl:247: :proc_lib.init_p_do_apply/3

[benwilson512]

This is unrelated, it generally is an indicator that you tried to access a bucket in a region other than the one configured.

[joshprice]

I've hit the same redirect issue above and have raised it as a separate issue #359

[Hinidu]

SQS returns throttling errors in this format: {:error, {:http_error, 403, %{code: "RequestThrottled", detail: "", message: "Request is throttled.", request_id: "33fe89cb-5b40-516d-aaaf-dd19d8cc1ab5", type: "Sender"}}}. It would be nice to handle them too. Btw it should be parsed to something like {:error, {"RequestThrottled", "Request is throttled."}} like the other similar errors.