Reoccurring Aws::CloudWatch::Errors::ExpiredToken errors

[Roy-Gal-Git]

Ever since we started using the sidekiq-cloudwatchmetrics gem in production environments we get this error a couple times a week:

Aws::CloudWatch::Errors::ExpiredToken: The security token included in the request is expired

The solution is simply stopping the leader sidekiq worker, causing a new leader to start and send metrics with a new Aws::CloudWatch::Client.

This is our Sidekiq::CloudWatchMetrics.enable! call:

Sidekiq::CloudWatchMetrics.enable!(
  client: Aws::CloudWatch::Client.new(region: PLACEHOLDER_REGION),
  namespace: PLACEHOLDER_NAMESPACE
)

I created a similar client (client = Aws::CloudWatch::Client.new(region: PLACEHOLDER_REGION)) to interact with, and saw that when I run client.config there are a couple of variables with interesting values:

#<struct
	credentials=
		#<Aws::InstanceProfileCredentials:0x00007f78ee260660
			...
			@credentials=#<Aws::Credentials access_key_id="PLACEHOLDER_ACCESS_KEY_ID">,
			...>
	...
	secret_access_key=nil
	session_token=nil
	...>

According to the AWS SDK docs, Aws::InstanceProfileCredentials is an auto-refreshing credential provider (source).

I checked the gem's code and it seems like accessing @client.config.credentials causes a refresh if it's near its expiration time.
Another option is explicitly refreshing the credentials using @client.config.credentials.refresh!.

When I tried the refresh! method, I saw that the credentials object's @expiration and @credentials attributes do change, but the @token attribute is refreshed only once it expires (which took me 6 hours of refreshes to make sure the default value of 6 hours would indeed be refreshed).

This is very confusing... As I wrote earlier, the AWS SDK docs state that Aws::InstanceProfileCredentials is an auto-refreshing credential provider yet it doesn't refresh automatically before a request - causing the Aws::CloudWatch::Errors::ExpiredToken I get.

Do I configure the Sidekiq::CloudWatchMetrics client correctly?
Is there something I or you can do to prevent the Aws::CloudWatch::Errors::ExpiredToken error?

[Roy-Gal-Git]

I submitted a new PR, please review it #49

[sj26]

My understanding is, as you say, the credential provider should be refreshing them for you. Reading the code, it looks like the credentials will be refreshed if the gem believes they will expire within the next ~5 minutes:

https://github.com/aws/aws-sdk-ruby/blob/version-3/gems/aws-sdk-core/lib/aws-sdk-core/refreshing_credentials.rb#L18

If you are seeing otherwise then I think it is a bug in aws sdk. Please submit a bug upstream:

https://github.com/aws/aws-sdk-ruby/issues

When you see these messages do metrics stop being reported? i.e. are you seeing these messages constantly after a while? Do metrics stop appearing in CloudWatch? Or is it an occasional message which then goes away, so reporting is recovering? i.e. I wonder if you can ignore the messages, and it's just a side-effect of a race condition within credential refreshing which self-heals.

[Roy-Gal-Git]

@sj26
When I see these messages metrics stop being reported until I stop the leader worker.
I constantly get those messages after a while.
Reporting does not recover until I stop the leader worker.