How to Resolve 429 Errors on 169.254.170.2 IMDS Path When Creating ActionMailer Objects in ActionMailer::DeliveryJob

[solomon-maeng]

Hello, I'm experiencing 429 errors on the 169.254.170.2 IMDS path when executing ActionMailer::DeliveryJob. This happens after enqueuing a large number of jobs using the Sidekiq Adapter with the perform_later approach, and when Sidekiq processes begin executing these enqueued jobs.

Previously, we configured AWS SES authentication using the SMTP method:

config.action_mailer.delivery_method = :smtp
config.action_mailer.default_url_options = { :host => 'https://abc.com' }
config.action_mailer.smtp_settings = {
  address: Rails.configuration.aws[:ses_server_name],
  port: 587,
  domain: 'abc.com',
  user_name: Rails.configuration.aws[:ses_username],
  password: Rails.configuration.aws[:ses_password],
  authentication: 'plain',
  enable_starttls_auto: true
}

Recently, we changed the configuration to use AWS ECS Task Role with SES Assume Role instead of setting SMTP credentials:

config.action_mailer.delivery_method = :aws_sdk

Since changing to the Assume Role configuration method, we encounter 429 errors on the 169.254.170.2 IMDS path whenever we send large volumes of emails through ActionMailer::DeliveryJob.

I believe the issue occurs because when classes inheriting from ActionMailer::Base create ActionMailer::DeliveryJob instances that are processed by the Sidekiq Adapter, there's no throttling mechanism in place. When creating ActionMailer objects for mass email delivery, each instance makes requests to the 169.254.170.2 IMDS path to obtain SES permissions through the Credentials Provider Chain, resulting in 429 errors.

If my understanding is correct, how can I resolve this issue by ensuring the SES object is created only once in the ActionMailer object? I want to prevent excessive requests to the 169.254.170.2 IMDS path.

[mullermp]

Are you using aws-sdk-rails and related gems? https://docs.aws.amazon.com/sdk-for-ruby/aws-actionmailer-ses/api/Aws/ActionMailer/SES/Mailer.html

I think you can fetch credentials ahead of time using the ECSCredentials class in your config during start up and pass it this way to the client. It will only refresh when near expiration.

I don't recall how action mailer creates instances, I will have to double check, but I thought it was once. In this gem we could also consider taking a client configuration option so that a client isn't created each time

[solomon-maeng]

First, thank you for your response.

Yes, I am using the aws-sdk-rails Gem and I'm using the configuration you provided in the link as follows:

In the production.rb file under the environments folder:
config.action_mailer.delivery_method = :aws_sdk

In the aws.rb file under the initializers folder:
Aws::Rails.add_action_mailer_delivery_method(:aws_sdk, { region: Rails.configuration.aws[:ses_region] })

Are there any additional considerations or settings I should be aware of for aws-sdk-rails beyond the configuration mentioned above?

[mullermp]

I think you may be using an older version. What version of aws-sdk-rails and aws-actionmailer-ses are you using?

In your initializer, try creating an instance of ECSCredentials and then pass it to your client configuration so that it is reused in client creation.

[solomon-maeng]

The Ruby version I'm using is 2.6.10, Rails version is 5.0.7.2, and aws-sdk-rails is 1.0.1. It appears that aws-actionmailer-ses wasn't added as a gem file due to the older versions of Ruby and Rails.

Is it still possible to create an instance of ECSCredentials and pass it to the client configuration when using these older versions of Ruby and Rails?

[mullermp]

Yes, you should be able to. Try passing the credentials config option in your initializer alongside your region.

[solomon-maeng]

I was thinking of using Aws::ECSCredentials.new in the initializer to pass to the config like this:

Aws::Rails.add_action_mailer_delivery_method(:aws_sdk, { 
    region: Rails.configuration.aws[:ses_region], 
    credentials: Aws::ECSCredentials.new 
})

Is my understanding correct? And if I explicitly create credentials, will there be any side effects? (This is because SNSClient and S3Client are created using Ruby's ||= assignment operator without explicitly passing credentials)