Skip to content

Latest commit

 

History

History
137 lines (96 loc) · 4.22 KB

queues.md

File metadata and controls

137 lines (96 loc) · 4.22 KB
Raw

Queues

Some tasks are too long to be processed synchronously. Instead, they can be processed in the background via a job queue and worker.

The queues component deploys a properly configured SQS queue with a worker running on AWS Lambda.

Quick start

service: my-app
provider:
  name: aws

queues:
  my-queue:
    worker:
      handler: src/report-generator.handler

plugins:
    - serverless-lift

On serverless deploy, a my-queue queue will be created, and a Lambda function will be deployed to process jobs (aka "messages") from the queue.

How it works

The queues component deploys the following resources:

  • An SQS queue: this is where jobs to process should be sent.
  • A "worker" Lambda function: this function will be processing each job sent to the SQS queue.
  • An SQS "Dead Letter Queue": this queue will contain all the jobs that failed to be processed.
  • Optionally, a CloudWatch alarm that sends an email if jobs land in the Dead Letter Queue.

To learn more about the architecture of this component, read this article.

Configuration reference

Worker

queues:
  my-queue:
    worker:
      # The Lambda function is configured here
      handler: src/report-generator.handler

Note: the Lambda "worker" function is configured in the queues component, instead of being defined in the functions section.

The only required setting is the handler: this should point to the code that handles SQS messages. The handler should be written to handle SQS events, for example in JavaScript:

exports.handler = async function(event, context) {
    event.Records.forEach(record => {
        // `record` contains the job that was pushed to SQS
    });
}

All settings allowed for functions can be used under the worker key. For example:

queues:
  my-queue:
    worker:
      handler: src/report-generator.handler
      memorySize: 512
      timeout: 10

Note: Lift will automatically configure the function to be triggered by SQS. It is not necessary to define events on the function.

Alarm

queues:
  my-queue:
    # ...
    alarm: [email protected]

It is possible to configure email alerts in case jobs end up in the dead letter queue.

After the first deployment, an email will be sent to the email address to confirm the subscription.

Retries

queues:
  my-queue:
    # ...
    maxRetries: 5

Default: 3 retries.

The maxRetries option configures how many times each job will be retried when failing.

If the job still fails after reaching the max retry count, it will be moved to the dead letter queue for storage.

Retry delay

When Lambda fails processing a SQS job (i.e. the code throws an error), the job will be retried after a delay. That delay is also called "Visibility Timeout" in SQS.

By default, Lift configures the retry delay to 6 times the worker functions timeout, per AWS' recommendation. Since Serverless deploy functions with a timeout of 6 seconds by default, that means that jobs will be retried every 36 seconds.

When the function's timeout is changed, the retry delay is configured accordingly:

queues:
  my-queue:
    # ...
    worker:
      handler: src/report-generator.handler
      # We change the timeout to 10 seconds
      timeout: 10
      # The retry delay on the queue will be 10*6 => 60 seconds

Batch size

queues:
  my-queue:
    # ...
    batchSize: 5 # Lambda will receive 5 messages at a time

Default: 1

When the SQS queue contains more than 1 job to process, it can invoke Lambda with a batch of multiple messages at once.

By default, Lambda will be invoked 1 messages at a time. The reason is to simplify error handling: in a batch, any failed message will fail the whole batch.

It is possible to change the batch size between 1 and 10.