SQS asynchronous tasks

SQS is a service (like RabbitMQ) that allows you to queue messages (aka “jobs”). It is a good fit for asynchronous tasks because it integrates natively with AWS Lambda.

Whenever a new message (job) is sent to SQS, Lambda is invoked with the message data. That means that there is no need to poll SQS, or run daemon/long-running processes to wait for messages. Lambda is invoked only when there are messages to process.

Handling SQS events

To handle SQS events , extend the SqsHandler class:

use Bref\Context\Context;
use Bref\Event\Sqs\SqsEvent;
use Bref\Event\Sqs\SqsHandler;
 
class MyHandler extends SqsHandler
{
    public function handleSqs(SqsEvent $event, Context $context): void
    {
        foreach ($event->getRecords() as $record) {
            // We can retrieve the message body of each record via `->getBody()`
            $body = $record->getBody();
 
            // do something
        }
    }
}

Then, create a Lambda function that listens to SQS events with the handler you created:

functions:
    # ...
    worker:
        handler: App\MyHandler
        runtime: php-84
        events:
            - sqs:
                arn: arn:aws:sqs:eu-west-1:111111111111:queue-name
                # process one message at a time
                batchSize: 1

functions:
    # ...
    worker:
        handler: App\MyHandler
        runtime: php-84
        events:
            - sqs:
                arn: arn:aws:sqs:eu-west-1:111111111111:queue-name
                # process one message at a time
                batchSize: 1

functions:
    # ...
    worker:
        handler: handler.php
        runtime: php-84
        events:
            - sqs:
                arn: arn:aws:sqs:eu-west-1:111111111111:queue-name
                # process one message at a time
                batchSize: 1

<?php
 
require __DIR__ . '/vendor/autoload.php';
 
return new MyHandler();

Creating SQS queues

It is possible to deploy a preconfigured SQS queue in serverless.yml using the Queue feature of the Lift plugin. For example:

constructs:
    my-queue:
        type: queue
        worker:
            handler: handler.php
            runtime: php-84

The queue created by Lift comes with a dead letter queue and has partial batch responses enabled.

Partial Batch Response

While handling a batch of records, you can mark it as partially successful to reprocess only the failed records.

In your function declaration in serverless.yml, set functionResponseType to ReportBatchItemFailures to let your function return a partial success result if one or more messages in the batch have failed (the Lift queue construct already does this).

functions:
    worker:
        handler: handler.php
        runtime: php-84
        events:
            - sqs:
                arn: arn:aws:sqs:eu-west-1:111111111111:queue-name
                batchSize: 10
                functionResponseType: ReportBatchItemFailures

In your PHP code, you can now use the markAsFailed method:

    public function handleSqs(SqsEvent $event, Context $context): void
    {
        foreach ($event->getRecords() as $record) {
            // do something
 
            // if something went wrong, mark the record as failed
            $this->markAsFailed($record);
        }
    }

Error handling

If the handler throws an exception, the whole batch fails: all its messages go back to the queue and will be processed again, including the ones that were processed successfully. Use partial batch responses to retry only the failed messages, or set batchSize: 1.

Failed messages are retried until they are moved to a dead letter queue, or until they expire (4 days by default). Configure a dead letter queue on the SQS queue itself (a “redrive policy” with a maximum number of receives): the onError option of the function does not apply to SQS events. Queues created with the Lift queue construct move messages to a dead letter queue after 3 failed attempts (maxRetries option).

S3, SNS and EventBridge invoke functions asynchronously instead: Lambda retries a failed invocation twice, then drops the event. To keep failed events, send them to a Lambda destination  with the destinations.onFailure option:

functions:
    my-function:
        # ...
        destinations:
            onFailure: arn:aws:sqs:eu-west-1:111111111111:failed-events

SQS polling and AWS costs

AWS Lambda maintains a continuous “poller” for SQS that invokes our function when messages are available. This process, while eliminating the need for manual polling, still creates SQS requests, which can incur charges. A single SQS queue with Lambda will be free since it fits in the AWS free tier. However, if you plan on deploying a lot of SQS queues, you should be aware of potential costs (about $0.26/month per queue).

If you want to reduce the number of SQS requests (and thus potential costs), you can set a maximum batching window: the maxBatchingWindow option of the Lift queue construct, or the maximumBatchingWindow option of sqs events. This setting delays message delivery to the Lambda function when batching, aiming to assemble the largest possible batch within the specified timeframe, counted as a single SQS API request. Note that this technique introduces a delay in message processing, so be aware of the trade-off.

Learn more

Learn more about all the options available for SQS events in serverless.yml in the Serverless Framework documentation .

You can also learn more about SQS, workers, scaling queues, and dealing with errors in Serverless Visually Explained .