AWS SNS With Lambda Pub-Sub using CDK

In the Microservices world, we often have multiple components calling each other over the distributed network. We should be able to scale and enhance individual services. The communication must happen in a loosely coupled and reliable manner. Publish/subscribe messaging, or pub/sub messaging, is a form of asynchronous service-to-service communication used in serverless and microservices architectures.

In the pub-sub model, the publisher sends a message from distributed microservice or other AWS service. It does not care who will consume the message. The subscribes who want to consume the message, can subscribe to the message channel. This separates the publisher and consumer dependencies.

In this blog post, we will see how to use the publish-subscribe messaging pattern using AWS Simple Notification Service and Lambda function.

AWS Simple Notification Service

AWS SNS is a fully managed pub-sub messaging service. Pub/sub messaging can be used to enable event-driven architectures or to decouple applications in order to increase performance, reliability, and scalability. There are two use cases for SNS. One is both the publisher and subscriber are applications called application-to-application communication(A2A). Another is the publisher is an application and the subscriber might be an end-user which is application-to-person communication(A2P).

The A2A type of messaging helps us to connect to various different distributed microservices and other AWS services. The application can push the events to a large number of subscriber applications such as AWS Lambda, AWS SQS queues, HTTP endpoints.

The A2P type of messaging helps us to directly send messages to the end-users via SMS, push Notification, and Email.

By using SNS, we can create applications using event-driven architecture. We break large applications into smaller, independent components which helps improve performance and we can scale applications easier. SNS provides the retry policy for the message delivery to the destinations. It has something called a dead-letter queue, which will store the messages that are not delivered to destinations. SNA supports FIFO topics to be created which helps in ordering the message deliveries.

The Amazon SNS supports various Event sources and Event destinations. Some of the common AWS event sources include AWS EventBridge, AWS Step functions. The AWS Event destinations include Lambda function, Amazon SQS which are A2A and SMS, Email which is A2P. For a full list of supported sources and destinations, check out the documentation of SNS Event sources and Destinations

Create a new directory on your system.

mkdir cdk-sns && cd cdk-sns

We will use cdk init to create a new Typescript CDK project:

cdk init --language typescript

The cdk init command creates a number of files and folders inside the cdk-sns directory to help us organize the source code for your AWS CDK app.

We can list the stacks in our app by running the below command. It will show CdkSnsStack.

$ cdk ls
CdkSnsStack

Let us install the AWS SNS construct.

npm install @aws-cdk/aws-sns

In AWS SNS, the publisher system sends a message to a topic. A topic is a communication channel where the publisher system publishes and other subscriber systems subscribe to.

Edit the file lib/cdk-sns-stack.ts to create an SNS topic as shown below.

 // Create an SNS Topic
    const topic = new sns.Topic(this,'notification', {
      topicName: 'notification-topic',
      displayName: 'notification-topic'
    })

Synthesize an AWS CloudFormation template for the app, as follows.

cdk synth

To deploy the stack using AWS CloudFormation which was generated, use the below command.

cdk deploy

As seen from above, the stack is successfully deployed. Now, we can log in to the AWS console to see the sns Topic notification-topic created as shown below.

To receive messages published to a topic notification-topic, we have to subscribe an endpoint to the topic. We will see how to subscribe with the Lambda endpoint.

Before creating a Lambda subscription endpoint, we have to create a Lambda function using CDK. For more detail on how to create, check out lambda function using CDK .

Create Lambda Function using CDK

Let us install the AWS lambda construct.

npm install @aws-cdk/aws-lambda

Edit the file lib/cdk-sns-stack.ts to create an AWS lambda resource as shown below.

// Create an lambda function
        const notificationLambda = new lambda.Function(this, "notification-lambda", {
            runtime: lambda.Runtime.NODEJS_14_X,
            code: lambda.Code.fromAsset("lambda"),
            handler: "index.handler",
            functionName: "notification-lambda"
        });

The handler code is loaded from the directory named lambda where we will add the lambda code.

Create a new folder lambda at the root. The name of the handler function is index.handler where the index is the name of the file and handler is exported function name.

Create a file index.js and add the below code.

exports.handler = async function(event) {
    console.log("Received Event:", JSON.stringify(event, undefined, 2));
    let message = event.Records[0].Sns.Message;
    let subject = event.Records[0].Sns.Subject;
    let type = event.Records[0].Sns.Type;
    let response = {
        message: message,
        subject: subject,
        type: type
    }
    console.log('SNS record: ', JSON.stringify(response, null, 2));
    return response
};

Now run the command cdk deploy which will deploy the lambda function. We can verify by logging into the AWS console.

Now, let us subscribe to the SNS topic using CDK.

Let us install the AWS SNS Subscription construct.

npm install @aws-cdk/aws-sns-subscriptions

Edit the file lib/cdk-sns-stack.ts to subscribe the lambda function to the SNS topic as shown below.

// subscribe an Lambda to SNS topic
        topic.addSubscription(new subscription.LambdaSubscription(notificationLambda));

Let us also output the Topic ARN of notification-topic, so that, we can send the events for testing. Add the below code in file lib/cdk-sns-stack.ts

// output ARN
        new cdk.CfnOutput(this, 'snsTopicARN', {
            value: topic.topicArn,
            description: 'The SNS notification-topic ARN'
        })

Now run the command cdk deploy which will subscribe the lambda function notification-lambda to the topic notification-topic. We can verify by logging into the AWS console.

We can also see the trigger for lambda is our SNS topic.

Trigger the event from SNS

To trigger an SNS event, we need to know the topic ARN of SNS notification-topic. We get it from the output, which we have added to our code. It is as shown below.

Now trigger the event using AWS CLI as shown below.

aws sns publish --subject "Notification-topic to Lambda" --message "This is A2A type of messaging" --topic-arn "arn:aws:sns
:ap-south-1:339619077582:notification-topic" --profile rahul-admin

We get the output with MessageId as shown below:

Let us verify if the SNS has triggered the lambda by going into the AWS console and looking into AWS Cloudwatch logs. We can see the event reached to lambda function and we can see the same MessageId, which we saw in the console when triggered from CLI.

Also, the Lambda output is also as expected as shown below.

Conclusion

In this blog post, we saw about AWS Simple Notification Service. We have created a topic for the publisher system to publish the message. We have used AWS Lambda as a subscriber to the topic. We saw how to build infrastructure for AWS SNS and Lambda using Cloud development Kit. We also saw how to trigger the SNS messages using AWS CLI.

GitHub Repo: https://github.com/rahulmlokurte/aws-usage/tree/main/aws-cdk/cdk-sns

How to use AWS SNS with Lambda subscriptions in Publisher-Subscriber messaging systems using CDK

Comments (1)

More from this blog

Understanding Token and Positional Embeddings in Transformers

Overview of RAG

Basics of Retrieval Augmented Generation

Managing Concurrent Writes and Object Locking in Amazon S3: Strategies and Best Practices

Saga Pattern using AWS Step Functions

AWS Simple Notification Service

Create Lambda Function using CDK

Trigger the event from SNS

Conclusion

Command Palette

Comments (1)

More from this blog

AWS Simple Notification Service

Create SNS topic using CDK

Create a Lambda Subscription to the SNS Topic

Create Lambda Function using CDK

Subscribe to the SNS topic

Trigger the event from SNS

Conclusion