Search This Blog

Tuesday, 28 April 2020

SNS SQS and the multi function Lambda

So I had a use case where I needed to listen to a 4/5 different notifications and then setup a Lambda which synced it. I could setup a different Lambda for each SNS notification and then perform the operation on each message type. But I did not want so many of them around - only differing in how they read the message. Instead I chose to setup a single Lamba Function that could process all the varied notification messages.
First step - define the generic Lambda that handles all messages.
public class NotificationHandler implements RequestHandler<SNSEvent, Void> {

    public Void handleRequest(SNSEvent request, Context context) {
        LambdaLogger logger = context.getLogger();
        logger.log("In Handler: Executing " + context.getFunctionName() + ", " + context.getFunctionVersion());
        logger.log(request.toString());
        for (SNSEvent.SNSRecord snsRecord : request.getRecords()) {
            SNSEvent.SNS snsMessage = snsRecord.getSNS();
            System.out.println(snsMessage.getMessage());
        }
        return null;
    }
}
The Lambda is trigger by an SNS subscription. The function code simple pulls out the SNSRecords in the event and the SNS Message within each record. The message is delivered as a simple text.
Next Step: Setup a simple SNS
As seen the Lambda is subscribed to the SNS. The SNS will be responsible for invoking the Lambda when a message is received on the topic. The Retry policy for Lambda is controlled by SNS and cannot be changed. (It is sufficient though, IMO). We can setup a dead letter queue if needed to handle failures.
I also added Cloud Watch logs for delivery. So when I sent a message, the logs of delivery details were generated for SNS:
{
    "notification": {
        "messageMD5Sum": "e0322987d21db5cd44565e26a800ea28",
        "messageId": "822521ea-223f-50c2-b6f1-76cba1662543",
        "topicArn": "arn:aws:sns:us-east-1:083144642440:AllTheTopicsInTheWorld",
        "timestamp": "2020-04-28 01:04:25.682"
    },
    "delivery": {
        "deliveryId": "17c14967-931c-5ac7-8cb1-1e4610452e66",
        "destination": "arn:aws:lambda:us-east-1:083144642440:function:SNSLambda",
        "providerResponse": "{\"lambdaRequestId\":\"e2ef1497-14e3-456d-9b24-d20b0a4f3869\"}",
        "dwellTimeMs": 105,
        "attempts": 1,
        "statusCode": 202
    },
    "status": "SUCCESS"
}
As seen the message was successfully delivered to my Lambda function.
Checking Lambda logs in Cloudwatch:
START RequestId: e2ef1497-14e3-456d-9b24-d20b0a4f3869 Version: $LATEST
In Handler: Executing SNSLambda, $LATEST
{[{sns: {messageAttributes: {},
signingCertUrl: https://sns.us-east-1.amazonaws.com/SimpleNotificationService-a86cb10b4e1f29c941702d737128f7b6.pem,
messageId: 822521ea-223f-50c2-b6f1-76cba1662543,message: {"Hello": "World"},
unsubscribeUrl: https://sns.us-east-1.amazonaws.com/?Action=Unsubscribe&SubscriptionArn=arn:aws:sns:us-east-1:083144642440:AllTheTopicsInTheWorld:fd95b7a7-7ff3-426b-a91e-5235442608ed,
type: Notification,signatureVersion: 1,signature: TBxwq15JBb6shTRB2eVWr3TTcJJ8L65U8ZCGRGDn1hgH9PDbxyf4LAqyEJn7tlG5NXZEKkHG8AvhceOURupv09AweTgBaoQZUHZRlMjt53s8Y6PsQmMRm0aqrLNM5t8Od+9NiocnlxISxLFxdxA5Uj2zV+qdDGqx0buiuOPoGPLxZJ0JRaaU4ZislJHCkdD9wbsCcmVEOO9iR1uX0HTS4dmZ4prZGfhLTqhlTS4fF9SrpvNmmHfFkRlEG8RHg29wrPrCwwPuaQqiWRwRZstl2Y6TQv18j94nAcx6HHUAPE+MNogoiAhiXaxOnlb4nUqozvzDBVhXsnzc64/vUEZz5g==,
timestamp: 2020-04-28T01:04:25.664Z,topicArn: arn:aws:sns:us-east-1:083144642440:AllTheTopicsInTheWorld},
eventVersion: 1.0,eventSource: aws:sns,eventSubscriptionArn: arn:aws:sns:us-east-1:083144642440:AllTheTopicsInTheWorld:fd95b7a7-7ff3-426b-a91e-5235442608ed}]}
{
    "Hello": "World"
}
END RequestId: e2ef1497-14e3-456d-9b24-d20b0a4f3869
REPORT RequestId: e2ef1497-14e3-456d-9b24-d20b0a4f3869	Duration: 465.16 ms	Billed Duration: 500 ms	Memory Size: 512 MB	Max Memory Used: 85 MB	Init Duration: 442.15 ms
Since the message can be read as a string, it is possible to identify the message type before trying to use it for different cases.
Similar setup could be achieved from SQS.
public class NotificationHandler implements RequestHandler<SQSEvent, Void> {

    public Void handleRequest(SQSEvent request, Context context) {
        LambdaLogger logger = context.getLogger();
        logger.log("In Handler: Executing " + context.getFunctionName() + ", " + context.getFunctionVersion());
        logger.log(request.toString());
        for (SQSEvent.SQSMessage sqsMessage : request.getRecords()) {
            String message = sqsMessage.getBody();
            System.out.println(message);
        }
        return null;
    }
}
The code is pretty much the same except for the interface used. The SQSEvent like SNSEvent can include multiple messages. Each message has a Body field which contains the Message.
Now to setup the SQS queue:

I setup an SQS with the default settings. Next step would be to attach a Lambda to SQS (similar to adding Lambda as subscriber to SNS) However here I got an error:
As can be seen, I am missing some permissions. I updated my Lambda's execution role to include the needed permissions:
The Lambda was now attached. The reason for the difference goes down to how SNS and SQS work. SNS being a push type system (SNS pushes notifications to you) while SQS is a pull type system (you poll the SQS for messages). From the AWS Docs

Lambda polls the queue and invokes your function synchronously with an event 
that contains queue messages. Lambda reads messages in batches and invokes 
your function once for each batch. When your function successfully processes a batch, 
 Lambda deletes its messages from the queue.
As Lambda needs to perform the SQS operations before and after calling your permissions, you need to provide it the necessary permissions. I pushed a message to my SQS Queue and soon enough the Function logs showed up in cloudwatch:
START RequestId: 0d128ef2-a17a-5727-8f74-d9d336762ae4 Version: $LATEST
In Handler: Executing SQSLambda, $LATEST
{Records: [{messageId: 63a8afb8-140e-4004-9d1c-ec3ee58a71a7,
receiptHandle: AQEBmJUhK4igIbCi4iXq2U1GmexcezeNynSmYUmblEecCyJ+TOo2VkQvQw9ZO6ZnE4fsAzBxRVam7Da26Kv2ftQrUu2TyKWmSnQEdjVJGa9G7cv65+/p91UwVKQhFEC1Ma78UcyMJFydNoD71ohHXSgeLooI8PqCJHG0+Dd/FwXDFST7181CbEFBnqqLoSlZPnmmv8w7SCTwGpqj9rn4HDAEd5eRsN39F0jMflCdvZh6iB8cvI5mNHPKkgV86vfTMQUTqaI0Bf4AOttaMJf8x/UrSNZyxExHB69kRJHYGBJb+ZW+bQsOiCHmreiGx9cpAwx1qN6BO9FtrrpGWU3bKxrOR6n2k5UvsI3AaKAplUS9GSRCD6JFhJGCLeqWFAU0yAK/
,eventSourceARN: arn:aws:sqs:us-east-1:083144642440:MyQueue,eventSource: aws:sqs,awsRegion: us-east-1,
body: Hello World,md5OfBody: b10a8db164e0754105b7a99be72e3fe5,attributes:
 {ApproximateReceiveCount=1, SentTimestamp=1588040913447, SenderId=AROARGW6OR6EJYJIENZDG:AMSPortalWebsite+P-robin, 
ApproximateFirstReceiveTimestamp=1588040913462},messageAttributes: {}}]}
Hello World
END RequestId: 0d128ef2-a17a-5727-8f74-d9d336762ae4
REPORT RequestId: 0d128ef2-a17a-5727-8f74-d9d336762ae4	Duration: 1.42 ms	Billed Duration: 100 ms	
Memory Size: 512 MB	Max Memory Used: 84 MB	
Unlike SNS, SQS has good metrics, so you do not need to configure any cloud watch logs. (Nor did I find a way to)

No comments:

Post a Comment