So I had a use case where I needed to listen to a 4/5 different notifications and then setup a Lambda which synced it. I could setup a different Lambda for each SNS notification and then perform the operation on each message type. But I did not want so many of them around - only differing in how they read the message. Instead I chose to setup a single Lamba Function that could process all the varied notification messages.
First step - define the generic Lambda that handles all messages.
Next Step: Setup a simple SNS
As seen the Lambda is subscribed to the SNS. The SNS will be responsible for invoking the Lambda when a message is received on the topic. The Retry policy for Lambda is controlled by SNS and cannot be changed. (It is sufficient though, IMO). We can setup a dead letter queue if needed to handle failures.
I also added Cloud Watch logs for delivery. So when I sent a message, the logs of delivery details were generated for SNS:
Checking Lambda logs in Cloudwatch:
Similar setup could be achieved from SQS.
Now to setup the SQS queue:
I setup an SQS with the default settings. Next step would be to attach a Lambda to SQS (similar to adding Lambda as subscriber to SNS) However here I got an error:
As can be seen, I am missing some permissions. I updated my Lambda's execution role to include the needed permissions:
The Lambda was now attached. The reason for the difference goes down to how SNS and SQS work. SNS being a push type system (SNS pushes notifications to you) while SQS is a pull type system (you poll the SQS for messages). From the AWS Docs
First step - define the generic Lambda that handles all messages.
public class NotificationHandler implements RequestHandler<SNSEvent, Void> { public Void handleRequest(SNSEvent request, Context context) { LambdaLogger logger = context.getLogger(); logger.log("In Handler: Executing " + context.getFunctionName() + ", " + context.getFunctionVersion()); logger.log(request.toString()); for (SNSEvent.SNSRecord snsRecord : request.getRecords()) { SNSEvent.SNS snsMessage = snsRecord.getSNS(); System.out.println(snsMessage.getMessage()); } return null; } }The Lambda is trigger by an SNS subscription. The function code simple pulls out the SNSRecords in the event and the SNS Message within each record. The message is delivered as a simple text.
Next Step: Setup a simple SNS
As seen the Lambda is subscribed to the SNS. The SNS will be responsible for invoking the Lambda when a message is received on the topic. The Retry policy for Lambda is controlled by SNS and cannot be changed. (It is sufficient though, IMO). We can setup a dead letter queue if needed to handle failures.
I also added Cloud Watch logs for delivery. So when I sent a message, the logs of delivery details were generated for SNS:
{ "notification": { "messageMD5Sum": "e0322987d21db5cd44565e26a800ea28", "messageId": "822521ea-223f-50c2-b6f1-76cba1662543", "topicArn": "arn:aws:sns:us-east-1:083144642440:AllTheTopicsInTheWorld", "timestamp": "2020-04-28 01:04:25.682" }, "delivery": { "deliveryId": "17c14967-931c-5ac7-8cb1-1e4610452e66", "destination": "arn:aws:lambda:us-east-1:083144642440:function:SNSLambda", "providerResponse": "{\"lambdaRequestId\":\"e2ef1497-14e3-456d-9b24-d20b0a4f3869\"}", "dwellTimeMs": 105, "attempts": 1, "statusCode": 202 }, "status": "SUCCESS" }As seen the message was successfully delivered to my Lambda function.
Checking Lambda logs in Cloudwatch:
START RequestId: e2ef1497-14e3-456d-9b24-d20b0a4f3869 Version: $LATEST In Handler: Executing SNSLambda, $LATEST {[{sns: {messageAttributes: {},
signingCertUrl: https://sns.us-east-1.amazonaws.com/SimpleNotificationService-a86cb10b4e1f29c941702d737128f7b6.pem,
messageId: 822521ea-223f-50c2-b6f1-76cba1662543,message: {"Hello": "World"},
unsubscribeUrl: https://sns.us-east-1.amazonaws.com/?Action=Unsubscribe&SubscriptionArn=arn:aws:sns:us-east-1:083144642440:AllTheTopicsInTheWorld:fd95b7a7-7ff3-426b-a91e-5235442608ed,
type: Notification,signatureVersion: 1,signature: TBxwq15JBb6shTRB2eVWr3TTcJJ8L65U8ZCGRGDn1hgH9PDbxyf4LAqyEJn7tlG5NXZEKkHG8AvhceOURupv09AweTgBaoQZUHZRlMjt53s8Y6PsQmMRm0aqrLNM5t8Od+9NiocnlxISxLFxdxA5Uj2zV+qdDGqx0buiuOPoGPLxZJ0JRaaU4ZislJHCkdD9wbsCcmVEOO9iR1uX0HTS4dmZ4prZGfhLTqhlTS4fF9SrpvNmmHfFkRlEG8RHg29wrPrCwwPuaQqiWRwRZstl2Y6TQv18j94nAcx6HHUAPE+MNogoiAhiXaxOnlb4nUqozvzDBVhXsnzc64/vUEZz5g==,
timestamp: 2020-04-28T01:04:25.664Z,topicArn: arn:aws:sns:us-east-1:083144642440:AllTheTopicsInTheWorld},
eventVersion: 1.0,eventSource: aws:sns,eventSubscriptionArn: arn:aws:sns:us-east-1:083144642440:AllTheTopicsInTheWorld:fd95b7a7-7ff3-426b-a91e-5235442608ed}]} { "Hello": "World" } END RequestId: e2ef1497-14e3-456d-9b24-d20b0a4f3869 REPORT RequestId: e2ef1497-14e3-456d-9b24-d20b0a4f3869 Duration: 465.16 ms Billed Duration: 500 ms Memory Size: 512 MB Max Memory Used: 85 MB Init Duration: 442.15 msSince the message can be read as a string, it is possible to identify the message type before trying to use it for different cases.
Similar setup could be achieved from SQS.
public class NotificationHandler implements RequestHandler<SQSEvent, Void> { public Void handleRequest(SQSEvent request, Context context) { LambdaLogger logger = context.getLogger(); logger.log("In Handler: Executing " + context.getFunctionName() + ", " + context.getFunctionVersion()); logger.log(request.toString()); for (SQSEvent.SQSMessage sqsMessage : request.getRecords()) { String message = sqsMessage.getBody(); System.out.println(message); } return null; } }The code is pretty much the same except for the interface used. The SQSEvent like SNSEvent can include multiple messages. Each message has a Body field which contains the Message.
Now to setup the SQS queue:
I setup an SQS with the default settings. Next step would be to attach a Lambda to SQS (similar to adding Lambda as subscriber to SNS) However here I got an error:
As can be seen, I am missing some permissions. I updated my Lambda's execution role to include the needed permissions:
The Lambda was now attached. The reason for the difference goes down to how SNS and SQS work. SNS being a push type system (SNS pushes notifications to you) while SQS is a pull type system (you poll the SQS for messages). From the AWS Docs
Lambda polls the queue and invokes your function synchronously with an event
that contains queue messages. Lambda reads messages in batches and invokes
your function once for each batch. When your function successfully processes a batch,
Lambda deletes its messages from the queue.
As Lambda needs to perform the SQS operations before and after calling your permissions, you need to provide it the necessary permissions.
I pushed a message to my SQS Queue and soon enough the Function logs showed up in cloudwatch:
START RequestId: 0d128ef2-a17a-5727-8f74-d9d336762ae4 Version: $LATEST In Handler: Executing SQSLambda, $LATEST {Records: [{messageId: 63a8afb8-140e-4004-9d1c-ec3ee58a71a7, receiptHandle: AQEBmJUhK4igIbCi4iXq2U1GmexcezeNynSmYUmblEecCyJ+TOo2VkQvQw9ZO6ZnE4fsAzBxRVam7Da26Kv2ftQrUu2TyKWmSnQEdjVJGa9G7cv65+/p91UwVKQhFEC1Ma78UcyMJFydNoD71ohHXSgeLooI8PqCJHG0+Dd/FwXDFST7181CbEFBnqqLoSlZPnmmv8w7SCTwGpqj9rn4HDAEd5eRsN39F0jMflCdvZh6iB8cvI5mNHPKkgV86vfTMQUTqaI0Bf4AOttaMJf8x/UrSNZyxExHB69kRJHYGBJb+ZW+bQsOiCHmreiGx9cpAwx1qN6BO9FtrrpGWU3bKxrOR6n2k5UvsI3AaKAplUS9GSRCD6JFhJGCLeqWFAU0yAK/ ,eventSourceARN: arn:aws:sqs:us-east-1:083144642440:MyQueue,eventSource: aws:sqs,awsRegion: us-east-1, body: Hello World,md5OfBody: b10a8db164e0754105b7a99be72e3fe5,attributes: {ApproximateReceiveCount=1, SentTimestamp=1588040913447, SenderId=AROARGW6OR6EJYJIENZDG:AMSPortalWebsite+P-robin, ApproximateFirstReceiveTimestamp=1588040913462},messageAttributes: {}}]} Hello World END RequestId: 0d128ef2-a17a-5727-8f74-d9d336762ae4 REPORT RequestId: 0d128ef2-a17a-5727-8f74-d9d336762ae4 Duration: 1.42 ms Billed Duration: 100 ms Memory Size: 512 MB Max Memory Used: 84 MBUnlike SNS, SQS has good metrics, so you do not need to configure any cloud watch logs. (Nor did I find a way to)
No comments:
Post a Comment