tag:blogger.com,1999:blog-62621287358184936742024-03-18T14:45:31.397+05:30Learning the code wayA small step towards sharing what I have learned.Robin Varghesehttp://www.blogger.com/profile/00841917517097216645noreply@blogger.comBlogger407125tag:blogger.com,1999:blog-6262128735818493674.post-88742148923773514142021-06-24T03:41:00.001+05:302021-06-24T03:41:12.731+05:30CDK - Diving deeper into the code<p>In the <a href="http://learningviacode.blogspot.com/2020/11/cdk-making-cloudformation-easier.html" target="_blank">last post</a> we setup a simple CDK project that resulted in creation of dynamodb table. Let us analyze the code further.</p><a name='more'></a>
Starting from where the code begins:
<pre style="background: rgb(255, 255, 255); color: black;"><span style="color: maroon; font-weight: bold;">const</span> app <span style="color: #808030;">=</span> <span style="color: maroon; font-weight: bold;">new</span> cdk<span style="color: #808030;">.</span>App<span style="color: #808030;">(</span><span style="color: #808030;">)</span><span style="color: purple;">;</span></pre>
From <a href="https://docs.aws.amazon.com/cdk/api/latest/docs/@aws-cdk_core.App.html" target="_blank">AWS Docs</a>:
<pre style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; color: black;"><span style="background-color: #fcff01;">A construct which represents an entire CDK app.</span><span style="background-color: white;"> This construct is normally
the root of the construct tree. You would normally define an App instance
in your program's entrypoint, then define constructs where the app is used
as the parent scope.
</span></pre>
Here we are introduced to another CDK term - <b>Construct</b>.
<pre style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; color: black;"><span style="background-color: white;">Constructs are the basic building blocks of AWS CDK apps. </span><span style="background-color: #fcff01;">A construct represents a
"cloud component" and encapsulates everything AWS CloudFormation needs to create
the component.</span><span style="background-color: white;">
A construct can represent a single resource, such as an Amazon S3 bucket, or it can
represent a higher-level component consisting of multiple AWS resources. Examples of
such components include a worker queue with its associated compute capacity, a cron job
with monitoring resources and a dashboard, or even an entire app spanning multiple AWS
accounts and regions.
</span></pre>
From what I understand the CDK code that we create can be thought of as a graph/tree of Constructs. In this graph we could have an S3Bucket Construct, a DynamoDb table construct etc. These constructs connect to each other in a graph. The root of this graph (or the node with no incoming edges) is the <b>App</b> Construct. Everything moves from there.
<pre style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; color: black;">The App construct doesn't require any initialization arguments, because it's the only
construct that can be used as a root for the construct tree.You can now use the App
instance as a scope for defining a single instance of your stack.
</pre>
All the other constructs that we create require a scope argument:
<pre style="background: rgb(255, 255, 255); color: black;"><span style="color: maroon; font-weight: bold;">new</span> Construct<span style="color: #808030;">(</span>scope<span style="color: purple;">:</span> Construct<span style="color: #808030;">,</span> id<span style="color: purple;">:</span> string<span style="color: #808030;">,</span> options<span style="color: purple;">?</span><span style="color: purple;">:</span> ConstructOptions<span style="color: #808030;">)</span>
</pre>
In many cases we will be passing in App instance as a scope parameter. The second argument is 'id' - An identifier that must be unique within this scope.
<pre style="background: rgb(255, 255, 255); color: black;">
This lets you instantiate and reuse constructs without concern for the constructs and identifiers
they might contain, and enables composing constructs into higher level abstractions.
</pre>
The last parameter 'ConstructOptions' - is an optional set of properties that define the construct's initial configuration. The next line is
<pre style="background: rgb(255, 255, 255); color: black;"><span style="color: maroon; font-weight: bold;">new</span> CdkAppStack<span style="color: #808030;">(</span>app<span style="color: #808030;">,</span> <span style="color: maroon;">'</span><span style="color: #0000e6;">CdkAppStack</span><span style="color: maroon;">'</span><span style="color: #808030;">)</span><span style="color: purple;">;</span>
</pre>
This will create an instance of CdkAppStack
<pre style="background: rgb(255, 255, 255); color: black;"><span style="color: maroon; font-weight: bold;">import</span> <span style="color: #808030;">*</span> as cdk from <span style="color: maroon;">'</span><span style="color: #0000e6;">@aws-cdk/core</span><span style="color: maroon;">'</span><span style="color: purple;">;</span>
<span style="color: maroon; font-weight: bold;">import</span> <span style="color: #808030;">*</span> as dd from <span style="color: maroon;">'</span><span style="color: #0000e6;">@aws-cdk/aws-dynamodb</span><span style="color: maroon;">'</span><span style="color: purple;">;</span>
<span style="color: maroon; font-weight: bold;">export</span> <span style="color: maroon; font-weight: bold;">class</span> CdkAppStack <span style="color: maroon; font-weight: bold;">extends</span> cdk<span style="color: #808030;">.</span>Stack <span style="color: purple;">{</span>
<span style="color: #797997;">constructor</span><span style="color: #808030;">(</span>scope<span style="color: purple;">:</span> cdk<span style="color: #808030;">.</span>Construct<span style="color: #808030;">,</span> id<span style="color: purple;">:</span> string<span style="color: #808030;">,</span> props<span style="color: purple;">?</span><span style="color: purple;">:</span> cdk<span style="color: #808030;">.</span>StackProps<span style="color: #808030;">)</span> <span style="color: purple;">{</span>
<span style="color: maroon; font-weight: bold;">super</span><span style="color: #808030;">(</span>scope<span style="color: #808030;">,</span> id<span style="color: #808030;">,</span> props<span style="color: #808030;">)</span><span style="color: purple;">;</span>
<span style="color: dimgrey;">// The code that defines your stack goes here</span>
<span style="color: maroon; font-weight: bold;">new</span> dd<span style="color: #808030;">.</span>Table<span style="color: #808030;">(</span><span style="color: maroon; font-weight: bold;">this</span><span style="color: #808030;">,</span> <span style="color: maroon;">'</span><span style="color: #0000e6;">cdkSampleTable</span><span style="color: maroon;">'</span><span style="color: #808030;">,</span> <span style="color: purple;">{</span>
tableName<span style="color: purple;">:</span> <span style="color: maroon;">'</span><span style="color: #0000e6;">cdkSampleTable</span><span style="color: maroon;">'</span><span style="color: #808030;">,</span>
billingMode<span style="color: purple;">:</span> dd<span style="color: #808030;">.</span>BillingMode<span style="color: #808030;">.</span>PROVISIONED<span style="color: #808030;">,</span>
partitionKey<span style="color: purple;">:</span> <span style="color: purple;">{</span> name<span style="color: purple;">:</span> <span style="color: maroon;">'</span><span style="color: #0000e6;">hashKey</span><span style="color: maroon;">'</span><span style="color: #808030;">,</span> type<span style="color: purple;">:</span> dd<span style="color: #808030;">.</span>AttributeType<span style="color: #808030;">.</span>STRING <span style="color: purple;">}</span><span style="color: #808030;">,</span>
readCapacity<span style="color: purple;">:</span> <span style="color: #008c00;">5</span><span style="color: #808030;">,</span>
writeCapacity<span style="color: purple;">:</span><span style="color: #008c00;">5</span>
<span style="color: purple;">}</span><span style="color: #808030;">)</span><span style="color: purple;">;</span>
<span style="color: purple;">}</span>
<span style="color: purple;">}</span>
</pre>
The class extends the cdk.<b>Stack</b> class. The class represents a a single CloudFormation stack. We saw in CloudFormation documents:
<pre style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; color: black;"><span style="background-color: #fcff01;">When you use CloudFormation, you manage related resources as a single unit called a stack.</span><span style="background-color: white;">
You create, update, and delete a collection of resources by creating, updating, and
deleting stacks. All the resources in a stack are defined by the stack's CloudFormation
template. Suppose you created a template that includes an Auto Scaling group, Elastic Load
Balancing load balancer, and an Amazon Relational Database Service (Amazon RDS) database
instance. To create those resources, you create a stack by submitting the template that you
created, and CloudFormation provisions all those resources for you.
</span></pre>
Our class 'CdkAppStack' represents an Cloudformation stack that we want to create. The stack includes a dynamoDb table. Accordingly we create an instance of Table construct. CDK includes constructs that represent all such AWS resources.<div>CDK follows the idea of <b><span style="color: #2b00fe;">composition</span>.</b> Our code is placed within an App Construct.<span style="color: #2b00fe;"> The App Construct is composed of one or more Stacks (which represent a CloudFormation Stack). The Stack construct is further composed of one or more AWS constructs based on the resources you need to create.</span></div><div><br /></div><div><b>Types of Constructs available:</b></div><div><ol style="text-align: left;"><li><b>L1 or Low Level Constructs</b> </li><ol><li>Provided by CDK.</li><li>They are resources defined by CloudFormation.</li><li>All names prefixed by "Cfn" - e.g. "CfnBucket" or "CfnMethod"</li><li>To use Cfn resources, we must explicitly configure all resource properties, - this means developer requires a complete understanding of underlying AWS CloudFormation resource model.</li></ol><li><b>L2 or High Level Constructs</b></li><ol><li>Provided by CDK.</li><li>These can be considered higher level wrappers for L1 Constructs.</li><li>They provide defaults for several properties and also convenience methods, meaning developer need not worry of being an expert of underlying AWS CloudFormation resource model. </li><li>s3.Bucket class represents ab S3Bucket or Method class represents API gateway method.</li></ol></ol><div>In our example we used Table which is an L2 construct. the equivalent L1 construct would be CfnTable.</div><div><br /></div><div>There is also a third type of Construct available - <b>patterns</b></div><div>Patterns are basically a combination of constructs that are designed for common AWS Use cases - <a href="https://docs.aws.amazon.com/solutions/latest/constructs/aws-lambda-dynamodb.html?did=cs_card&trk=cs_card" target="_blank">aws-lambda-dynamodb pattern</a> provides readymade constructs to setup AWS Lambda function and Amazon DynamoDB table with least-privilege permissions</div><div>A comprehensive list can be found <a href="https://aws.amazon.com/solutions/constructs/patterns" target="_blank">here</a>. I plan to test out these in upcoming posts</div><div><br /></div></div>Robin Varghesehttp://www.blogger.com/profile/00841917517097216645noreply@blogger.com3tag:blogger.com,1999:blog-6262128735818493674.post-27847708246569269432020-11-16T03:21:00.000+05:302020-11-16T03:21:03.220+05:30CDK - Making CloudFormation easier<p> In the previous post I created my first AWS Resource using AWS CloudFormation. This was done using a JSON file called the CloudFormation template.<span></span></p><a name='more'></a><span>While CloudFormation makes it possible to codify our AWS resources, the templates used for CloudFormation have their limitations:</span><p></p><div><ul style="text-align: left;"><li><span>The Templates can be represented in only JSON or YAML.</span></li><li><span>This means the files tend to be very verbose. </span></li><li><span>We do not have the logic abilities that a programming language provides.</span></li></ul><div>Enter CDK (or <b>Cloud Development Kit</b>)</div><ul style="text-align: left;"><li><span>Most programmers are comfortable working with programming languages like </span>Java, and .NET. CDK allows us to model our resources in these languages (also Python or TypeScript)</li><li>Programming languages allow for better organization of infrastructure into modules, reusing these modules as well as IDE benefits.</li><li>At its backend CDK uses CloudFormation. So all benefits of CloudFormation are available here.</li></ul><div>The first step is to ensure you have AWS CLI installed (since at the end of the day, CDK will use CloudFormation). Also ensure that the AWS user credentials being used to run cli has admin access.</div></div><div>The next step is to install AWS CDK. CDK requires Node.js to be installed. With that CDK can be installed through the npm package manager.</div><div><br /></div><div>Now that installation is done we can setup and run CDK. To create a CDK project</div>
<div style="background: rgb(248, 248, 248); border-color: gray; border-image: initial; border-style: solid; border-width: 0.1em 0.1em 0.1em 0.8em; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;"><table><tbody><tr><td><pre style="line-height: 125%; margin: 0px;">1</pre></td><td><pre style="line-height: 125%; margin: 0px;"><span style="color: #888888;">cdk init app --language typescript</span>
</pre></td></tr></tbody></table></div>
This will create a basic project with required default configurations.
<div style="background: rgb(248, 248, 248); border-color: gray; border-image: initial; border-style: solid; border-width: 0.1em 0.1em 0.1em 0.8em; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;"><table><tbody><tr><td><pre style="line-height: 125%; margin: 0px;">1
2
3</pre></td><td><pre style="line-height: 125%; margin: 0px;"><span style="color: #888888;">ls</span>
<span style="color: #888888;">README.md cdk.json jest.config.js node_modules package.json</span></pre><pre style="line-height: 125%; margin: 0px;"><span style="color: #888888;">tsconfig.json</span><span> </span><span style="color: #888888;">bin cdk.out lib </span></pre><pre style="line-height: 125%; margin: 0px;"><span style="color: #888888;">package-lock.json test</span>
</pre></td></tr></tbody></table></div>
The application starter file is based in the bin folder:
<!--HTML generated using hilite.me--><div style="background: rgb(248, 248, 248); border-color: gray; border-image: initial; border-style: solid; border-width: 0.1em 0.1em 0.1em 0.8em; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;"><table><tbody><tr><td><pre style="line-height: 125%; margin: 0px;">1
2
3
4
5
6
7
8
9</pre></td><td><pre style="line-height: 125%; margin: 0px;"><span style="color: #aa22ff; font-weight: bold;">import</span> <span style="color: #bb4444;">'source-map-support/register'</span>;
<span style="color: #aa22ff; font-weight: bold;">import</span> <span style="color: #666666;">*</span> as cdk from <span style="color: #bb4444;">'@aws-cdk/core'</span>;
<span style="color: #aa22ff; font-weight: bold;">import</span> {
CdkAppStack
} from <span style="color: #bb4444;">'../lib/cdk-app-stack'</span>;
<span style="color: #008800; font-style: italic;">//This is the starting point for the app</span>
<span style="background-color: #fcff01;"><span style="color: #aa22ff; font-weight: bold;">const</span> app <span style="color: #666666;">=</span> <span style="color: #aa22ff; font-weight: bold;">new</span> cdk.App();
<span style="color: #aa22ff; font-weight: bold;">new</span> CdkAppStack(app, <span style="color: #bb4444;">'CdkAppStack'</span>);</span>
</pre></td></tr></tbody></table></div>
<div><br /></div><div>The actual code is in CdkAppStack.ts :
<!--HTML generated using hilite.me--><div style="background: rgb(248, 248, 248); border-color: gray; border-image: initial; border-style: solid; border-width: 0.1em 0.1em 0.1em 0.8em; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;"><table><tbody><tr><td><pre style="line-height: 125%; margin: 0px;"> 1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17</pre></td><td><pre style="line-height: 125%; margin: 0px;"><span style="color: #aa22ff; font-weight: bold;">import</span> <span style="color: #666666;">*</span> as cdk from <span style="color: #bb4444;">'@aws-cdk/core'</span>;
<span style="color: #aa22ff; font-weight: bold;">import</span> <span style="color: #666666;">*</span> as dd from <span style="color: #bb4444;">'@aws-cdk/aws-dynamodb'</span>;
<span style="color: #aa22ff; font-weight: bold;">export</span> <span style="color: #aa22ff; font-weight: bold;">class</span> CdkAppStack <span style="color: #aa22ff; font-weight: bold;">extends</span> cdk.Stack {
constructor(scope<span style="color: #666666;">:</span> cdk.Construct, id<span style="color: #666666;">:</span> string, props<span style="color: #666666;">?:</span> cdk.StackProps) {
<span style="color: #aa22ff; font-weight: bold;">super</span>(scope, id, props);
<span style="color: #008800; font-style: italic;">// The code that defines your stack goes here</span>
<span style="color: #aa22ff; font-weight: bold;">new</span> dd.Table(<span style="color: #aa22ff; font-weight: bold;">this</span>, <span style="color: #bb4444;">'cdkSampleTable'</span>, {
tableName<span style="color: #666666;">:</span> <span style="color: #bb4444;">'cdkSampleTable'</span>,
billingMode<span style="color: #666666;">:</span> dd.BillingMode.PROVISIONED,
partitionKey<span style="color: #666666;">:</span> { name<span style="color: #666666;">:</span> <span style="color: #bb4444;">'hashKey'</span>, type<span style="color: #666666;">:</span> dd.AttributeType.STRING },
readCapacity<span style="color: #666666;">:</span> <span style="color: #666666;">5</span>,
writeCapacity<span style="color: #666666;">:5</span>
});
}
}
</pre></td></tr></tbody></table></div>
This is all the code needed to create a Dynamo db table. Compare this to the CloudFormation Template I had to write in the previous post. I can even write tests to ensure the code runs correctly.</div><div>To deploy this code:</div>
<div style="background: rgb(248, 248, 248); border-color: gray; border-image: initial; border-style: solid; border-width: 0.1em 0.1em 0.1em 0.8em; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;"><pre style="line-height: 125%; margin: 0px;"><b>cdk deploy</b>
CdkAppStack<span style="color: #666666;">:</span> deploying...
CdkAppStack<span style="color: #666666;">:</span> creating CloudFormation changeset...
[<span style="border: 1px solid rgb(255, 0, 0);">██████████████████████████████████████████████████████████</span>] (<span style="color: #666666;">3/3</span>)
<span style="border: 1px solid rgb(255, 0, 0);">✅</span> CdkAppStack
Stack ARN<span style="color: #666666;">:</span>
arn<span style="color: #666666;">:</span>aws<span style="color: #666666;">:</span>cloudformation<span style="color: #666666;">:</span>us<span style="color: #666666;">-</span>east<span style="color: #666666;">-1:060315252813:</span>stack<span style="color: #666666;">/</span>CdkAppStack<span style="color: #666666;">/083</span>f6000<span style="color: #666666;">-26</span>d6<span style="color: #666666;">-11</span>eb<span style="color: #666666;">-8629-0</span>a818cac4dad
</pre></div>
Based on the aws configure settings, the Dynamo Table will be setup in the set region and set account.
To see the CloudFormation template generated by the code, cdk provides a command - cdk synth
<div style="background: rgb(248, 248, 248); border-color: gray; border-image: initial; border-style: solid; border-width: 0.1em 0.1em 0.1em 0.8em; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;"><pre style="line-height: 125%; margin: 0px;"><b>cdk synth</b>
Resources:
cdkSampleTable964E91EB:
Type: AWS::DynamoDB::Table
Properties:
KeySchema:
- AttributeName: hashKey
KeyType: HASH
AttributeDefinitions:
- AttributeName: hashKey
AttributeType: S
ProvisionedThroughput:
ReadCapacityUnits: 5
WriteCapacityUnits: 5
TableName: cdkSampleTable
UpdateReplacePolicy: Retain
DeletionPolicy: Retain
Metadata:
aws:cdk:path: CdkAppStack/cdkSampleTable/Resource
CDKMetadata:
Type: AWS::CDK::Metadata
</pre></div>
There was one another setup step that was needed - I needed to install the aws-dynamodb package
<div style="background: rgb(248, 248, 248); border-color: gray; border-image: initial; border-style: solid; border-width: 0.1em 0.1em 0.1em 0.8em; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;"><pre style="line-height: 125%; margin: 0px;"><span><b>npm install @aws-cdk/aws-dynamodb</b></span>
<span style="color: #888888;">npm WARN cdk-app@0.1.0 No repository field.</span>
<span style="color: #888888;">npm WARN cdk-app@0.1.0 No license field.</span>
<span style="color: #888888;">+ @aws-cdk/aws-dynamodb@1.73.0</span>
<span style="color: #888888;">added 20 packages from 1 contributor and audited 762 packages in 7.773s</span>
<span style="color: #888888;">27 packages are looking for funding</span>
<span style="color: #888888;"> run `npm fund` for details</span>
<span style="color: #888888;">found 0 vulnerabilities</span>
</pre></div>
Robin Varghesehttp://www.blogger.com/profile/00841917517097216645noreply@blogger.com7tag:blogger.com,1999:blog-6262128735818493674.post-19605461324949483142020-11-14T02:49:00.003+05:302020-11-14T02:49:30.291+05:30AWS CloudFormation - my first stackBefore I start - CloudFormation is by far one of the weirdest and coolest things to have come out of AWS. It took me some time to grasp it (and am still grasping) but mann its awesome for AWS users. <span><a name='more'></a></span><div><b>Pre CloudFormation</b> - (or all AWS posts to date in this blog) we had to create everything manually - either via AWS Console or AWS CLI. This meant each user running adhoc set of commands that they hoped were not having any unintended consequences. </div><div><b>Enter CloudFormation</b>. With CloudFormation you can define all the AWS resources you want to create in a file. Specifically a JSON or YAML file. This file is what CloudFormation calls a template.</div><div>The file declares all the aws resources we need to create for your application. For example consider the below JSON template:</div>
<div style="background: rgb(248, 248, 248); border-color: gray; border-image: initial; border-style: solid; border-width: 0.1em 0.1em 0.1em 0.8em; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;"><table><tbody><tr><td><pre style="line-height: 125%; margin: 0px;"> 1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70</pre></td><td><pre style="line-height: 125%; margin: 0px;">{
<span style="color: green; font-weight: bold;">"<span style="background-color: #fcff01;">AWSTemplateFormatVersion</span>"</span> : <span style="color: #bb4444;">"2010-09-09"</span>, <span style="color: #2b00fe;"><b>- optional</b></span>
<span style="color: green; font-weight: bold;">"<span style="background-color: #fcff01;">Description</span>"</span> : <span style="color: #bb4444;">"AWS CloudFormation Sample Template DynamoDB_Table: </span></pre><pre style="line-height: 125%; margin: 0px;"><span style="color: #bb4444;"> This template demonstrates creation of DynamoDB table.</span><b style="color: #2b00fe;">- optional</b><br />
<span style="color: green; font-weight: bold;">"Parameters"</span> : { <span style="color: #2b00fe;"><b>- optional, Values to pass to template at runtime,</b></span></pre><pre style="line-height: 125%; margin: 0px;"><span style="color: #2b00fe;"><b> when creating/updating stack</b></span></pre><pre style="line-height: 125%; margin: 0px;"> <span style="color: green; font-weight: bold;">"HashKeyElementName"</span> : {
<span style="color: green; font-weight: bold;">"Description"</span> : <span style="color: #bb4444;">"HashType PrimaryKey Name"</span>,
<span style="color: green; font-weight: bold;">"Type"</span> : <span style="color: #bb4444;">"String"</span>,
<span style="color: green; font-weight: bold;">"AllowedPattern"</span> : <span style="color: #bb4444;">"[a-zA-Z0-9]*"</span>,
<span style="color: green; font-weight: bold;">"MinLength"</span>: <span style="color: #bb4444;">"1"</span>,
<span style="color: green; font-weight: bold;">"MaxLength"</span>: <span style="color: #bb4444;">"2048"</span>,
<span style="color: green; font-weight: bold;">"ConstraintDescription"</span> : <span style="color: #bb4444;">"must contain only alphanumberic characters"</span>
},
<span style="color: green; font-weight: bold;">"HashKeyElementType"</span> : {
<span style="color: green; font-weight: bold;">"Description"</span> : <span style="color: #bb4444;">"HashType PrimaryKey Type"</span>,
<span style="color: green; font-weight: bold;">"Type"</span> : <span style="color: #bb4444;">"String"</span>,
<span style="color: green; font-weight: bold;">"Default"</span> : <span style="color: #bb4444;">"S"</span>,
<span style="color: green; font-weight: bold;">"AllowedPattern"</span> : <span style="color: #bb4444;">"[S|N]"</span>,
<span style="color: green; font-weight: bold;">"MinLength"</span>: <span style="color: #bb4444;">"1"</span>,
<span style="color: green; font-weight: bold;">"MaxLength"</span>: <span style="color: #bb4444;">"1"</span>,
<span style="color: green; font-weight: bold;">"ConstraintDescription"</span> : <span style="color: #bb4444;">"must be either S or N"</span>
},
<span style="color: green; font-weight: bold;">"ReadCapacityUnits"</span> : {
<span style="color: green; font-weight: bold;">"Description"</span> : <span style="color: #bb4444;">"Provisioned read throughput"</span>,
<span style="color: green; font-weight: bold;">"Type"</span> : <span style="color: #bb4444;">"Number"</span>,
<span style="color: green; font-weight: bold;">"Default"</span> : <span style="color: #bb4444;">"5"</span>,
<span style="color: green; font-weight: bold;">"MinValue"</span>: <span style="color: #bb4444;">"5"</span>,
<span style="color: green; font-weight: bold;">"MaxValue"</span>: <span style="color: #bb4444;">"10000"</span>,
<span style="color: green; font-weight: bold;">"ConstraintDescription"</span> : <span style="color: #bb4444;">"must be between 5 and 10000"</span>
},
<span style="color: green; font-weight: bold;">"WriteCapacityUnits"</span> : {
<span style="color: green; font-weight: bold;">"Description"</span> : <span style="color: #bb4444;">"Provisioned write throughput"</span>,
<span style="color: green; font-weight: bold;">"Type"</span> : <span style="color: #bb4444;">"Number"</span>,
<span style="color: green; font-weight: bold;">"Default"</span> : <span style="color: #bb4444;">"10"</span>,
<span style="color: green; font-weight: bold;">"MinValue"</span>: <span style="color: #bb4444;">"5"</span>,
<span style="color: green; font-weight: bold;">"MaxValue"</span>: <span style="color: #bb4444;">"10000"</span>,
<span style="color: green; font-weight: bold;">"ConstraintDescription"</span> : <span style="color: #bb4444;">"must be between 5 and 10000"</span>
}
},
<span style="color: green; font-weight: bold;">"Resources"</span> : { <b><span style="color: #2b00fe;">- required, The stack resources and their properties</span></b></pre><pre style="line-height: 125%; margin: 0px;"><b><span style="color: #2b00fe;"><br /></span></b></pre><pre style="line-height: 125%; margin: 0px;"> '<span style="color: #2b00fe;">myDynamoDBTable' is the Logical ID for the resource being created here</span></pre><pre style="line-height: 125%; margin: 0px;"> <span style="color: green; font-weight: bold;">"myDynamoDBTable"</span> : { <b><span style="color: #2b00fe;">- the resource being created, can be referred </span></b></pre><pre style="line-height: 125%; margin: 0px;"><b><span style="color: #2b00fe;"> in resources and outputs </span></b>
<span style="color: green; font-weight: bold;">"Type"</span> : <span style="color: #bb4444;">"AWS::DynamoDB::Table"</span>, <b><span style="color: #2b00fe;">- Resource type</span></b>
<span style="color: green; font-weight: bold;">"Properties"</span> : { <b><span style="color: #2b00fe;">- Set of properties</span></b>
<span style="color: green; font-weight: bold;">"AttributeDefinitions"</span>: [ {
<span style="color: green; font-weight: bold;">"AttributeName"</span> : {<span style="color: green; font-weight: bold;">"Ref"</span> : <span style="color: #bb4444;">"HashKeyElementName"</span>},
<span style="color: green; font-weight: bold;">"AttributeType"</span> : {<span style="color: green; font-weight: bold;">"Ref"</span> : <span style="color: #bb4444;">"HashKeyElementType"</span>}
} ],
<span style="color: green; font-weight: bold;">"KeySchema"</span>: [
{ <span style="color: green; font-weight: bold;">"AttributeName"</span>: {<span style="color: green; font-weight: bold;">"Ref"</span> : <span style="color: #bb4444;">"HashKeyElementName"</span>}, <span style="color: green; font-weight: bold;">"KeyType"</span>: <span style="color: #bb4444;">"HASH"</span> }
],
<span style="color: green; font-weight: bold;">"ProvisionedThroughput"</span> : {
<span style="color: green; font-weight: bold;">"ReadCapacityUnits"</span> : {<span style="color: green; font-weight: bold;">"Ref"</span> : <span style="color: #bb4444;">"ReadCapacityUnits"</span>},
<span style="color: green; font-weight: bold;">"WriteCapacityUnits"</span> : {<span style="color: green; font-weight: bold;">"Ref"</span> : <span style="color: #bb4444;">"WriteCapacityUnits"</span>}
}
}
}
},
<span style="color: green; font-weight: bold;">"Outputs"</span> : { <b><span style="color: #2b00fe;">- optional, Describes values returned when we view stack's properties</span></b>
<b><span style="color: #2b00fe;">'TableName' is the logical Id used for below output value</span></b></pre><pre style="line-height: 125%; margin: 0px;"><span style="color: green; font-weight: bold;"> "TableName"</span> : {
<span style="color: green; font-weight: bold;">"Value"</span> : {<span style="color: green; font-weight: bold;">"Ref"</span> : <span style="color: #bb4444;">"myDynamoDBTable"</span>}, <b><span style="color: #2b00fe;">Value to return</span></b>
<span style="color: green; font-weight: bold;">"Description"</span> : <span style="color: #bb4444;">"Table name of the newly created DynamoDB table"</span></pre><pre style="line-height: 125%; margin: 0px;"><pre style="line-height: 16.25px; margin-bottom: 0px; margin-top: 0px;"> <span style="color: #2b00fe;"><b>If we want to use this value in a different stack we use the export attribute</b></span></pre> }</pre><pre style="line-height: 125%; margin: 0px;">
}
}
</pre></td></tr></tbody></table></div>
This CloudFormation template (available in <a href="https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/sample-templates-services-us-west-2.html#w2ab1c35c58c13c15" target="_blank">AWS Website</a> BTW) will setup an DynamoDb table. <div><br /></div><div>All the resources that we have listed here together form what is called a <b>stack</b>. In the above example the Stack is composed of a DynamoDb table. I will be using the AWS UI for this.</div><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg5PLpkmMdu_Qxkk4zfEFSwsGsPnkGf-pdNAkUZviXBcLUK3jV4i6FxQvL647zRMXKCi_I4mp7m0ISiOWF6heYBAik7ZUlX0TzUE6QBZaYME7OIffCtp9Wd6wk0BKd9YSlc2bJBaM3an9A/s1306/CFN-Step1.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="738" data-original-width="1306" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg5PLpkmMdu_Qxkk4zfEFSwsGsPnkGf-pdNAkUZviXBcLUK3jV4i6FxQvL647zRMXKCi_I4mp7m0ISiOWF6heYBAik7ZUlX0TzUE6QBZaYME7OIffCtp9Wd6wk0BKd9YSlc2bJBaM3an9A/s16000/CFN-Step1.png" /></a></div>I uploaded the template file from local machine.<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjFQl-Rpfe9MADQ-ZnowOCZ0ofpQm-vH3qv4KgriF6GAdhVmWgOQoMnQ0RnFLbPXd1b-3SjqGiDfQz9EKw-pGGe8fpMYgUjdpV8I7oU2OsJ3H3mV-wd3ry93fXlcnCVood5d1K5U9Q6k9A/s1302/CFN-Step2.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="818" data-original-width="1302" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjFQl-Rpfe9MADQ-ZnowOCZ0ofpQm-vH3qv4KgriF6GAdhVmWgOQoMnQ0RnFLbPXd1b-3SjqGiDfQz9EKw-pGGe8fpMYgUjdpV8I7oU2OsJ3H3mV-wd3ry93fXlcnCVood5d1K5U9Q6k9A/s16000/CFN-Step2.png" /></a></div><div><br /><div>The next step requires us to provide the name for the Stack. Also any parameters specified in template need to be provided here. The next couple of steps can be ignored now and I started the create stack step.</div></div><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi8rmEOyTzO6qkrXH-aTG2-qNg5_VdS7c6S78xtKAVITSvG_WxEIAjjYvGoasO_3L-E60VoIYC7vCqnY4J8t6w3ZTs4ychhmpYomNtzAvsu248SHbPSV5TUy3wV3X_qAhEnok76_a6ELEs/s990/CFN-Events.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="562" data-original-width="990" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi8rmEOyTzO6qkrXH-aTG2-qNg5_VdS7c6S78xtKAVITSvG_WxEIAjjYvGoasO_3L-E60VoIYC7vCqnY4J8t6w3ZTs4ychhmpYomNtzAvsu248SHbPSV5TUy3wV3X_qAhEnok76_a6ELEs/s16000/CFN-Events.png" /></a></div>As seen CloudFormation created the stack with the resource 'myDynamoDbTable'<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhimpzLOkLDUO6jQhRBX2kVT0_gIWmaRaqyZ4w56q-D34BW4n8gydNRx0FjUieaneWEkrtYWMvIadQ966oLM7QzN0zl5OYZ9uhbHTgPVeb4Ene_Z5maqggxu094nIIhckVp2s8J8WtixTw/s1210/CFN-Resources.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="381" data-original-width="1210" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhimpzLOkLDUO6jQhRBX2kVT0_gIWmaRaqyZ4w56q-D34BW4n8gydNRx0FjUieaneWEkrtYWMvIadQ966oLM7QzN0zl5OYZ9uhbHTgPVeb4Ene_Z5maqggxu094nIIhckVp2s8J8WtixTw/s16000/CFN-Resources.png" /></a></div><br /><div>The output from template is the Stack Resource</div><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjlBOA7TkooSv5ejaJkpFWWEuWd3_1S5MTylB19pWFevSnL00Pcdya4pO0h6I-mblZh_silYPa2UMFRInNkAuojFW2zSkKlQgPPnWg9NUYKD_fAHHGW1OHkBg3pok3l0cTtmJJN6AH7QHU/s818/CFN-Output.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="408" data-original-width="818" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjlBOA7TkooSv5ejaJkpFWWEuWd3_1S5MTylB19pWFevSnL00Pcdya4pO0h6I-mblZh_silYPa2UMFRInNkAuojFW2zSkKlQgPPnWg9NUYKD_fAHHGW1OHkBg3pok3l0cTtmJJN6AH7QHU/s16000/CFN-Output.png" /></a></div><br /><div>We have the dynamo table ready and setup to use - <b>Infrastructure as Code</b> is achieved.</div><div><br /><div><br /></div></div>Robin Varghesehttp://www.blogger.com/profile/00841917517097216645noreply@blogger.com3tag:blogger.com,1999:blog-6262128735818493674.post-72240545293558736582020-08-09T01:49:00.000+05:302020-08-09T01:49:24.540+05:30An SNS primer - also Auto Routing SNS messages to different SQS<p> It is a standard decoupling pattern to have publishers send their messages to an SNS owned by them. Consumers will setup SQS endpoints that subscribe to SNS and get the messages. However consider that there are many consumers all of whom are interested in different subset of messages sent by the SNS.<span></span></p><a name='more'></a>
Consider I have an SNS that publishes information about items available for sale. The SNS publishes information about consumables, furniture, outdoor equipment. I have three consumers who are interested in one of these items. Rather then have their SQS filled with all kinds of items, we can configure SQS to only receive the items that we are interested in.<div><b>Step 1: Setup an SNS in AWS</b></div><div>Ill use this opportunity to look deep into the options available for setup</div><div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEimGGC1lgRiAzSYh5vXSNvW2F9CynCNM81v3UHSM7YsW4Ue45lRDkesK7R3-ydvT3sjCg6yPsn6ODe2VzNCRIcBQGyHqDCmo0K-sEvBhDya3kCGSt4QL8f3KtOBfWf3liGs5C1Y-oY_c1o/s1111/SnsSetup.png" style="display: block; padding: 1em 0px;"><img border="0" data-original-height="852" data-original-width="1111" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEimGGC1lgRiAzSYh5vXSNvW2F9CynCNM81v3UHSM7YsW4Ue45lRDkesK7R3-ydvT3sjCg6yPsn6ODe2VzNCRIcBQGyHqDCmo0K-sEvBhDya3kCGSt4QL8f3KtOBfWf3liGs5C1Y-oY_c1o/s640/SnsSetup.png" width="640" /></a></div>
The topic has a name and an optional display name. All message published to the topic are then stored there, before SNS attempts to deliver them to the recipients.
<div style="background: rgb(248, 248, 248); border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;"><pre style="line-height: 125%; margin: 0px;">Amazon SNS provides durable storage of all messages that it receives. When Amazon SNS
receives your Publish request, it stores multiple copies of your message to disk.
Before Amazon SNS confirms to you that it received your request, it stores the message
in multiple isolated locations known as Availability Zones. The message is stored in
Availability Zones that are located within your chosen AWS Region, such as the US East
(N. Virginia) Region. Although rare, should a failure occur in one Availability Zone,
Amazon SNS remains operational, and the durability of your messages persists.
</pre></div><div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjSe9gFiEhX-rXoVsN62N-r6V641j5X0b4prFpEsrkWOAgfsmD6d517RQdcHuOg8o7t-4Dou3JB8CZVx5DywcIkCeWGGv5TsjfJwpKeTIktOqyfKEvl9G_7kXpZt0INnWdNU3ScbnuGfes/s1154/SnsSetup2.png" style="display: block; padding: 1em 0px;"><img border="0" data-original-height="806" data-original-width="1154" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjSe9gFiEhX-rXoVsN62N-r6V641j5X0b4prFpEsrkWOAgfsmD6d517RQdcHuOg8o7t-4Dou3JB8CZVx5DywcIkCeWGGv5TsjfJwpKeTIktOqyfKEvl9G_7kXpZt0INnWdNU3ScbnuGfes/s640/SnsSetup2.png" width="640" /></a></div><div>Access Policy allows us to define who can publish and subscribe to these topics. The topic owner is the AWS Account where we create this SNS Topic.</div><div>'Everyone' is an interesting setting - like making it public consumable. We can also restrict the access to specific AWS accounts. </div><div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhAWCIlHUYPiaPE7kObsXdMpcqoYjbBsNijQd0ugm0qaxrX3kOY8rrXguV8fq2527__7dcF0h16hZUz2lqhWROLsWdVgpW1MSeu0ESiDUkJnEaXX-nwkrMO4WjIv89Q-XQDotXCy02MpwA/s1384/SnsSetup3.png" style="display: block; padding: 1em 0px;"><img border="0" data-original-height="990" data-original-width="1384" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhAWCIlHUYPiaPE7kObsXdMpcqoYjbBsNijQd0ugm0qaxrX3kOY8rrXguV8fq2527__7dcF0h16hZUz2lqhWROLsWdVgpW1MSeu0ESiDUkJnEaXX-nwkrMO4WjIv89Q-XQDotXCy02MpwA/s640/SnsSetup3.png" width="640" /></a></div><div>The next options are related to retry behavior when publishing to SNS<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEisAGnp9HNH-n7t7s4w6b3_rvG6gqxzHH2MtBDnoI-E1IaxIjkd8LFCUocA-X_iBw0jtijYKV5KvXbfb10B68kN6av6qeRFvqUKqongJBXuKCls9YXEFfhQfNIa6Stif0eJPyTlqZlRqFE/s1036/SnsSetup4.png" style="display: block; padding: 1em 0px;"><img border="0" data-original-height="958" data-original-width="1036" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEisAGnp9HNH-n7t7s4w6b3_rvG6gqxzHH2MtBDnoI-E1IaxIjkd8LFCUocA-X_iBw0jtijYKV5KvXbfb10B68kN6av6qeRFvqUKqongJBXuKCls9YXEFfhQfNIa6Stif0eJPyTlqZlRqFE/s640/SnsSetup4.png" width="640" /></a></div></div><div>Last is logging. In this case we have enabled cloud watch logs for SQS publishing. (This means we are paying for CloudWatch)</div><div>I had to create an IAM role to allow SNS access to CloudWatch</div><div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjgEdqdHiH3MTn20e6ERkNggkk_6Yy6n6L-jjR6awuvHHQUq68nsl1jLjiPkLcCm8tLOzGL7zAJzKpM0IHBj5kLP1BH3gyr95M7FCeCl5wpkwfbbV8CeHHaqBuQTUQm009JW7640UDEBSk/s1108/SNSCloudWatchIamRole.png" style="display: block; padding: 1em 0px;"><img border="0" data-original-height="652" data-original-width="1108" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjgEdqdHiH3MTn20e6ERkNggkk_6Yy6n6L-jjR6awuvHHQUq68nsl1jLjiPkLcCm8tLOzGL7zAJzKpM0IHBj5kLP1BH3gyr95M7FCeCl5wpkwfbbV8CeHHaqBuQTUQm009JW7640UDEBSk/s640/SNSCloudWatchIamRole.png" width="640" /></a></div><div><br /></div><div>This completes the creation of SNS.</div><div><br /></div>
<div>Next Step: A Lambda that publishes items to SNS every minute.</div>
<!--HTML generated using hilite.me--><div style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;"><pre style="line-height: 125%; margin: 0px;"><span style="background-color: #f8f8f8; color: #aa22ff; font-weight: bold;">public</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #aa22ff; font-weight: bold;">class</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: blue;">SellerLambda</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #aa22ff; font-weight: bold;">implements</span><span style="background-color: #f8f8f8;"> RequestHandler</span><span style="background-color: #f8f8f8; color: #666666;"><</span><span style="background-color: #f8f8f8;">ScheduledEvent</span><span style="background-color: #f8f8f8; color: #666666;">,</span><span style="background-color: #f8f8f8;"> Void</span><span style="background-color: #f8f8f8; color: #666666;">></span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #666666;">{</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #fcff01;"><span style="color: #aa22ff; font-weight: bold;">private</span> String topicArn <span style="color: #666666;">=</span> <span style="color: #bb4444;">"arn:aws:sns:us-east-1:---------:ItemsToSellSNS"</span></span><span style="background-color: #f8f8f8; color: #666666;">;</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #aa22ff; font-weight: bold;">private</span><span style="background-color: #f8f8f8;"> AmazonSNS snsClient</span><span style="background-color: #f8f8f8; color: #666666;">;</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #aa22ff; font-weight: bold;">public</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #00a000;">SellerLambda</span><span style="background-color: #f8f8f8; color: #666666;">()</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #666666;">{</span><span style="background-color: #f8f8f8;">
AmazonSNSClientBuilder snsClientBuilder </span><span style="background-color: #f8f8f8; color: #666666;">=</span><span style="background-color: #f8f8f8;"> AmazonSNSClientBuilder</span><span style="background-color: #f8f8f8; color: #666666;">.</span><span style="background-color: #f8f8f8; color: #bb4444;">standard</span><span style="background-color: #f8f8f8; color: #666666;">();</span><span style="background-color: #f8f8f8;">
snsClientBuilder</span><span style="background-color: #f8f8f8; color: #666666;">.</span><span style="background-color: #f8f8f8; color: #bb4444;">setClientConfiguration</span><span style="background-color: #f8f8f8; color: #666666;">(</span><span style="background-color: #f8f8f8; color: #aa22ff; font-weight: bold;">new</span><span style="background-color: #f8f8f8;"> ClientConfiguration</span><span style="background-color: #f8f8f8; color: #666666;">());</span><span style="background-color: #f8f8f8;">
snsClientBuilder</span><span style="background-color: #f8f8f8; color: #666666;">.</span><span style="background-color: #f8f8f8; color: #bb4444;">setCredentials</span><span style="background-color: #f8f8f8; color: #666666;">(</span><span style="background-color: #f8f8f8; color: #aa22ff; font-weight: bold;">new</span><span style="background-color: #f8f8f8;"> AWSStaticCredentialsProvider</span><span style="background-color: #f8f8f8; color: #666666;">(</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #aa22ff; font-weight: bold;">new</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #00a000;">BasicAWSCredentials</span><span style="background-color: #f8f8f8; color: #666666;">(</span><span style="background-color: #f8f8f8; color: #bb4444;">"AccessKey"</span><span style="background-color: #f8f8f8; color: #666666;">,</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #bb4444;">"SecretKey"</span><span style="background-color: #f8f8f8; color: #666666;">)));</span><span style="background-color: #f8f8f8;">
snsClientBuilder</span><span style="background-color: #f8f8f8; color: #666666;">.</span><span style="background-color: #f8f8f8; color: #bb4444;">setRegion</span><span style="background-color: #f8f8f8; color: #666666;">(</span><span style="background-color: #f8f8f8;">Regions</span><span style="background-color: #f8f8f8; color: #666666;">.</span><span style="background-color: #f8f8f8; color: #bb4444;">US_EAST_1</span><span style="background-color: #f8f8f8; color: #666666;">.</span><span style="background-color: #f8f8f8; color: #bb4444;">getName</span><span style="background-color: #f8f8f8; color: #666666;">());</span><span style="background-color: #f8f8f8;">
snsClient </span><span style="background-color: #f8f8f8; color: #666666;">=</span><span style="background-color: #f8f8f8;"> snsClientBuilder</span><span style="background-color: #f8f8f8; color: #666666;">.</span><span style="background-color: #f8f8f8; color: #bb4444;">build</span><span style="background-color: #f8f8f8; color: #666666;">();</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #666666;">}</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #aa22ff;">@Override</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #aa22ff; font-weight: bold;">public</span><span style="background-color: #f8f8f8;"> Void </span><span style="background-color: #f8f8f8; color: #00a000;">handleRequest</span><span style="background-color: #f8f8f8; color: #666666;">(</span><span style="background-color: #f8f8f8;">ScheduledEvent input</span><span style="background-color: #f8f8f8; color: #666666;">,</span><span style="background-color: #f8f8f8;"> Context context</span><span style="background-color: #f8f8f8; color: #666666;">)</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #666666;">{</span><span style="background-color: #f8f8f8;">
LambdaLogger logger </span><span style="background-color: #f8f8f8; color: #666666;">=</span><span style="background-color: #f8f8f8;"> context</span><span style="background-color: #f8f8f8; color: #666666;">.</span><span style="background-color: #f8f8f8; color: #bb4444;">getLogger</span><span style="background-color: #f8f8f8; color: #666666;">();</span><span style="background-color: #f8f8f8;">
logger</span><span style="background-color: #f8f8f8; color: #666666;">.</span><span style="background-color: #f8f8f8; color: #bb4444;">log</span><span style="background-color: #f8f8f8; color: #666666;">(</span><span style="background-color: #f8f8f8; color: #bb4444;">"In Handler: Executing "</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #666666;">+</span><span style="background-color: #f8f8f8;"> context</span><span style="background-color: #f8f8f8; color: #666666;">.</span><span style="background-color: #f8f8f8; color: #bb4444;">getFunctionName</span><span style="background-color: #f8f8f8; color: #666666;">()</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #666666;">+</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #bb4444;">", "</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #666666;">+</span><span style="background-color: #f8f8f8;"> context</span><span style="background-color: #f8f8f8; color: #666666;">.</span><span style="background-color: #f8f8f8; color: #bb4444;">getFunctionVersion</span><span style="background-color: #f8f8f8; color: #666666;">()</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #666666;">+</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #bb4444;">", event triggered at "</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #666666;">+</span><span style="background-color: #f8f8f8;"> input</span><span style="background-color: #f8f8f8; color: #666666;">.</span><span style="background-color: #f8f8f8; color: #bb4444;">getTime</span><span style="background-color: #f8f8f8; color: #666666;">());</span><span style="background-color: #f8f8f8;">
logger</span><span style="background-color: #f8f8f8; color: #666666;">.</span><span style="background-color: #f8f8f8; color: #bb4444;">log</span><span style="background-color: #f8f8f8; color: #666666;">(</span><span style="background-color: #f8f8f8;">input</span><span style="background-color: #f8f8f8; color: #666666;">.</span><span style="background-color: #f8f8f8; color: #bb4444;">toString</span><span style="background-color: #f8f8f8; color: #666666;">());</span><span style="background-color: #f8f8f8;">
Set</span><span style="background-color: #f8f8f8; color: #666666;"><</span><span style="background-color: #f8f8f8;">String</span><span style="background-color: #f8f8f8; color: #666666;">></span><span style="background-color: #f8f8f8;"> items </span><span style="background-color: #f8f8f8; color: #666666;">=</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #aa22ff; font-weight: bold;">new</span><span style="background-color: #f8f8f8;"> HashSet</span><span style="background-color: #f8f8f8; color: #666666;"><>();</span><span style="background-color: #f8f8f8;">
items</span><span style="background-color: #f8f8f8; color: #666666;">.</span><span style="background-color: #f8f8f8; color: #bb4444;">add</span><span style="background-color: #f8f8f8; color: #666666;">(</span><span style="background-color: #f8f8f8; color: #bb4444;">"Furniture"</span><span style="background-color: #f8f8f8; color: #666666;">);</span><span style="background-color: #f8f8f8;">
items</span><span style="background-color: #f8f8f8; color: #666666;">.</span><span style="background-color: #f8f8f8; color: #bb4444;">add</span><span style="background-color: #f8f8f8; color: #666666;">(</span><span style="background-color: #f8f8f8; color: #bb4444;">"Outdoors"</span><span style="background-color: #f8f8f8; color: #666666;">);</span><span style="background-color: #f8f8f8;">
items</span><span style="background-color: #f8f8f8; color: #666666;">.</span><span style="background-color: #f8f8f8; color: #bb4444;">add</span><span style="background-color: #f8f8f8; color: #666666;">(</span><span style="background-color: #f8f8f8; color: #bb4444;">"Food"</span><span style="background-color: #f8f8f8; color: #666666;">);</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #aa22ff; font-weight: bold;">for</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #666666;">(</span><span style="background-color: #f8f8f8;">String item </span><span style="background-color: #f8f8f8; color: #666666;">:</span><span style="background-color: #f8f8f8;"> items</span><span style="background-color: #f8f8f8; color: #666666;">)</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #666666;">{</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #aa22ff; font-weight: bold;">final</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #fcff01;">PublishRequest publishRequest <span style="color: #666666;">=</span> <span style="color: #aa22ff; font-weight: bold;">new</span> PublishRequest<span style="color: #666666;">(</span>topicArn<span style="color: #666666;">,</span> item <span style="color: #666666;">+</span> Math<span style="color: #666666;">.</span><span style="color: #bb4444;">random</span><span style="color: #666666;">())</span></span><span style="background-color: #f8f8f8; color: #666666;">;</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #fcff01;">MessageAttributeValue messageAttributeValue <span style="color: #666666;">=</span> <span style="color: #aa22ff; font-weight: bold;">new</span> MessageAttributeValue<span style="color: #666666;">();</span></span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #fcff01;">messageAttributeValue<span style="color: #666666;">.</span><span style="color: #bb4444;">setDataType</span><span style="color: #666666;">(</span><span style="color: #bb4444;">"String"</span><span style="color: #666666;">);</span></span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #fcff01;">messageAttributeValue<span style="color: #666666;">.</span><span style="color: #bb4444;">setStringValue</span><span style="color: #666666;">(</span>item<span style="color: #666666;">);</span></span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #fcff01;">publishRequest<span style="color: #666666;">.</span><span style="color: #bb4444;">addMessageAttributesEntry</span><span style="color: #666666;">(</span><span style="color: #bb4444;">"ITEM_TYPE"</span><span style="color: #666666;">,</span> messageAttributeValue<span style="color: #666666;">);</span></span><span style="background-color: #f8f8f8;">
</span></pre><pre style="line-height: 125%; margin: 0px;"><span style="background-color: #f8f8f8;"> </span><span style="background-color: #fcff01;">PublishResult publishResult <span style="color: #666666;">=</span> snsClient<span style="color: #666666;">.</span><span style="color: #bb4444;">publish</span><span style="color: #666666;">(</span>publishRequest<span style="color: #666666;">);</span></span><span style="background-color: #f8f8f8;">
logger</span><span style="background-color: #f8f8f8; color: #666666;">.</span><span style="background-color: #f8f8f8; color: #bb4444;">log</span><span style="background-color: #f8f8f8; color: #666666;">(</span><span style="background-color: #f8f8f8; color: #bb4444;">"Result of publishing message for item "</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #666666;">+</span><span style="background-color: #f8f8f8;"> item </span><span style="background-color: #f8f8f8; color: #666666;">+</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #bb4444;">" is "</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #666666;">+</span><span style="background-color: #f8f8f8;"> publishResult</span><span style="background-color: #f8f8f8; color: #666666;">);</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #666666;">}</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #aa22ff; font-weight: bold;">return</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #aa22ff; font-weight: bold;">null</span><span style="background-color: #f8f8f8; color: #666666;">;</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #666666;">}</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #666666;">}</span><span style="background-color: #f8f8f8;">
</span></pre></div>
The SNS Client is pretty straight forward - It can work with all topics under the account.<div>It exposes a batch message to publish SNS messages - but in that case the input is plain String messages.</div><div>The method we used here is one that publishes a message at a time. Each message includes message attributes. The publish method returns a response - that can be used to verify successful publish.</div><div>I setup a simple Cloud Watch Rule that triggers the lambda every minute.</div><div>The Cloud Watch logs of one successful run is as below:</div><div><br /></div>
<div style="background: rgb(248, 248, 248); border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;"><pre style="line-height: 125%; margin: 0px;">START RequestId: 14c925ed-c066-4d9d-ba91-36fc466d21a6 Version: $LATEST
In Handler: Executing SellerNotificationsFunction, $LATEST, event triggered at 2020-08-08T18:47:14.000Z
{account: XXXXXXXXXXX,region: us-east-1,detail: {},detailType: Scheduled Event,source: aws.events,id: 6a450b80-fa20-04cc-4f68-f3f793330be7,time: 2020-08-08T18:47:14.000Z,resources: [arn:aws:events:us-east-1:XXXXXXXXXXX:rule/SellerNotificationScheduledTrigger]}
Result of publishing message for item Outdoors is {MessageId: 1944ddbe-94ac-566e-9a47-8a166174abbf}
Result of publishing message for item Furniture is {MessageId: bcab8e23-0329-5b49-bef2-d98bf6f09698}
Result of publishing message for item Food is {MessageId: 23b23ebe-ab06-5f59-a3eb-69165edf9288}
END RequestId: 14c925ed-c066-4d9d-ba91-36fc466d21a6
REPORT RequestId: 14c925ed-c066-4d9d-ba91-36fc466d21a6 Duration: 3592.67 ms Billed Duration: 3600 ms Memory Size: 512 MB Max Memory Used: 135 MB Init Duration: 2082.80 ms
</pre></div>
<b>Next step is to setup SQS queues and configure them to recieve messages from SNS topic.</b><div>I setup an SQS queue without any filtering - this subscribes to all messages from the SNS. However my Cloudwatch logs indicate that there was a delivery failure:</div>
<!--HTML generated using hilite.me--><div style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;"><pre style="line-height: 125%; margin: 0px;"><span style="background-color: #f8f8f8;">{
</span><span style="background-color: #f8f8f8; color: green; font-weight: bold;">"notification"</span><span style="background-color: #f8f8f8;">: {
</span><span style="background-color: #f8f8f8; color: green; font-weight: bold;">"messageMD5Sum"</span><span style="background-color: #f8f8f8;">: </span><span style="background-color: #f8f8f8; color: #bb4444;">"52d7dff0bcb7e0301b55896a038a417e"</span><span style="background-color: #f8f8f8;">,
</span><span style="background-color: #f8f8f8; color: green; font-weight: bold;">"messageId"</span><span style="background-color: #f8f8f8;">: </span><span style="background-color: #f8f8f8; color: #bb4444;">"be9a9c5a-6d07-52a8-bff5-cbf36d9f8360"</span><span style="background-color: #f8f8f8;">,
</span><span style="background-color: #f8f8f8; color: green; font-weight: bold;">"topicArn"</span><span style="background-color: #f8f8f8;">: </span><span style="background-color: #f8f8f8; color: #bb4444;">"arn:aws:sns:us-east-1:XXXXXX:ItemsToSellSNS"</span><span style="background-color: #f8f8f8;">,
</span><span style="background-color: #f8f8f8; color: green; font-weight: bold;">"timestamp"</span><span style="background-color: #f8f8f8;">: </span><span style="background-color: #f8f8f8; color: #bb4444;">"2020-08-08 19:34:48.021"</span><span style="background-color: #f8f8f8;">
},
</span><span style="background-color: #f8f8f8; color: green; font-weight: bold;">"delivery"</span><span style="background-color: #f8f8f8;">: {
</span><span style="background-color: #f8f8f8; color: green; font-weight: bold;">"deliveryId"</span><span style="background-color: #f8f8f8;">: </span><span style="background-color: #f8f8f8; color: #bb4444;">"c85f2b94-3d32-51ce-9945-7c3d3daa6426"</span><span style="background-color: #f8f8f8;">,
</span><span style="background-color: #f8f8f8; color: green; font-weight: bold;">"destination"</span><span style="background-color: #f8f8f8;">: </span><span style="background-color: #f8f8f8; color: #bb4444;">"arn:aws:sqs:us-east-1:XXXXXX:FurnitureQueue"</span><span style="background-color: #f8f8f8;">,
</span><span style="background-color: #f8f8f8; color: green; font-weight: bold;">"providerResponse"</span><span style="background-color: #f8f8f8;">: </span><span style="color: #bb4444;"><span style="background-color: #f8f8f8;">"{</span><span style="background-color: #fcff01;">\"ErrorCode\":\"AccessDenied\",\"ErrorMessage\":\"Access to the resource https://sqs.us-east-1.amazonaws.com/XXXXXX/FurnitureQueue is denied.\",\"sqsRequestId\":\"Unrecoverable\"</span><span style="background-color: #f8f8f8;">}"</span></span><span style="background-color: #f8f8f8;">,
</span><span style="background-color: #f8f8f8; color: green; font-weight: bold;">"dwellTimeMs"</span><span style="background-color: #f8f8f8;">: </span><span style="background-color: #f8f8f8; color: #666666;">34</span><span style="background-color: #f8f8f8;">,
</span><span style="background-color: #f8f8f8; color: green; font-weight: bold;">"attempts"</span><span style="background-color: #f8f8f8;">: </span><span style="background-color: #f8f8f8; color: #666666;">1</span><span style="background-color: #f8f8f8;">,
</span><span style="background-color: #f8f8f8; color: green; font-weight: bold;">"statusCode"</span><span style="background-color: #f8f8f8;">: </span><span style="background-color: #f8f8f8; color: #666666;">403</span><span style="background-color: #f8f8f8;">
},
</span><span style="background-color: #f8f8f8; color: green; font-weight: bold;">"status"</span><span style="background-color: #f8f8f8;">: </span><span style="background-color: #f8f8f8; color: #bb4444;">"FAILURE"</span><span style="background-color: #f8f8f8;">
}
</span></pre></div>
This indicates there was a permission issue. I updated the SQS permission to allow everyone to publish to the SQS. (This enabled SNS to publish messages successfully to SQS)<div><br /></div><div>The next step is to add a attribute filter to the subscription</div><div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgphZ_u90kTubrH7fWmq0QmtorsF6FGx-wTFCTrePbVvz7H1bp-zxNIDkUQLSf8w4XKseSqPnbFJ3US1AX6jg39CepJ7Pi7mdQwDJMmt4AYIEnP6S2iZdObhbvOANzJ9oI5dVOgBM7XTt8/s1322/SQSTopicSubscription.png" style="display: block; padding: 1em 0px;"><img border="0" data-original-height="734" data-original-width="1322" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgphZ_u90kTubrH7fWmq0QmtorsF6FGx-wTFCTrePbVvz7H1bp-zxNIDkUQLSf8w4XKseSqPnbFJ3US1AX6jg39CepJ7Pi7mdQwDJMmt4AYIEnP6S2iZdObhbvOANzJ9oI5dVOgBM7XTt8/s640/SQSTopicSubscription.png" width="640" /></a></div><div>The subscription policy here means a message will be sent to the SQS Queue only if the message has an attribute "ITEM_TYPE" with value "Furniture".</div><div>I verified this from the SNS logs:</div><div>At this point I have two queue </div><div><ul style="text-align: left;"><li>AllItemsQueue - The queue receives all messages from ItemsToSellSNS Topic</li><li>FurnitureQueue - The queue receives only message matching the filter policy from ItemsToSellSNS Topic</li></ul></div><div>-
From Lambda logs:<div><!--HTML generated using hilite.me--><div style="background: rgb(248, 248, 248); border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;"><pre style="line-height: 125%; margin: 0px;">Result of publishing message for item Furniture is
{MessageId: 06810711-2867-5e8b-a528-50b820abc0b4}
Result of publishing message for item Outdoors is
{MessageId: c5f1d357-491a-5e0b-81ae-84fe41d68034}
</pre></div>
I checked the SNS delivery logs and it indicates the Furniture message was sent to two queues and the Outdoors message was sent to a single queue</div></div>
<!-- HTML generated using hilite.me --><div style="background: #f8f8f8; overflow:auto;width:auto;border:solid gray;border-width:.1em .1em .1em .8em;padding:.2em .6em;"><pre style="margin: 0; line-height: 125%">{
<span style="color: #008000; font-weight: bold">"notification"</span>: {
<span style="color: #008000; font-weight: bold">"messageMD5Sum"</span>: <span style="color: #BB4444">"314f159e953ed9c955d3c65504f679c6"</span>,
<span style="color: #008000; font-weight: bold">"messageId"</span>: <span style="color: #BB4444">"06810711-2867-5e8b-a528-50b820abc0b4"</span>,
<span style="color: #008000; font-weight: bold">"topicArn"</span>: <span style="color: #BB4444">"arn:aws:sns:us-east-1:686673491323:ItemsToSellSNS"</span>,
<span style="color: #008000; font-weight: bold">"timestamp"</span>: <span style="color: #BB4444">"2020-08-08 20:09:48.365"</span>
},
<span style="color: #008000; font-weight: bold">"delivery"</span>: {
<span style="color: #008000; font-weight: bold">"deliveryId"</span>: <span style="color: #BB4444">"e5551c8f-511f-5092-919d-4b2659d39cac"</span>,
<span style="color: #008000; font-weight: bold">"destination"</span>: <span style="color: #BB4444">"arn:aws:sqs:us-east-1:686673491323:FurnitureQueue"</span>,
<span style="color: #008000; font-weight: bold">"providerResponse"</span>: <span style="color: #BB4444">"{\"sqsRequestId\":\"81390137-859b-5f33-9596-569e8d573181\",\"sqsMessageId\":\"1cb34c87-fe5e-4c04-a162-88a88d0a0615\"}"</span>,
<span style="color: #008000; font-weight: bold">"dwellTimeMs"</span>: <span style="color: #666666">39</span>,
<span style="color: #008000; font-weight: bold">"attempts"</span>: <span style="color: #666666">1</span>,
<span style="color: #008000; font-weight: bold">"statusCode"</span>: <span style="color: #666666">200</span>
},
<span style="color: #008000; font-weight: bold">"status"</span>: <span style="color: #BB4444">"SUCCESS"</span>
}
{
<span style="color: #008000; font-weight: bold">"notification"</span>: {
<span style="color: #008000; font-weight: bold">"messageMD5Sum"</span>: <span style="color: #BB4444">"376323bb15b81d0b29736a68d46a6b84"</span>,
<span style="color: #008000; font-weight: bold">"messageId"</span>: <span style="color: #BB4444">"c5f1d357-491a-5e0b-81ae-84fe41d68034"</span>,
<span style="color: #008000; font-weight: bold">"topicArn"</span>: <span style="color: #BB4444">"arn:aws:sns:us-east-1:686673491323:ItemsToSellSNS"</span>,
<span style="color: #008000; font-weight: bold">"timestamp"</span>: <span style="color: #BB4444">"2020-08-08 20:09:48.344"</span>
},
<span style="color: #008000; font-weight: bold">"delivery"</span>: {
<span style="color: #008000; font-weight: bold">"deliveryId"</span>: <span style="color: #BB4444">"e4dcdc25-c3c0-5d00-96bf-8b8ccb0c1f97"</span>,
<span style="color: #008000; font-weight: bold">"destination"</span>: <span style="color: #BB4444">"arn:aws:sqs:us-east-1:686673491323:AllItemsQueue"</span>,
<span style="color: #008000; font-weight: bold">"providerResponse"</span>: <span style="color: #BB4444">"{\"sqsRequestId\":\"8a1a41c1-8d06-5f9a-ad6b-166885caeb05\",\"sqsMessageId\":\"a19ec52f-32c8-4797-87e2-400bd5664b5d\"}"</span>,
<span style="color: #008000; font-weight: bold">"dwellTimeMs"</span>: <span style="color: #666666">52</span>,
<span style="color: #008000; font-weight: bold">"attempts"</span>: <span style="color: #666666">1</span>,
<span style="color: #008000; font-weight: bold">"statusCode"</span>: <span style="color: #666666">200</span>
},
<span style="color: #008000; font-weight: bold">"status"</span>: <span style="color: #BB4444">"SUCCESS"</span>
}
</pre></div>
The first one as seen went to both queues while the second message was only delivered to AllItemsQueueRobin Varghesehttp://www.blogger.com/profile/00841917517097216645noreply@blogger.com1tag:blogger.com,1999:blog-6262128735818493674.post-24559280199166741072020-07-07T17:12:00.004+05:302020-07-07T17:12:35.008+05:30Amazon ECS : My first AWS container - Part 2<div dir="ltr" style="text-align: left;" trbidi="on">
In the previous post we created a container image for our code and pushed the image onto ECR. In this post, we will setup an ECS application<br />
<a name='more'></a>Going back to ECS console, the first step is to configure the Container definition:<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhlYvTy3iez0m3ttup0CPOlQIrPpdwjjUiiBM_B08PPkMaepH3p2T64aoOaxd4GPk5JGaScf0NHrcGsLxOhjugBC4SQjgSG4YL7uu47y02H3FfOcaD-xopahGPV5mzrYjk1_oTupbYi8lo/s1600/ContainerDefn.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="525" data-original-width="671" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhlYvTy3iez0m3ttup0CPOlQIrPpdwjjUiiBM_B08PPkMaepH3p2T64aoOaxd4GPk5JGaScf0NHrcGsLxOhjugBC4SQjgSG4YL7uu47y02H3FfOcaD-xopahGPV5mzrYjk1_oTupbYi8lo/s1600/ContainerDefn.png" /></a></div>
The Container image is from the ECR. I upgraded memory requirements to 256 MB. Also did not add any ports since there is no communication here.<br />
The next steps were approved as is:
<br />
<div style="background: #f8f8f8; border-width: 0.1em 0.1em 0.1em 0.8em; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="line-height: 125%; margin: 0;">
A task definition is a blueprint for your application, and describes one or more
containers through attributes. Some attributes are configured at the task level but
the majority of attributes are configured per container.
</pre>
</div>
The default task definition was what I used:<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEigVemaq6HIEXH0Lv3_HT4PiQjH5B47wa87fBcMHmfDGvjajbMxnm94GCNfC8LDn3tPLJJunVFj5hCeSsaxTgNjaFqrtntqFo6nYzfly_nyr7xF46pM77H21UErQCcdkMHOmfCkE5l6NkE/s1600/TaskDefnECS.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="252" data-original-width="468" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEigVemaq6HIEXH0Lv3_HT4PiQjH5B47wa87fBcMHmfDGvjajbMxnm94GCNfC8LDn3tPLJJunVFj5hCeSsaxTgNjaFqrtntqFo6nYzfly_nyr7xF46pM77H21UErQCcdkMHOmfCkE5l6NkE/s1600/TaskDefnECS.png" /></a></div>
Next is the service definition:<br />
<div style="background: #f8f8f8; border-width: 0.1em 0.1em 0.1em 0.8em; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="line-height: 125%; margin: 0;">
A service allows you to run and maintain a specified number (the "desired count")
of simultaneous instances of a task definition in an ECS cluster.
</pre>
</div>
Here too I went with the defaults:
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjh21OWMHsou0ZnlQq-19tt3Nz0Pge616dYy5vaJ7lh_WuEcbmkxTikCtDx7-g1UMGcg_6jA9yyK1IM_psF9UJIU1kGOmzzo5WEWyGIDx6MbEXj1dOFKJ6e6fYL-kztlgObIdCGNmR1OWo/s1600/ServiceDefnEcs.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="331" data-original-width="871" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjh21OWMHsou0ZnlQq-19tt3Nz0Pge616dYy5vaJ7lh_WuEcbmkxTikCtDx7-g1UMGcg_6jA9yyK1IM_psF9UJIU1kGOmzzo5WEWyGIDx6MbEXj1dOFKJ6e6fYL-kztlgObIdCGNmR1OWo/s1600/ServiceDefnEcs.png" /></a></div>
<br />
The last part is the cluster definition:
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgx0c3yc48g5tBZcDLYDd1CNdQ5LyFuNbhvqzZkMKWLmN3UScYb8pajSyP64FXFjZo5VCYjj7JJOBIa1Tgz9fw1GncvrbeNEqskwl3jDp_9DPfA4Bf2BbU96PGzoF0pLbEBspaOoK1blJ8/s1600/ClusterDefn.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="301" data-original-width="859" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgx0c3yc48g5tBZcDLYDd1CNdQ5LyFuNbhvqzZkMKWLmN3UScYb8pajSyP64FXFjZo5VCYjj7JJOBIa1Tgz9fw1GncvrbeNEqskwl3jDp_9DPfA4Bf2BbU96PGzoF0pLbEBspaOoK1blJ8/s1600/ClusterDefn.png" /></a></div>
We used a Fargate cluster<br />
With this the ECS cluster setup is complete.<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEggNilYXKqqQaUbe1tJAicYj99007Va94Kg4zqZWhOipzIU9ZA0_4KDvXkcejOqu-pmNxtGfDJaGt2OCrZ9o3AhM5P3id3eb7-YLvUINo2aKCHtXz7TSjZegF4tW0f3614Lc25j-ryIzx0/s1600/EcsSetup.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="675" data-original-width="692" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEggNilYXKqqQaUbe1tJAicYj99007Va94Kg4zqZWhOipzIU9ZA0_4KDvXkcejOqu-pmNxtGfDJaGt2OCrZ9o3AhM5P3id3eb7-YLvUINo2aKCHtXz7TSjZegF4tW0f3614Lc25j-ryIzx0/s1600/EcsSetup.png" /></a></div>
As can be seen a new log group was setup along with new VPC, subnets and security group.<br />
My container is now up and running:<br />
<!-- HTML generated using hilite.me --><br />
<div style="background: #f8f8f8; border-width: 0.1em 0.1em 0.1em 0.8em; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="line-height: 125%; margin: 0;">No messages found in queue
Running a check ...
</pre>
</div>
<br /></div>
Robin Varghesehttp://www.blogger.com/profile/00841917517097216645noreply@blogger.com3tag:blogger.com,1999:blog-6262128735818493674.post-87659312857885049692020-07-07T06:53:00.002+05:302020-07-07T06:53:53.008+05:30Amazon ECS : My first AWS container - Part 1<div dir="ltr" style="text-align: left;" trbidi="on">
I had done a post sometime back on <a href="https://learningviacode.blogspot.com/2020/06/container-deployments-some-basic.html" target="_blank">Container basics and AWS</a>. Here I will explore ECS - another Container offering from AWS<br />
<a name='more'></a>In Pre Container days when Virtual Machines was the craze, AWS provided EC2 which where virtual hosts available for customers to scale and use without worrying about the underlying physical hardware. To launch an application customers had to create the deployment environment for the application. For example to launch a Java application:<br />
<br />
<ol style="text-align: left;">
<li>Select the Amazon Machine Image (the O.S. to install on EC2 instance)</li>
<li>Launch the Ec2 instance. </li>
<li>Install necessary software (E.g. if the default java version is not what we need, install the correct version). Install Tomcat (for a web application)</li>
<li>Copy your code to the correct location (The jar or war file)</li>
<li>Run the application (Jar command or launch Tomcat)</li>
</ol>
<div>
This is not bad. Infact it is similar to steps you would do to launch any server instance. Once containers arrived, people began to run containers on the EC2 instances. This allowed them to leverage the container benefits on AWS. Users would now setup an Ec2 instance and then install Docker on it.</div>
<div>
They would then deploy docker containers and manage them. (Check this <a href="https://docs.aws.amazon.com/AmazonECS/latest/developerguide/docker-basics.html#install_docker" target="_blank">link</a> on running Docker on Ec2)</div>
<div>
Companies could now have hundreds of EC2 instances, each running hundreds of containers if they needed. </div>
<div>
Though users got container benefits in AWS, it came with the added responsibilities of managing the containers and the EC2 instances.</div>
<div>
This is where Amazon ECS comes in. [<a href="https://aws.amazon.com/ecs/" target="_blank">AWS Docs</a>]</div>
<div style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="line-height: 125%; margin: 0px;"><span style="background-color: #f8f8f8;">Amazon Elastic Container Service (Amazon ECS) is a </span><span style="background-color: yellow;">fully managed container
orchestration service</span><span style="background-color: #f8f8f8;">.You can choose to run your ECS clusters using AWS
Fargate, which is serverless compute for containers. </span><span style="background-color: yellow;">Fargate removes the need
to provision and manage servers</span><span style="background-color: #f8f8f8;">, lets you specify and pay for resources per
application, and improves security through application isolation by design.
</span></pre>
</div>
Thus with a Fargate and ECS combination we can be free of management of EC2 instances and Containers, allowing us to focus more on application development.
<br />
In this post I will create a simple Container application - one that reads from a SQS Queue and inserts record into Dynamo.
<!-- HTML generated using hilite.me --><br />
<div style="background: #f8f8f8; border-width: 0.1em 0.1em 0.1em 0.8em; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="line-height: 125%; margin: 0;"><span style="color: #aa22ff; font-weight: bold;">public</span> <span style="color: #aa22ff; font-weight: bold;">class</span> <span style="color: blue;">SqsConsumer</span> <span style="color: #666666;">{</span>
<span style="color: #aa22ff; font-weight: bold;">private</span> <span style="color: #aa22ff; font-weight: bold;">static</span> AmazonSQS sqsClient<span style="color: #666666;">;</span>
<span style="color: #aa22ff; font-weight: bold;">private</span> <span style="color: #aa22ff; font-weight: bold;">static</span> DynamoDB dbClient<span style="color: #666666;">;</span>
<span style="color: #aa22ff; font-weight: bold;">private</span> <span style="color: #aa22ff; font-weight: bold;">static</span> <span style="color: #aa22ff; font-weight: bold;">final</span> String queueUrl <span style="color: #666666;">=</span> <span style="color: #bb4444;">"https://sqs.us-east-1.amazonaws.com/XXXX/CollectionQueue"</span><span style="color: #666666;">;</span>
<span style="color: #aa22ff; font-weight: bold;">public</span> <span style="color: #aa22ff; font-weight: bold;">static</span> <span style="color: #00bb00; font-weight: bold;">void</span> <span style="color: #00a000;">initialize</span><span style="color: #666666;">()</span> <span style="color: #666666;">{</span>
System<span style="color: #666666;">.</span><span style="color: #bb4444;">out</span><span style="color: #666666;">.</span><span style="color: #bb4444;">println</span><span style="color: #666666;">(</span><span style="color: #bb4444;">"Initializing code..."</span><span style="color: #666666;">);</span>
AWSCredentials awsCredentials <span style="color: #666666;">=</span> <span style="color: #aa22ff; font-weight: bold;">new</span> BasicAWSCredentials<span style="color: #666666;">(</span><span style="color: #bb4444;">"AccessKey"</span><span style="color: #666666;">,</span>
<span style="color: #bb4444;">"SecretKey"</span><span style="color: #666666;">);</span>
AWSCredentialsProvider awsCredentialsProvider <span style="color: #666666;">=</span> <span style="color: #aa22ff; font-weight: bold;">new</span> AWSStaticCredentialsProvider<span style="color: #666666;">(</span>awsCredentials<span style="color: #666666;">);</span>
dbClient <span style="color: #666666;">=</span> <span style="color: #aa22ff; font-weight: bold;">new</span> DynamoDB<span style="color: #666666;">(</span>AmazonDynamoDBClientBuilder<span style="color: #666666;">.</span><span style="color: #bb4444;">standard</span><span style="color: #666666;">()</span>
<span style="color: #666666;">.</span><span style="color: #bb4444;">withRegion</span><span style="color: #666666;">(</span><span style="color: #bb4444;">"us-east-1"</span><span style="color: #666666;">)</span>
<span style="color: #666666;">.</span><span style="color: #bb4444;">withCredentials</span><span style="color: #666666;">(</span>awsCredentialsProvider<span style="color: #666666;">).</span><span style="color: #bb4444;">build</span><span style="color: #666666;">());</span>
sqsClient <span style="color: #666666;">=</span> AmazonSQSClientBuilder<span style="color: #666666;">.</span><span style="color: #bb4444;">standard</span><span style="color: #666666;">()</span>
<span style="color: #666666;">.</span><span style="color: #bb4444;">withRegion</span><span style="color: #666666;">(</span>Regions<span style="color: #666666;">.</span><span style="color: #bb4444;">US_EAST_1</span><span style="color: #666666;">)</span>
<span style="color: #666666;">.</span><span style="color: #bb4444;">withCredentials</span><span style="color: #666666;">(</span>awsCredentialsProvider<span style="color: #666666;">).</span><span style="color: #bb4444;">build</span><span style="color: #666666;">();</span>
System<span style="color: #666666;">.</span><span style="color: #bb4444;">out</span><span style="color: #666666;">.</span><span style="color: #bb4444;">println</span><span style="color: #666666;">(</span><span style="color: #bb4444;">"Initializing complete"</span><span style="color: #666666;">);</span>
<span style="color: #666666;">}</span>
<span style="color: #aa22ff; font-weight: bold;">public</span> <span style="color: #aa22ff; font-weight: bold;">static</span> <span style="color: #00bb00; font-weight: bold;">void</span> <span style="color: #00a000;">main</span><span style="color: #666666;">(</span>String<span style="color: #666666;">[]</span> args<span style="color: #666666;">)</span> <span style="color: #666666;">{</span>
initialize<span style="color: #666666;">();</span>
Timer timer <span style="color: #666666;">=</span> <span style="color: #aa22ff; font-weight: bold;">new</span> Timer<span style="color: #666666;">();</span>
<span style="color: #aa22ff; font-weight: bold;">final</span> Table table <span style="color: #666666;">=</span> dbClient<span style="color: #666666;">.</span><span style="color: #bb4444;">getTable</span><span style="color: #666666;">(</span><span style="color: #bb4444;">"Messages"</span><span style="color: #666666;">);</span>
timer<span style="color: #666666;">.</span><span style="color: #bb4444;">scheduleAtFixedRate</span><span style="color: #666666;">(</span><span style="color: #aa22ff; font-weight: bold;">new</span> TimerTask<span style="color: #666666;">()</span> <span style="color: #666666;">{</span>
<span style="color: #aa22ff;">@Override</span>
<span style="color: #aa22ff; font-weight: bold;">public</span> <span style="color: #00bb00; font-weight: bold;">void</span> <span style="color: #00a000;">run</span><span style="color: #666666;">()</span> <span style="color: #666666;">{</span>
System<span style="color: #666666;">.</span><span style="color: #bb4444;">out</span><span style="color: #666666;">.</span><span style="color: #bb4444;">println</span><span style="color: #666666;">(</span><span style="color: #bb4444;">"Running a check ..."</span><span style="color: #666666;">);</span>
ReceiveMessageResult receiveMessageResult <span style="color: #666666;">=</span> sqsClient<span style="color: #666666;">.</span><span style="color: #bb4444;">receiveMessage</span><span style="color: #666666;">(</span>queueUrl<span style="color: #666666;">);</span>
List<span style="color: #666666;"><</span>Message<span style="color: #666666;">></span> messages <span style="color: #666666;">=</span> receiveMessageResult<span style="color: #666666;">.</span><span style="color: #bb4444;">getMessages</span><span style="color: #666666;">();</span>
<span style="color: #aa22ff; font-weight: bold;">if</span> <span style="color: #666666;">(</span>messages<span style="color: #666666;">.</span><span style="color: #bb4444;">isEmpty</span><span style="color: #666666;">())</span> <span style="color: #666666;">{</span>
System<span style="color: #666666;">.</span><span style="color: #bb4444;">out</span><span style="color: #666666;">.</span><span style="color: #bb4444;">println</span><span style="color: #666666;">(</span><span style="color: #bb4444;">"No messages found in queue"</span><span style="color: #666666;">);</span>
<span style="color: #666666;">}</span> <span style="color: #aa22ff; font-weight: bold;">else</span> <span style="color: #666666;">{</span>
System<span style="color: #666666;">.</span><span style="color: #bb4444;">out</span><span style="color: #666666;">.</span><span style="color: #bb4444;">println</span><span style="color: #666666;">(</span>messages<span style="color: #666666;">.</span><span style="color: #bb4444;">size</span><span style="color: #666666;">()</span> <span style="color: #666666;">+</span> <span style="color: #bb4444;">" Messages found in Queue"</span><span style="color: #666666;">);</span>
<span style="color: #aa22ff; font-weight: bold;">for</span> <span style="color: #666666;">(</span>Message message <span style="color: #666666;">:</span> messages<span style="color: #666666;">)</span> <span style="color: #666666;">{</span>
Item item <span style="color: #666666;">=</span> <span style="color: #aa22ff; font-weight: bold;">new</span> Item<span style="color: #666666;">()</span>
<span style="color: #666666;">.</span><span style="color: #bb4444;">withPrimaryKey</span><span style="color: #666666;">(</span><span style="color: #bb4444;">"messageId"</span><span style="color: #666666;">,</span> message<span style="color: #666666;">.</span><span style="color: #bb4444;">getMessageId</span><span style="color: #666666;">())</span>
<span style="color: #666666;">.</span><span style="color: #bb4444;">withString</span><span style="color: #666666;">(</span><span style="color: #bb4444;">"message"</span><span style="color: #666666;">,</span> message<span style="color: #666666;">.</span><span style="color: #bb4444;">getBody</span><span style="color: #666666;">());</span>
PutItemOutcome putItemOutcome <span style="color: #666666;">=</span> table<span style="color: #666666;">.</span><span style="color: #bb4444;">putItem</span><span style="color: #666666;">(</span>item<span style="color: #666666;">);</span>
System<span style="color: #666666;">.</span><span style="color: #bb4444;">out</span><span style="color: #666666;">.</span><span style="color: #bb4444;">println</span><span style="color: #666666;">(</span>putItemOutcome<span style="color: #666666;">);</span>
sqsClient<span style="color: #666666;">.</span><span style="color: #bb4444;">deleteMessage</span><span style="color: #666666;">(</span>queueUrl<span style="color: #666666;">,</span> message<span style="color: #666666;">.</span><span style="color: #bb4444;">getReceiptHandle</span><span style="color: #666666;">());</span>
<span style="color: #666666;">}</span>
<span style="color: #666666;">}</span>
<span style="color: #666666;">}</span>
<span style="color: #666666;">},</span> <span style="color: #666666;">3</span><span style="color: #008800; font-style: italic;">/*milliseonds*/</span><span style="color: #666666;">,</span> <span style="color: #666666;">100</span><span style="color: #008800; font-style: italic;">/*milliseonds*/</span><span style="color: #666666;">);</span>
<span style="color: #666666;">}</span>
<span style="color: #666666;">}</span>
</pre>
</div>
The class uses a Timer task which execute a code block every 100 milliseconds. The code fetches SQS messages and then adds them to a dynamo table.<br />
I used maven-shade-plugin to create a single uber jar.
<br />
<div style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="line-height: 125%; margin: 0px;"><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: green; font-weight: bold;"><plugins></span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: green; font-weight: bold;"><plugin></span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: green; font-weight: bold;"><groupId></span><span style="background-color: #f8f8f8;">org.apache.maven.plugins</span><span style="background-color: #f8f8f8; color: green; font-weight: bold;"></groupId></span><span style="background-color: #f8f8f8;">
</span><span style="background-color: yellow;"><span style="color: green; font-weight: bold;"><artifactId></span>maven-shade-plugin<span style="color: green; font-weight: bold;"></artifactId></span></span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: green; font-weight: bold;"><version></span><span style="background-color: #f8f8f8;">3.2.2</span><span style="background-color: #f8f8f8; color: green; font-weight: bold;"></version></span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: green; font-weight: bold;"><configuration></span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: green; font-weight: bold;"><createDependencyReducedPom></span><span style="background-color: #f8f8f8;">false</span><span style="background-color: #f8f8f8; color: green; font-weight: bold;"></createDependencyReducedPom></span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: green; font-weight: bold;"></configuration></span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: green; font-weight: bold;"><executions></span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: green; font-weight: bold;"><execution></span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: green; font-weight: bold;"><phase></span><span style="background-color: #f8f8f8;">package</span><span style="background-color: #f8f8f8; color: green; font-weight: bold;"></phase></span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: green; font-weight: bold;"><goals></span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: green; font-weight: bold;"><goal></span><span style="background-color: #f8f8f8;">shade</span><span style="background-color: #f8f8f8; color: green; font-weight: bold;"></goal></span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: green; font-weight: bold;"></goals></span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: green; font-weight: bold;"></execution></span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: green; font-weight: bold;"></executions></span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: green; font-weight: bold;"></plugin></span><span style="background-color: #f8f8f8;">
</span></pre>
</div>
I also set the jar name to be 'sqsconsumer'
<br />
<div style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="line-height: 125%; margin: 0px;"><span style="background-color: #f8f8f8; color: green; font-weight: bold;"><plugin></span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: green; font-weight: bold;"><groupId></span><span style="background-color: #f8f8f8;">org.apache.maven.plugins</span><span style="background-color: #f8f8f8; color: green; font-weight: bold;"></groupId></span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: green; font-weight: bold;"><artifactId></span><span style="background-color: #f8f8f8;">maven-jar-plugin</span><span style="background-color: #f8f8f8; color: green; font-weight: bold;"></artifactId></span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: green; font-weight: bold;"><configuration></span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: green; font-weight: bold;"><archive></span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: green; font-weight: bold;"><manifest></span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: green; font-weight: bold;"><addClasspath></span><span style="background-color: #f8f8f8;">true</span><span style="background-color: #f8f8f8; color: green; font-weight: bold;"></addClasspath></span><span style="background-color: #f8f8f8;">
</span><span style="background-color: yellow;"><span style="color: green; font-weight: bold;"><mainClass></span>org.learningviacode.SqsConsumer<span style="color: green; font-weight: bold;"></mainClass></span></span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: green; font-weight: bold;"></manifest></span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: green; font-weight: bold;"></archive></span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: green; font-weight: bold;"></configuration></span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: green; font-weight: bold;"></plugin></span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: green; font-weight: bold;"></plugins></span><span style="background-color: #f8f8f8;">
</span><span style="background-color: yellow;"><span style="color: green; font-weight: bold;"><finalName></span>sqsconsumer<span style="color: green; font-weight: bold;"></finalName></span></span><span style="background-color: #f8f8f8;">
</span></pre>
</div>
The jar creation plugin can also be instructed to add details to manifest file - This ensures my uber jar is an executable
<br />
This ensures that I now have an executable jar. Next step is to create a Docker Image for use in the container.<br />
For this I added a Dockerfile in the project root with below details:
<br />
<div style="background: #f8f8f8; border-width: 0.1em 0.1em 0.1em 0.8em; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="line-height: 125%; margin: 0;">FROM openjdk:8
ADD target/sqsconsumer.jar sqsconsumer.jar
ENTRYPOINT ["java", "-jar", "sqsconsumer.jar"]
</pre>
</div>
The file instructions are:
<br />
<br />
<ol style="text-align: left;">
<li>Use the openjdk container image with tag 8 (or JDK 1.8)</li>
<li>copy the jar into the container</li>
<li>Execute the command "java -jar sqsconsumer.jar" in the Container</li>
</ol>
<div>
Now I am ready to execute this Container in ECS. (I already tested this Image by deploying locally and verifying the working)</div>
<div>
<br /></div>
<div>
The ECS setup page took me to "Getting Started with Amazon Elastic Container Service (Amazon ECS) using Fargate" - I guess EC2 option is not encouraged anymore.</div>
<div>
The setup process has the below steps:</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgGBT5Dkuwym43fPyqVJmwmXAH2y4JPpT625xqiYZb8fFQobsLFe0VQhLg_Bjq1BYwPZyzvRoLc2U5Ye_n2_atJ561sTRiHPCkXQq90MtLCr9Ak_vA_OvN5x3Z2vvQ50e8LJOdJ4ITS4ro/s1600/ECSObjects.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="300" data-original-width="492" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgGBT5Dkuwym43fPyqVJmwmXAH2y4JPpT625xqiYZb8fFQobsLFe0VQhLg_Bjq1BYwPZyzvRoLc2U5Ye_n2_atJ561sTRiHPCkXQq90MtLCr9Ak_vA_OvN5x3Z2vvQ50e8LJOdJ4ITS4ro/s1600/ECSObjects.png" /></a></div>
<div>
<br /></div>
<div>
The first step is selecting a Container Image (Container Definition). AWS provides some default container images, but in this case, I wanted to use my custom image. </div>
<div>
To use the generated image, I need to upload it to a <b>Container Registry</b></div>
<div style="background: #f8f8f8; border-width: 0.1em 0.1em 0.1em 0.8em; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="line-height: 125%; margin: 0;">The Registry is a stateless, highly scalable server side application that stores and lets you distribute Docker images.
</pre>
</div>
Docker built an open source app that can be use to store Container Images. They also deployed a version of the app - <a href="https://hub.docker.com/" target="_blank">Docker Hub</a>. However I do not want to upload my container here. Primarily because the very basic code includes my Account credentials and I do not want to expose them out there.<br />
I also do not want to setup my own container registry to solve for this post. So how do I get secure private storage for my Container Images ?<br />
AWS comes to the rescue here. They have setup a Container Registry Service called ECR
<br />
<div style="background: #f8f8f8; border-width: 0.1em 0.1em 0.1em 0.8em; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="line-height: 125%; margin: 0;">Amazon Elastic Container Registry (Amazon ECR) is a managed AWS Docker registry
service that is secure, scalable, and reliable. Amazon ECR supports private Docker
repositories with resource-based permissions using AWS IAM so that specific users
or Amazon EC2 instances can access repositories and images. Developers can use the
Docker CLI to push, pull, and manage images.
</pre>
</div>
Great - so my next step is <b>to upload my Container Image to ECR.
</b><br />
I will do this using the aws cli<br />
<!-- HTML generated using hilite.me --><br />
<div style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="line-height: 125%; margin: 0px;"><span style="background-color: #f8f8f8;">> docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
</span><span style="background-color: yellow;">sqs-consumer</span><span style="background-color: #f8f8f8;"> latest cc34b3c2741b 16 hours ago 519MB
docker-spring-boot latest 904768e87c3c 18 hours ago 527MB
getting-started latest 20b9e72c44e8 45 hours ago 179MB
node 12-alpine 057fa4cc38c2 6 days ago 89.3MB
docker/getting-started latest 73f5385a001d 7 days ago 25.1MB
openjdk 8 b190ad78b520 3 weeks ago 510MB
> aws ecr create-repository --repository-name sqs-consumer --region us-east-1
{
"repository": {
"repositoryArn": "arn:aws:ecr:us-east-1:123456789012:repository/sqs-consumer",
"registryId": "123456789012",
"repositoryName": "sqs-consumer",
"repositoryUri": "123456789012.dkr.ecr.us-east-1.amazonaws.com/sqs-consumer",
"createdAt": "2020-07-06T15:33:31-07:00",
"imageTagMutability": "MUTABLE",
"imageScanningConfiguration": {
"scanOnPush": false
}
}
}
> aws ecr describe-repositories --region us-east-1
{
"repositories": [
{
"repositoryArn": "arn:aws:ecr:us-east-1:123456789012:repository/samples/java",
"registryId": "123456789012",
"repositoryName": "samples/java",
"repositoryUri": "123456789012.dkr.ecr.us-east-1.amazonaws.com/samples/java",
"createdAt": "2020-07-06T00:10:46-07:00",
"imageTagMutability": "MUTABLE",
"imageScanningConfiguration": {
"scanOnPush": true
}
},
{
"repositoryArn": "arn:aws:ecr:us-east-1:123456789012:repository/</span><span style="background-color: yellow;">sqs-consumer</span><span style="background-color: #f8f8f8;">",
"registryId": "123456789012",
"repositoryName": "sqs-consumer",
"repositoryUri": "123456789012.dkr.ecr.us-east-1.amazonaws.com/sqs-consumer",
"createdAt": "2020-07-06T15:33:31-07:00",
"imageTagMutability": "MUTABLE",
"imageScanningConfiguration": {
"scanOnPush": false
}
}
]
}
> aws ecr get-login-password --region us-east-1 | docker login --username AWS --password-stdin 123456789012.dkr.ecr.us-east-1.amazonaws.com
Login Succeeded
> docker tag sqs-consumer:latest 123456789012.dkr.ecr.us-east-1.amazonaws.com/sqs-consumer:latest
> docker push 123456789012.dkr.ecr.us-east-1.amazonaws.com/sqs-consumer:latest
The push refers to repository [123456789012.dkr.ecr.us-east-1.amazonaws.com/sqs-consumer]
cf667b61818e: Pushed
cb8e2372b23c: Pushed
68bb2d422178: Pushed
f5181c7ef902: Pushed
2e5b4ca91984: Pushed
527ade4639e0: Pushed
c2c789d2d3c5: Pushed
8803ef42039d: Pushed
latest: digest: sha256:0263548c54967d070dfc48001e4ed1e4bb2030ea376ddd0c6d6c7b45001322ce size: 2005
</span></pre>
</div>
There is a whole lot of commands here:
<br />
<br />
<ol style="text-align: left;">
<li>The first command list the details of images on my docker installation.We can see sqs-consumer in the list</li>
<li> The next command creates a container registry in my selected region. (I executed aws configure before this command)</li>
<li>'aws ecr describe-repositories' command lists the repositories in ECR. The result shows the one that I just created in previous step.</li>
<li>The next command 'aws ecr get-login-password' is interesting. Amazon ECR CLI provides its own API to push and pull images, while also supporting docker commands to push/pull images from repositores. The Docker CLI does not support native IAM authentication methods. So this command is executed to allow Amazon ECR to authenticate and authorize Docker push and pull requests.</li>
<li>Post this we execute the docker tag command. Here we have tagged our current image with latest tag.</li>
<li>The next step is to push the image to the registry.</li>
</ol>
<div>
Post execution, I can see the image in ECR:</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEisIBM9GPos-HTq08z2j4IMBvsGVGuxHnpG78i8NHh4IFgQcsUeR-9tLAdYMW-aKL4J3SVa6rOOE0wMdh9rGlTKVk9-KyyjQIqqj8d8GYJBCHD49HeFGPQWdB-8-_dfukibielnNN0kzkw/s1600/ECRImage.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="299" data-original-width="714" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEisIBM9GPos-HTq08z2j4IMBvsGVGuxHnpG78i8NHh4IFgQcsUeR-9tLAdYMW-aKL4J3SVa6rOOE0wMdh9rGlTKVk9-KyyjQIqqj8d8GYJBCHD49HeFGPQWdB-8-_dfukibielnNN0kzkw/s1600/ECRImage.png" /></a></div>
<div>
In the next post, I will continue with setting up my ECS solution.</div>
</div>
Robin Varghesehttp://www.blogger.com/profile/00841917517097216645noreply@blogger.com2tag:blogger.com,1999:blog-6262128735818493674.post-7161355766097839982020-07-05T02:10:00.001+05:302020-07-05T02:10:53.790+05:30Dynamo Db - locks continued<div dir="ltr" style="text-align: left;" trbidi="on">
In the last post I used optimistic locking with the Version attribute provided by DynamoDBMapper. Here I am going to look at the pessimistic locking method<br />
<a name='more'></a>Pessimistic locking techniques require us to acquire a lock on the resource before we can modify it. As only one process can acquire the lock, e are guaranteed to be working with the latest version of the object.<br />
<div>
Dynamo Db inherently does not provide a lock implementation, Amazon has however released an open source library called the <a href="https://github.com/amazon-archives/dynamodb-lock-client" target="_blank">Amazon DynamoDB Lock Client</a>, which can be used for this purpose.</div>
<div>
Similar to our previous post I created a simple Cake class:</div>
<!-- HTML generated using hilite.me --><br />
<div style="background: #f8f8f8; border-width: 0.1em 0.1em 0.1em 0.8em; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="line-height: 125%; margin: 0;"><span style="color: #aa22ff;">@DynamoDBTable</span><span style="color: #666666;">(</span>tableName <span style="color: #666666;">=</span> <span style="color: #bb4444;">"Cake"</span><span style="color: #666666;">)</span>
<span style="color: #aa22ff; font-weight: bold;">public</span> <span style="color: #aa22ff; font-weight: bold;">class</span> <span style="color: blue;">Cake</span> <span style="color: #666666;">{</span>
<span style="color: #aa22ff;">@DynamoDBHashKey</span><span style="color: #666666;">(</span>attributeName <span style="color: #666666;">=</span> <span style="color: #bb4444;">"id"</span><span style="color: #666666;">)</span>
<span style="color: #aa22ff; font-weight: bold;">private</span> String id<span style="color: #666666;">;</span>
<span style="color: #aa22ff; font-weight: bold;">private</span> <span style="color: #00bb00; font-weight: bold;">int</span> noOfSlices<span style="color: #666666;">;</span>
<span style="color: #aa22ff; font-weight: bold;">private</span> String lastEatenBy<span style="color: #666666;">;</span>
<span style="color: #008800; font-style: italic;">//setter getters</span>
<span style="color: #666666;">}</span>
</pre>
</div>
Notice there is no version attribute here.
The DynamoDb representation is as below:
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgEEIAje8U0DtR_tgDFrgkZU5V9txIRcAn9I066GwMI19aq3nMJSH9p8C_ZlpHlIlgK2hX9seHOxKg5Hpcx6X2FFF2uq-yJhjB5OTAYAXjmrmWSwHVMEjZ9h1Lr3Rdpc9BYY9Txpf1s8GY/s1600/CakeEntry.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="200" data-original-width="383" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgEEIAje8U0DtR_tgDFrgkZU5V9txIRcAn9I066GwMI19aq3nMJSH9p8C_ZlpHlIlgK2hX9seHOxKg5Hpcx6X2FFF2uq-yJhjB5OTAYAXjmrmWSwHVMEjZ9h1Lr3Rdpc9BYY9Txpf1s8GY/s1600/CakeEntry.png" /></a></div>
<br />
The next step is to setup the Lock Client. The Lock client uses a separate DynamoDb table to manage locks. Instead of manually setting up the table, I used a utility method provided by the LockClient library:<br />
<!-- HTML generated using hilite.me --><br />
<div style="background: #f8f8f8; border-width: 0.1em 0.1em 0.1em 0.8em; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="line-height: 125%; margin: 0;">CreateDynamoDBTableOptions lockTableOptions <span style="color: #666666;">=</span> CreateDynamoDBTableOptions<span style="color: #666666;">.</span><span style="color: #bb4444;">builder</span><span style="color: #666666;">(</span>amazonDynamoDB<span style="color: #666666;">,</span>
<span style="color: #aa22ff; font-weight: bold;">new</span> <span style="color: #00a000;">ProvisionedThroughput</span><span style="color: #666666;">(25L,</span> <span style="color: #666666;">25L),</span> <span style="color: #bb4444;">"CakeLock"</span><span style="color: #666666;">).</span><span style="color: #bb4444;">build</span><span style="color: #666666;">();</span>
AmazonDynamoDBLockClient<span style="color: #666666;">.</span><span style="color: #bb4444;">createLockTableInDynamoDB</span><span style="color: #666666;">(</span>lockTableOptions<span style="color: #666666;">);</span>
</pre>
</div>
The table created is as below:
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEia6tzCK81oXp3_Wu_ZajdJXu4kMYhp9xPEkMNVhLxojbYcrFDn8CfvzVh5SxzSsDhyykiaKx1MMeMD98qmdM5XTrmScn-h89unHa5QuaaCNMcekmzTypHlvnHLgcKkqB_Y6IIjkAOj5PE/s1600/CakeLockTable.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="400" data-original-width="615" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEia6tzCK81oXp3_Wu_ZajdJXu4kMYhp9xPEkMNVhLxojbYcrFDn8CfvzVh5SxzSsDhyykiaKx1MMeMD98qmdM5XTrmScn-h89unHa5QuaaCNMcekmzTypHlvnHLgcKkqB_Y6IIjkAOj5PE/s1600/CakeLockTable.png" /></a></div>
<div class="separator" style="clear: both; text-align: center;">
</div>
A table is created with the hash key as 'key'. This table is used to determine if a Cake item is locked and who owns the lock.<br />
The Cake consumer threads is as below:<br />
<!-- HTML generated using hilite.me --><br />
<div style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="line-height: 125%; margin: 0px;"><span style="background-color: #f8f8f8; color: #aa22ff; font-weight: bold;">static</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #aa22ff; font-weight: bold;">class</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: blue;">CakeEater</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #aa22ff; font-weight: bold;">implements</span><span style="background-color: #f8f8f8;"> Callable</span><span style="background-color: #f8f8f8; color: #666666;"><</span><span style="background-color: #f8f8f8;">Void</span><span style="background-color: #f8f8f8; color: #666666;">></span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #666666;">{</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #aa22ff; font-weight: bold;">private</span><span style="background-color: #f8f8f8;"> String name</span><span style="background-color: #f8f8f8; color: #666666;">;</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #aa22ff; font-weight: bold;">private</span><span style="background-color: #f8f8f8;"> DynamoDBMapper mapper</span><span style="background-color: #f8f8f8; color: #666666;">;</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #aa22ff; font-weight: bold;">private</span><span style="background-color: #f8f8f8;"> String cakeKey</span><span style="background-color: #f8f8f8; color: #666666;">;</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #aa22ff; font-weight: bold;">public</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #00a000;">CakeEater</span><span style="background-color: #f8f8f8; color: #666666;">(</span><span style="background-color: #f8f8f8;">String name</span><span style="background-color: #f8f8f8; color: #666666;">,</span><span style="background-color: #f8f8f8;"> DynamoDBMapper mapper</span><span style="background-color: #f8f8f8; color: #666666;">,</span><span style="background-color: #f8f8f8;"> String cakeKey</span><span style="background-color: #f8f8f8; color: #666666;">)</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #666666;">{</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #aa22ff; font-weight: bold;">this</span><span style="background-color: #f8f8f8; color: #666666;">.</span><span style="background-color: #f8f8f8; color: #bb4444;">name</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #666666;">=</span><span style="background-color: #f8f8f8;"> name</span><span style="background-color: #f8f8f8; color: #666666;">;</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #aa22ff; font-weight: bold;">this</span><span style="background-color: #f8f8f8; color: #666666;">.</span><span style="background-color: #f8f8f8; color: #bb4444;">mapper</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #666666;">=</span><span style="background-color: #f8f8f8;"> mapper</span><span style="background-color: #f8f8f8; color: #666666;">;</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #aa22ff; font-weight: bold;">this</span><span style="background-color: #f8f8f8; color: #666666;">.</span><span style="background-color: #f8f8f8; color: #bb4444;">cakeKey</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #666666;">=</span><span style="background-color: #f8f8f8;"> cakeKey</span><span style="background-color: #f8f8f8; color: #666666;">;</span><span style="background-color: #f8f8f8;">
System</span><span style="background-color: #f8f8f8; color: #666666;">.</span><span style="background-color: #f8f8f8; color: #bb4444;">out</span><span style="background-color: #f8f8f8; color: #666666;">.</span><span style="background-color: #f8f8f8; color: #bb4444;">println</span><span style="background-color: #f8f8f8; color: #666666;">(</span><span style="background-color: #f8f8f8; color: #bb4444;">"Eater "</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #666666;">+</span><span style="background-color: #f8f8f8;"> name </span><span style="background-color: #f8f8f8; color: #666666;">+</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #bb4444;">" ready to start eating"</span><span style="background-color: #f8f8f8; color: #666666;">);</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #666666;">}</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #aa22ff;">@Override</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #aa22ff; font-weight: bold;">public</span><span style="background-color: #f8f8f8;"> Void </span><span style="background-color: #f8f8f8; color: #00a000;">call</span><span style="background-color: #f8f8f8; color: #666666;">()</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #aa22ff; font-weight: bold;">throws</span><span style="background-color: #f8f8f8;"> IOException </span><span style="background-color: #f8f8f8; color: #666666;">{</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #008800; font-style: italic;">// Whether or not to create a heartbeating background thread</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #aa22ff; font-weight: bold;">final</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #00bb00; font-weight: bold;">boolean</span><span style="background-color: #f8f8f8;"> createHeartbeatBackgroundThread </span><span style="background-color: #f8f8f8; color: #666666;">=</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #aa22ff; font-weight: bold;">false</span><span style="background-color: #f8f8f8; color: #666666;">;</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #008800; font-style: italic;">//build the lock client</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #00bb00; font-weight: bold;">long</span><span style="background-color: #f8f8f8;"> leaseDuration </span><span style="background-color: #f8f8f8; color: #666666;">=</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #666666;">2L;</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #aa22ff; font-weight: bold;">final</span><span style="background-color: #f8f8f8;"> AmazonDynamoDBLockClient client </span><span style="background-color: #f8f8f8; color: #666666;">=</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: yellow;"><span style="color: #aa22ff; font-weight: bold;">new</span> AmazonDynamoDBLockClient<span style="color: #666666;">(</span>
AmazonDynamoDBLockClientOptions<span style="color: #666666;">.</span><span style="color: #bb4444;">builder</span><span style="color: #666666;">(</span>amazonDynamoDB<span style="color: #666666;">,</span> <span style="color: #bb4444;">"CakeLock"</span><span style="color: #666666;">)</span>
<span style="color: #666666;">.</span><span style="color: #bb4444;">withTimeUnit</span><span style="color: #666666;">(</span>TimeUnit<span style="color: #666666;">.</span><span style="color: #bb4444;">MILLISECONDS</span><span style="color: #666666;">)</span>
<span style="color: #666666;">.</span><span style="color: #bb4444;">withLeaseDuration</span><span style="color: #666666;">(</span>leaseDuration<span style="color: #666666;">)</span>
<span style="color: #666666;">.</span><span style="color: #bb4444;">withHeartbeatPeriod</span><span style="color: #666666;">(2L)</span>
<span style="color: #666666;">.</span><span style="color: #bb4444;">withCreateHeartbeatBackgroundThread</span><span style="color: #666666;">(</span>createHeartbeatBackgroundThread<span style="color: #666666;">)</span>
<span style="color: #666666;">.</span><span style="color: #bb4444;">build</span><span style="color: #666666;">());</span></span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #aa22ff; font-weight: bold;">while</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #666666;">(</span><span style="background-color: #f8f8f8; color: #aa22ff; font-weight: bold;">true</span><span style="background-color: #f8f8f8; color: #666666;">)</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #666666;">{</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #008800; font-style: italic;">//try to acquire a lock on the partition key - cakeKey</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #aa22ff; font-weight: bold;">try</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #666666;">{</span><span style="background-color: #f8f8f8;">
System</span><span style="background-color: #f8f8f8; color: #666666;">.</span><span style="background-color: #f8f8f8; color: #bb4444;">out</span><span style="background-color: #f8f8f8; color: #666666;">.</span><span style="background-color: #f8f8f8; color: #bb4444;">println</span><span style="background-color: #f8f8f8; color: #666666;">(</span><span style="background-color: #f8f8f8; color: #bb4444;">"Attempt to acquire lock by "</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #666666;">+</span><span style="background-color: #f8f8f8;"> name</span><span style="background-color: #f8f8f8; color: #666666;">);</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #aa22ff; font-weight: bold;">final</span><span style="background-color: #f8f8f8;"> Optional</span><span style="background-color: #f8f8f8; color: #666666;"><</span><span style="background-color: #f8f8f8;">LockItem</span><span style="background-color: #f8f8f8; color: #666666;">></span><span style="background-color: #f8f8f8;"> lockItem </span><span style="background-color: #f8f8f8; color: #666666;">=</span><span style="background-color: #f8f8f8;"> Optional</span><span style="background-color: #f8f8f8; color: #666666;">.</span><span style="background-color: #f8f8f8; color: #bb4444;">ofNullable</span><span style="background-color: #f8f8f8; color: #666666;">(</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: yellow;">client<span style="color: #666666;">.</span><span style="color: #bb4444;">acquireLock</span><span style="color: #666666;">(</span>AcquireLockOptions<span style="color: #666666;">.</span><span style="color: #bb4444;">builder</span><span style="color: #666666;">(</span>cakeKey<span style="color: #666666;">)</span></span><span style="background-color: #f8f8f8; color: #666666;">.</span><span style="background-color: #f8f8f8; color: #bb4444;">build</span><span style="background-color: #f8f8f8; color: #666666;">()));</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #aa22ff; font-weight: bold;">if</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #666666;">(</span><span style="background-color: #f8f8f8;">lockItem</span><span style="background-color: #f8f8f8; color: #666666;">.</span><span style="background-color: #f8f8f8; color: #bb4444;">isPresent</span><span style="background-color: #f8f8f8; color: #666666;">())</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #666666;">{</span><span style="background-color: #f8f8f8;">
System</span><span style="background-color: #f8f8f8; color: #666666;">.</span><span style="background-color: #f8f8f8; color: #bb4444;">out</span><span style="background-color: #f8f8f8; color: #666666;">.</span><span style="background-color: #f8f8f8; color: #bb4444;">println</span><span style="background-color: #f8f8f8; color: #666666;">(</span><span style="background-color: #f8f8f8; color: #bb4444;">"Acquired lock by "</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #666666;">+</span><span style="background-color: #f8f8f8;"> name</span><span style="background-color: #f8f8f8; color: #666666;">);</span><span style="background-color: #f8f8f8;">
Cake cakeId </span><span style="background-color: #f8f8f8; color: #666666;">=</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #aa22ff; font-weight: bold;">new</span><span style="background-color: #f8f8f8;"> Cake</span><span style="background-color: #f8f8f8; color: #666666;">();</span><span style="background-color: #f8f8f8;">
cakeId</span><span style="background-color: #f8f8f8; color: #666666;">.</span><span style="background-color: #f8f8f8; color: #bb4444;">setId</span><span style="background-color: #f8f8f8; color: #666666;">(</span><span style="background-color: #f8f8f8; color: #aa22ff; font-weight: bold;">this</span><span style="background-color: #f8f8f8; color: #666666;">.</span><span style="background-color: #f8f8f8; color: #bb4444;">cakeKey</span><span style="background-color: #f8f8f8; color: #666666;">);</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #008800; font-style: italic;">//Time to eat the cake quickly - within 200 millis</span><span style="background-color: #f8f8f8;">
Cake cake </span><span style="background-color: #f8f8f8; color: #666666;">=</span><span style="background-color: #f8f8f8;"> mapper</span><span style="background-color: #f8f8f8; color: #666666;">.</span><span style="background-color: #f8f8f8; color: #bb4444;">load</span><span style="background-color: #f8f8f8; color: #666666;">(</span><span style="background-color: #f8f8f8;">cakeId</span><span style="background-color: #f8f8f8; color: #666666;">);</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #aa22ff; font-weight: bold;">if</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #666666;">(</span><span style="background-color: #f8f8f8;">cake </span><span style="background-color: #f8f8f8; color: #666666;">!=</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #aa22ff; font-weight: bold;">null</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #666666;">&&</span><span style="background-color: #f8f8f8;"> cake</span><span style="background-color: #f8f8f8; color: #666666;">.</span><span style="background-color: #f8f8f8; color: #bb4444;">getNoOfSlices</span><span style="background-color: #f8f8f8; color: #666666;">()</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #666666;">></span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #666666;">0)</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #666666;">{</span><span style="background-color: #f8f8f8;">
System</span><span style="background-color: #f8f8f8; color: #666666;">.</span><span style="background-color: #f8f8f8; color: #bb4444;">out</span><span style="background-color: #f8f8f8; color: #666666;">.</span><span style="background-color: #f8f8f8; color: #bb4444;">println</span><span style="background-color: #f8f8f8; color: #666666;">(</span><span style="background-color: #f8f8f8;">name </span><span style="background-color: #f8f8f8; color: #666666;">+</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #bb4444;">" looking to eat. Available Pieces "</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #666666;">+</span><span style="background-color: #f8f8f8;"> cake</span><span style="background-color: #f8f8f8; color: #666666;">.</span><span style="background-color: #f8f8f8; color: #bb4444;">getNoOfSlices</span><span style="background-color: #f8f8f8; color: #666666;">()</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #666666;">+</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #bb4444;">".Last eaten by "</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #666666;">+</span><span style="background-color: #f8f8f8;"> cake</span><span style="background-color: #f8f8f8; color: #666666;">.</span><span style="background-color: #f8f8f8; color: #bb4444;">getLastEatenBy</span><span style="background-color: #f8f8f8; color: #666666;">());</span><span style="background-color: #f8f8f8;">
cake</span><span style="background-color: #f8f8f8; color: #666666;">.</span><span style="background-color: #f8f8f8; color: #bb4444;">setNoOfSlices</span><span style="background-color: #f8f8f8; color: #666666;">(</span><span style="background-color: #f8f8f8;">cake</span><span style="background-color: #f8f8f8; color: #666666;">.</span><span style="background-color: #f8f8f8; color: #bb4444;">getNoOfSlices</span><span style="background-color: #f8f8f8; color: #666666;">()</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #666666;">-</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #666666;">1);</span><span style="background-color: #f8f8f8;">
cake</span><span style="background-color: #f8f8f8; color: #666666;">.</span><span style="background-color: #f8f8f8; color: #bb4444;">setLastEatenBy</span><span style="background-color: #f8f8f8; color: #666666;">(</span><span style="background-color: #f8f8f8;">name</span><span style="background-color: #f8f8f8; color: #666666;">);</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #aa22ff; font-weight: bold;">try</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #666666;">{</span><span style="background-color: #f8f8f8;">
mapper</span><span style="background-color: #f8f8f8; color: #666666;">.</span><span style="background-color: #f8f8f8; color: #bb4444;">save</span><span style="background-color: #f8f8f8; color: #666666;">(</span><span style="background-color: #f8f8f8;">cake</span><span style="background-color: #f8f8f8; color: #666666;">);</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #008800; font-style: italic;">// processing complete</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #666666;">}</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #aa22ff; font-weight: bold;">catch</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #666666;">(</span><span style="background-color: #f8f8f8;">Exception e</span><span style="background-color: #f8f8f8; color: #666666;">)</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #666666;">{</span><span style="background-color: #f8f8f8;">
System</span><span style="background-color: #f8f8f8; color: #666666;">.</span><span style="background-color: #f8f8f8; color: #bb4444;">err</span><span style="background-color: #f8f8f8; color: #666666;">.</span><span style="background-color: #f8f8f8; color: #bb4444;">println</span><span style="background-color: #f8f8f8; color: #666666;">(</span><span style="background-color: #f8f8f8; color: #bb4444;">"Saving Cake for "</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #666666;">+</span><span style="background-color: #f8f8f8;"> name </span><span style="background-color: #f8f8f8; color: #666666;">+</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #bb4444;">" failed. "</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #666666;">+</span><span style="background-color: #f8f8f8;"> e</span><span style="background-color: #f8f8f8; color: #666666;">.</span><span style="background-color: #f8f8f8; color: #bb4444;">getMessage</span><span style="background-color: #f8f8f8; color: #666666;">());</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #666666;">}</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #666666;">}</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #aa22ff; font-weight: bold;">else</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #666666;">{</span><span style="background-color: #f8f8f8;">
System</span><span style="background-color: #f8f8f8; color: #666666;">.</span><span style="background-color: #f8f8f8; color: #bb4444;">out</span><span style="background-color: #f8f8f8; color: #666666;">.</span><span style="background-color: #f8f8f8; color: #bb4444;">println</span><span style="background-color: #f8f8f8; color: #666666;">(</span><span style="background-color: #f8f8f8; color: #bb4444;">"Cake is over. "</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #666666;">+</span><span style="background-color: #f8f8f8;"> name </span><span style="background-color: #f8f8f8; color: #666666;">+</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #bb4444;">" stopping"</span><span style="background-color: #f8f8f8; color: #666666;">);</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #aa22ff; font-weight: bold;">break</span><span style="background-color: #f8f8f8; color: #666666;">;</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #666666;">}</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: yellow;">client<span style="color: #666666;">.</span><span style="color: #bb4444;">releaseLock</span><span style="color: #666666;">(</span>lockItem<span style="color: #666666;">.</span><span style="color: #bb4444;">get</span><span style="color: #666666;">());</span></span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #008800; font-style: italic;">//lock released</span><span style="background-color: #f8f8f8;">
System</span><span style="background-color: #f8f8f8; color: #666666;">.</span><span style="background-color: #f8f8f8; color: #bb4444;">out</span><span style="background-color: #f8f8f8; color: #666666;">.</span><span style="background-color: #f8f8f8; color: #bb4444;">println</span><span style="background-color: #f8f8f8; color: #666666;">(</span><span style="background-color: #f8f8f8;">name </span><span style="background-color: #f8f8f8; color: #666666;">+</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #bb4444;">" taking a break"</span><span style="background-color: #f8f8f8; color: #666666;">);</span><span style="background-color: #f8f8f8;">
Thread</span><span style="background-color: #f8f8f8; color: #666666;">.</span><span style="background-color: #f8f8f8; color: #bb4444;">sleep</span><span style="background-color: #f8f8f8; color: #666666;">(</span><span style="background-color: #f8f8f8;">leaseDuration </span><span style="background-color: #f8f8f8; color: #666666;">*</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #666666;">3);</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #666666;">}</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #aa22ff; font-weight: bold;">else</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #666666;">{</span><span style="background-color: #f8f8f8;">
System</span><span style="background-color: #f8f8f8; color: #666666;">.</span><span style="background-color: #f8f8f8; color: #bb4444;">out</span><span style="background-color: #f8f8f8; color: #666666;">.</span><span style="background-color: #f8f8f8; color: #bb4444;">println</span><span style="background-color: #f8f8f8; color: #666666;">(</span><span style="background-color: #f8f8f8;">name </span><span style="background-color: #f8f8f8; color: #666666;">+</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #bb4444;">" failed to acquire lock"</span><span style="background-color: #f8f8f8; color: #666666;">);</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #666666;">}</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #666666;">}</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #aa22ff; font-weight: bold;">catch</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #666666;">(</span><span style="background-color: #f8f8f8;">InterruptedException e</span><span style="background-color: #f8f8f8; color: #666666;">)</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #666666;">{</span><span style="background-color: #f8f8f8;">
e</span><span style="background-color: #f8f8f8; color: #666666;">.</span><span style="background-color: #f8f8f8; color: #bb4444;">printStackTrace</span><span style="background-color: #f8f8f8; color: #666666;">();</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #666666;">}</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #666666;">}</span><span style="background-color: #f8f8f8;">
client</span><span style="background-color: #f8f8f8; color: #666666;">.</span><span style="background-color: #f8f8f8; color: #bb4444;">close</span><span style="background-color: #f8f8f8; color: #666666;">();</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #aa22ff; font-weight: bold;">return</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #aa22ff; font-weight: bold;">null</span><span style="background-color: #f8f8f8; color: #666666;">;</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #666666;">}</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #666666;">}</span><span style="background-color: #f8f8f8;">
</span></pre>
</div>
The code works as below:<br />
<br />
<ol style="text-align: left;">
<li>Code tries to acquire a lock for 2 milliseconds. </li>
<li>If lock is acquired, code will fetch the cake record and eat a piece. </li>
<li>Once done the code releases the lock. </li>
</ol>
<br />
The change in lock values can be seen in the Cake Locks table:<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhMJJI1_qUxeW6vXMTUj842Vclrt5YQruEx_9KqbRCNWmQc8TNeWjLDbfEF4J2u8VPk2QslVZUoUfwUVoZDkpd8yl1zYvekxcic7lH6BZk0FO3Dkgm4zustHNH1DdgwUOHBs4n2n2LebPI/s1600/Locks.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="436" data-original-width="1123" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhMJJI1_qUxeW6vXMTUj842Vclrt5YQruEx_9KqbRCNWmQc8TNeWjLDbfEF4J2u8VPk2QslVZUoUfwUVoZDkpd8yl1zYvekxcic7lH6BZk0FO3Dkgm4zustHNH1DdgwUOHBs4n2n2LebPI/s1600/Locks.png" /></a></div>
The assumption here is that code will finish its operations within 2 milliseconds. How does the locking logic work ?
<br />
<div style="background: #f8f8f8; border-width: 0.1em 0.1em 0.1em 0.8em; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="line-height: 125%; margin: 0;">The way locks are expired is that a call to acquireLock reads in the current lock,
checks the RecordVersionNumber of the lock (which is a GUID) and starts a timer.
If the lock still has the same GUID after the lease duration time has passed, the
client will determine that the lock is stale and expire it
</pre>
</div>
So any code that reads the Lock table - if it finds an entry will always assume that someone is holding the lock. The assumption is also that lock is being held for atleast 'DurationTime' as specified in the Table.<br />
<i>So in our code, Lets say Thread 0 wins the lock and will hold it for 2 milliseconds. It after 1 millisecond, Thread 1 arrives it will see that the lock is being held for a duration of 2 milliseconds and will therefore wait for 2 milliseconds before trying to acquire again. In effect this is 3 milliseconds since Thread 0 acquired the lock.</i><br />
<i>When Thread 1 finally checks again, the record would have been deleted (as Thread 0 had released the lock after 2 milliseconds). So Thread 1 will acquire the lock.</i><br />
What if Thread 0 had crashed while holding the lock ?<br />
If Thread had crashed without releasing the lock, an entry would remain in table. When Thread 1 checks again, the recordVersionNumber would be same as when it checked earlier. This means, lock is stale and so Thread 0 will acquire the lock.<br />
The lock client has additional features - non blocking wait, automatic heartbeats that extend your lock duration etc.<br />
My code output is as below:<br />
<div style="background: #f8f8f8; overflow:auto;width:auto;border:solid gray;border-width:.1em .1em .1em .8em;padding:.2em .6em;"><pre style="margin: 0; line-height: 125%">Cake has been saved
Eater Eater0 ready to start eating
Eater Eater1 ready to start eating
Eater Eater2 ready to start eating
Attempt to acquire lock by Eater1
Attempt to acquire lock by Eater0
Attempt to acquire lock by Eater2
Acquired lock by Eater0
Eater0 looking to eat. Available Pieces 10.Last eaten by null
Eater0 taking a break
Attempt to acquire lock by Eater0
Acquired lock by Eater0
Eater0 looking to eat. Available Pieces 9.Last eaten by Eater0
Eater0 taking a break
Attempt to acquire lock by Eater0
Acquired lock by Eater0
Eater0 looking to eat. Available Pieces 8.Last eaten by Eater0
Eater0 taking a break
Attempt to acquire lock by Eater0
Acquired lock by Eater0
</pre></div>
What fits my use case better ?
I compared both locking mechanisms - for my usecase, Optimistic Locking is a better fit. The DynamoMapper can handle the contention among multiple write threads ensuring none of them updates stale information.
Pessimistic locking is more of an advisory locking mechanism, with more responsibility on the code to get it right. A more appropriate use case for this would be when I need to perform some operations if I have a lock on the record.
</div>
Robin Varghesehttp://www.blogger.com/profile/00841917517097216645noreply@blogger.com3tag:blogger.com,1999:blog-6262128735818493674.post-30848561800592852592020-06-26T20:55:00.003+05:302021-06-24T03:51:08.528+05:30Dynamo Db - locks <div dir="ltr" style="text-align: left;" trbidi="on">
When I studies Databases in college, we learnt the concept of locks<br />
<div style="background: rgb(248, 248, 248); border-width: 0.1em 0.1em 0.1em 0.8em; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="line-height: 125%; margin: 0px;">A database lock is used to “lock” some data in a database so that only one
database user/session may update that particular data. So, database locks
exist to prevent two or more database users from updating the same exact
piece of data at the same exact time.
</pre>
</div>
The idea was to achieve transactional isolation. Or two operations operating on the same record did not have any side effects on the other.<br />
<br />
<a name='more'></a>We studied pessimistic locking and optimistic locking approaches. I also came across this terms more frequently when working with Hibernate and SQL databases.<br />
Quoting liberally from <a href="https://docs.jboss.org/jbossas/docs/Server_Configuration_Guide/4/html/TransactionJTA_Overview-Pessimistic_and_optimistic_locking.html#:~:text=With%20pessimistic%20locking%2C%20locks%20are,in%20a%20fail%2Dsafe%20way.&text=The%20lock%20exists%20until%20the,is%20accessed%20by%20a%20transaction." target="_blank">Jboss docs</a>:<br />
Pessimistic locking:
<br />
<div style="background: rgb(248, 248, 248); border-width: 0.1em 0.1em 0.1em 0.8em; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="line-height: 125%; margin: 0px;">With pessimistic locking, locks are applied in a fail-safe way<span style="color: blue;">.A resource is
locked from the time it is first accessed in a transaction until the
transaction is finished, making it inaccessible to other transactions during
that time.</span>
If most transactions simply look at the resource and never change it, an
exclusive lock may be overkill as it may cause lock contention, and optimistic
locking may be a better approach.
</pre>
</div>
Optimistic Locking:
<br />
<div style="background: rgb(248, 248, 248); border-width: 0.1em 0.1em 0.1em 0.8em; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="line-height: 125%; margin: 0px;">With optimistic locking, <span style="color: blue;">a resource is not actually locked when it is first
is accessed by a transaction. Instead, the state of the resource at the time
when it would have been locked with the pessimistic locking approach is saved.</span>
Other transactions are able to concurrently access to the resource and the
possibility of conflicting changes is possible. At commit time, when the
resource is about to be updated in persistent storage, the state of the resource
is read from storage again and compared to the state that was saved when the
resource was first accessed in the transaction. If the two states differ, a
conflicting update was made, and the transaction will be rolled back.
</pre>
</div>
Consider a banking application example<br />
With Pessimistic Locking, <i>an account is locked as soon as it is accessed in a transaction. Attempts to use the account in other transactions while it is locked will either result in the other process being delayed until the account lock is released, or that the process transaction will be rolled back. The lock exists until the transaction has either been committed or rolled back.</i></div><div dir="ltr" style="text-align: left;" trbidi="on">
With Optimistic locking <i>the amount of an account is saved when the account is first accessed in a transaction. If the transaction changes the account amount, the amount is read from the store again just before the amount is about to be updated. If the amount has changed since the transaction began, the transaction will fail itself, otherwise the new amount is written to persistent storage.</i><br />
With Dynamo Db (which is noSql) what are my options ?<br />
<b>Optimistic Locking:</b> This approach is supported in Dynamo Db API. I decided to try out an example for myself:<br />
I created a Pie and added a host of eaters. Each trying to eat a piece of the pie. Since we have several simultaneous consumers, we need to ensure the Pie consumption is managed in a synchronous manner.<br />
The Pie here is an DynamoDb instance:<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhpKBbowgjSTZBv0XfgGldeqo4L20HSB0pAVx_Q1vhSfXj-y-W7g8CytX-GeS1l14DUAT9mtNOaNTmubk9INDl8esW_9ohLY7Ok9nPoC46Ws9FVI_WKlJc7pQ6QQb4kPhhQFdfjNbo1fOI/s1600/Tbl.png" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="316" data-original-width="518" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhpKBbowgjSTZBv0XfgGldeqo4L20HSB0pAVx_Q1vhSfXj-y-W7g8CytX-GeS1l14DUAT9mtNOaNTmubk9INDl8esW_9ohLY7Ok9nPoC46Ws9FVI_WKlJc7pQ6QQb4kPhhQFdfjNbo1fOI/s1600/Tbl.png" /></a></div>
My Java POJO is<br />
<div style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="line-height: 125%; margin: 0px;"><span style="background-color: #f8f8f8; color: #aa22ff;">@DynamoDBTable</span><span style="background-color: #f8f8f8; color: #666666;">(</span><span style="background-color: #f8f8f8;">tableName </span><span style="background-color: #f8f8f8; color: #666666;">=</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #bb4444;">"Pie"</span><span style="background-color: #f8f8f8; color: #666666;">)</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #aa22ff; font-weight: bold;">public</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #aa22ff; font-weight: bold;">class</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: blue;">Pie</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #666666;">{</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #aa22ff;">@DynamoDBHashKey</span><span style="background-color: #f8f8f8; color: #666666;">(</span><span style="background-color: #f8f8f8;">attributeName </span><span style="background-color: #f8f8f8; color: #666666;">=</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #bb4444;">"id"</span><span style="background-color: #f8f8f8; color: #666666;">)</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #aa22ff; font-weight: bold;">private</span><span style="background-color: #f8f8f8;"> String id</span><span style="background-color: #f8f8f8; color: #666666;">;</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #aa22ff; font-weight: bold;">private</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #00bb00; font-weight: bold;">int</span><span style="background-color: #f8f8f8;"> noOfSlices</span><span style="background-color: #f8f8f8; color: #666666;">;</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #aa22ff; font-weight: bold;">private</span><span style="background-color: #f8f8f8;"> String lastEatenBy</span><span style="background-color: #f8f8f8; color: #666666;">;</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #008800; font-style: italic;">/**</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #008800; font-style: italic;"> * Version field - used by DynamoDbMapper for optimistic version</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #008800; font-style: italic;"> */</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: yellow; color: #aa22ff;">@DynamoDBVersionAttribute</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #aa22ff; font-weight: bold;">private</span><span style="background-color: #f8f8f8;"> Long version</span><span style="background-color: #f8f8f8; color: #666666;">;</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #008800; font-style: italic;">//setters and getters</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #666666;">}</span><span style="background-color: #f8f8f8;">
</span></pre>
</div>
The Version field is used to ensure Optimistic locking of the Pie when multiple consumers try to eat it.
My Consumer class is as below:
<!--HTML generated using hilite.me--><br />
<div style="background: rgb(248, 248, 248); border-width: 0.1em 0.1em 0.1em 0.8em; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="line-height: 125%; margin: 0px;"><span style="color: #aa22ff; font-weight: bold;">static</span> <span style="color: #aa22ff; font-weight: bold;">class</span> <span style="color: blue;">PieEater</span> <span style="color: #aa22ff; font-weight: bold;">implements</span> Callable<span style="color: #666666;"><</span>Void<span style="color: #666666;">></span> <span style="color: #666666;">{</span>
<span style="color: #aa22ff; font-weight: bold;">private</span> String name<span style="color: #666666;">;</span>
<span style="color: #aa22ff; font-weight: bold;">private</span> DynamoDBMapper mapper<span style="color: #666666;">;</span>
<span style="color: #aa22ff; font-weight: bold;">private</span> String pieKey<span style="color: #666666;">;</span>
<span style="color: #aa22ff; font-weight: bold;">public</span> <span style="color: #00a000;">PieEater</span><span style="color: #666666;">(</span>String name<span style="color: #666666;">,</span> DynamoDBMapper mapper<span style="color: #666666;">,</span> String pieKey<span style="color: #666666;">)</span> <span style="color: #666666;">{</span>
<span style="color: #aa22ff; font-weight: bold;">this</span><span style="color: #666666;">.</span><span style="color: #bb4444;">name</span> <span style="color: #666666;">=</span> name<span style="color: #666666;">;</span>
<span style="color: #aa22ff; font-weight: bold;">this</span><span style="color: #666666;">.</span><span style="color: #bb4444;">mapper</span> <span style="color: #666666;">=</span> mapper<span style="color: #666666;">;</span>
<span style="color: #aa22ff; font-weight: bold;">this</span><span style="color: #666666;">.</span><span style="color: #bb4444;">pieKey</span> <span style="color: #666666;">=</span> pieKey<span style="color: #666666;">;</span>
System<span style="color: #666666;">.</span><span style="color: #bb4444;">out</span><span style="color: #666666;">.</span><span style="color: #bb4444;">println</span><span style="color: #666666;">(</span><span style="color: #bb4444;">"Eater "</span> <span style="color: #666666;">+</span> name <span style="color: #666666;">+</span> <span style="color: #bb4444;">" ready to start eating"</span><span style="color: #666666;">);</span>
<span style="color: #666666;">}</span>
<span style="color: #aa22ff;">@Override</span>
<span style="color: #aa22ff; font-weight: bold;">public</span> Void <span style="color: #00a000;">call</span><span style="color: #666666;">()</span> <span style="color: #666666;">{</span>
System<span style="color: #666666;">.</span><span style="color: #bb4444;">out</span><span style="color: #666666;">.</span><span style="color: #bb4444;">println</span><span style="color: #666666;">(</span><span style="color: #aa22ff; font-weight: bold;">this</span><span style="color: #666666;">.</span><span style="color: #bb4444;">name</span> <span style="color: #666666;">+</span> <span style="color: #bb4444;">" Starting up on "</span> <span style="color: #666666;">+</span> <span style="color: #aa22ff; font-weight: bold;">this</span><span style="color: #666666;">.</span><span style="color: #bb4444;">pieKey</span><span style="color: #666666;">);</span>
Pie pieKey <span style="color: #666666;">=</span> <span style="color: #aa22ff; font-weight: bold;">new</span> Pie<span style="color: #666666;">();</span>
pieKey<span style="color: #666666;">.</span><span style="color: #bb4444;">setId</span><span style="color: #666666;">(</span><span style="color: #aa22ff; font-weight: bold;">this</span><span style="color: #666666;">.</span><span style="color: #bb4444;">pieKey</span><span style="color: #666666;">);</span>
<span style="color: #aa22ff; font-weight: bold;">while</span> <span style="color: #666666;">(</span><span style="color: #aa22ff; font-weight: bold;">true</span><span style="color: #666666;">)</span> <span style="color: #666666;">{</span>
Pie pie <span style="color: #666666;">=</span> mapper<span style="color: #666666;">.</span><span style="color: #bb4444;">load</span><span style="color: #666666;">(</span>pieKey<span style="color: #666666;">);</span>
<span style="color: #aa22ff; font-weight: bold;">if</span> <span style="color: #666666;">(</span>pie <span style="color: #666666;">!=</span> <span style="color: #aa22ff; font-weight: bold;">null</span> <span style="color: #666666;">&&</span> pie<span style="color: #666666;">.</span><span style="color: #bb4444;">getNoOfSlices</span><span style="color: #666666;">()</span> <span style="color: #666666;">></span> <span style="color: #666666;">0)</span> <span style="color: #666666;">{</span>
System<span style="color: #666666;">.</span><span style="color: #bb4444;">out</span><span style="color: #666666;">.</span><span style="color: #bb4444;">println</span><span style="color: #666666;">(</span>name <span style="color: #666666;">+</span> <span style="color: #bb4444;">" looking to eat a piece of pie. Available Pieces "</span> <span style="color: #666666;">+</span> pie<span style="color: #666666;">.</span><span style="color: #bb4444;">getNoOfSlices</span><span style="color: #666666;">()</span>
<span style="color: #666666;">+</span> <span style="color: #bb4444;">". Last eaten by "</span> <span style="color: #666666;">+</span> pie<span style="color: #666666;">.</span><span style="color: #bb4444;">getLastEatenBy</span><span style="color: #666666;">()</span> <span style="color: #666666;">+</span> <span style="color: #bb4444;">" [Version] "</span> <span style="color: #666666;">+</span> pie<span style="color: #666666;">.</span><span style="color: #bb4444;">getVersion</span><span style="color: #666666;">());</span>
pie<span style="color: #666666;">.</span><span style="color: #bb4444;">setNoOfSlices</span><span style="color: #666666;">(</span>pie<span style="color: #666666;">.</span><span style="color: #bb4444;">getNoOfSlices</span><span style="color: #666666;">()</span> <span style="color: #666666;">-</span> <span style="color: #666666;">1);</span>
pie<span style="color: #666666;">.</span><span style="color: #bb4444;">setLastEatenBy</span><span style="color: #666666;">(</span>name<span style="color: #666666;">);</span>
<span style="color: #aa22ff; font-weight: bold;">try</span> <span style="color: #666666;">{</span>
mapper<span style="color: #666666;">.</span><span style="color: #bb4444;">save</span><span style="color: #666666;">(</span>pie<span style="color: #666666;">);</span> <span style="color: #008800; font-style: italic;">// processing complete</span>
<span style="color: #666666;">}</span> <span style="color: #aa22ff; font-weight: bold;">catch</span> <span style="color: #666666;">(</span>Exception e<span style="color: #666666;">)</span> <span style="color: #666666;">{</span>
System<span style="color: #666666;">.</span><span style="color: #bb4444;">err</span><span style="color: #666666;">.</span><span style="color: #bb4444;">println</span><span style="color: #666666;">(</span><span style="color: #bb4444;">"Saving pie for "</span> <span style="color: #666666;">+</span> name <span style="color: #666666;">+</span> <span style="color: #bb4444;">" failed. "</span> <span style="color: #666666;">+</span> e<span style="color: #666666;">.</span><span style="color: #bb4444;">getMessage</span><span style="color: #666666;">());</span>
<span style="color: #008800; font-style: italic;">//e.printStackTrace();</span>
<span style="color: #666666;">}</span>
<span style="color: #666666;">}</span> <span style="color: #aa22ff; font-weight: bold;">else</span> <span style="color: #666666;">{</span>
System<span style="color: #666666;">.</span><span style="color: #bb4444;">out</span><span style="color: #666666;">.</span><span style="color: #bb4444;">println</span><span style="color: #666666;">(</span><span style="color: #bb4444;">"Pie is over. "</span> <span style="color: #666666;">+</span> name <span style="color: #666666;">+</span> <span style="color: #bb4444;">" stopping"</span><span style="color: #666666;">);</span>
<span style="color: #aa22ff; font-weight: bold;">break</span><span style="color: #666666;">;</span>
<span style="color: #666666;">}</span>
<span style="color: #666666;">}</span>
<span style="color: #aa22ff; font-weight: bold;">return</span> <span style="color: #aa22ff; font-weight: bold;">null</span><span style="color: #666666;">;</span>
<span style="color: #666666;">}</span>
<span style="color: #666666;">}</span>
</pre>
</div>
The class attempts to fetch the Pie and eat a piece of it. This continues until any pie pieces remain.<br />
The Main class that sets it all up together is as below:<br />
<div style="background: rgb(248, 248, 248); border-width: 0.1em 0.1em 0.1em 0.8em; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="line-height: 125%; margin: 0px;"><span style="color: #aa22ff; font-weight: bold;">private</span> <span style="color: #aa22ff; font-weight: bold;">static</span> DynamoDBMapper mapper<span style="color: #666666;">;</span>
<span style="color: #aa22ff; font-weight: bold;">private</span> <span style="color: #aa22ff; font-weight: bold;">static</span> <span style="color: #00bb00; font-weight: bold;">void</span> <span style="color: #00a000;">init</span><span style="color: #666666;">()</span> <span style="color: #666666;">{</span>
AWSCredentials awsCredentials <span style="color: #666666;">=</span> <span style="color: #aa22ff; font-weight: bold;">new</span> BasicAWSCredentials<span style="color: #666666;">(</span><span style="color: #bb4444;">"AccessKey"</span><span style="color: #666666;">,</span>
<span style="color: #bb4444;">"SecretKey"</span><span style="color: #666666;">);</span>
AWSCredentialsProvider awsCredentialsProvider <span style="color: #666666;">=</span> <span style="color: #aa22ff; font-weight: bold;">new</span> AWSStaticCredentialsProvider<span style="color: #666666;">(</span>awsCredentials<span style="color: #666666;">);</span>
AmazonDynamoDB client <span style="color: #666666;">=</span> AmazonDynamoDBClientBuilder<span style="color: #666666;">.</span><span style="color: #bb4444;">standard</span><span style="color: #666666;">()</span>
<span style="color: #666666;">.</span><span style="color: #bb4444;">withRegion</span><span style="color: #666666;">(</span><span style="color: #bb4444;">"us-east-1"</span><span style="color: #666666;">)</span>
<span style="color: #666666;">.</span><span style="color: #bb4444;">withCredentials</span><span style="color: #666666;">(</span>awsCredentialsProvider<span style="color: #666666;">).</span><span style="color: #bb4444;">build</span><span style="color: #666666;">();</span>
mapper <span style="color: #666666;">=</span> <span style="color: #aa22ff; font-weight: bold;">new</span> DynamoDBMapper<span style="color: #666666;">(</span>client<span style="color: #666666;">);</span>
<span style="color: #666666;">}</span>
<span style="color: #aa22ff; font-weight: bold;">public</span> <span style="color: #aa22ff; font-weight: bold;">static</span> <span style="color: #00bb00; font-weight: bold;">void</span> <span style="color: #00a000;">main</span><span style="color: #666666;">(</span>String<span style="color: #666666;">[]</span> args<span style="color: #666666;">)</span> <span style="color: #666666;">{</span>
init<span style="color: #666666;">();</span>
Pie pie <span style="color: #666666;">=</span> <span style="color: #aa22ff; font-weight: bold;">new</span> Pie<span style="color: #666666;">();</span>
pie<span style="color: #666666;">.</span><span style="color: #bb4444;">setNoOfSlices</span><span style="color: #666666;">(10);</span>
pie<span style="color: #666666;">.</span><span style="color: #bb4444;">setId</span><span style="color: #666666;">(</span><span style="color: #bb4444;">"StrawBerryPie"</span><span style="color: #666666;">);</span>
mapper<span style="color: #666666;">.</span><span style="color: #bb4444;">save</span><span style="color: #666666;">(</span>pie<span style="color: #666666;">);</span>
System<span style="color: #666666;">.</span><span style="color: #bb4444;">out</span><span style="color: #666666;">.</span><span style="color: #bb4444;">println</span><span style="color: #666666;">(</span><span style="color: #bb4444;">"Pie has been saved"</span><span style="color: #666666;">);</span>
<span style="color: #008800; font-style: italic;">//Start Worker Threads</span>
List<span style="color: #666666;"><</span>Thread<span style="color: #666666;">></span> threads <span style="color: #666666;">=</span> <span style="color: #aa22ff; font-weight: bold;">new</span> ArrayList<span style="color: #666666;"><>(10);</span>
<span style="color: #aa22ff; font-weight: bold;">for</span> <span style="color: #666666;">(</span><span style="color: #00bb00; font-weight: bold;">int</span> i <span style="color: #666666;">=</span> <span style="color: #666666;">0;</span> i <span style="color: #666666;"><</span> <span style="color: #666666;">10;</span> i<span style="color: #666666;">++)</span> <span style="color: #666666;">{</span>
PieEater pieEater <span style="color: #666666;">=</span> <span style="color: #aa22ff; font-weight: bold;">new</span> PieEater<span style="color: #666666;">(</span><span style="color: #bb4444;">"Eater"</span> <span style="color: #666666;">+</span> i<span style="color: #666666;">,</span> mapper<span style="color: #666666;">,</span> pie<span style="color: #666666;">.</span><span style="color: #bb4444;">getId</span><span style="color: #666666;">());</span>
FutureTask<span style="color: #666666;"><</span>Void<span style="color: #666666;">></span> futureTask <span style="color: #666666;">=</span> <span style="color: #aa22ff; font-weight: bold;">new</span> FutureTask<span style="color: #666666;"><>(</span>pieEater<span style="color: #666666;">);</span>
Thread thread <span style="color: #666666;">=</span> <span style="color: #aa22ff; font-weight: bold;">new</span> Thread<span style="color: #666666;">(</span>futureTask<span style="color: #666666;">);</span>
thread<span style="color: #666666;">.</span><span style="color: #bb4444;">start</span><span style="color: #666666;">();</span>
threads<span style="color: #666666;">.</span><span style="color: #bb4444;">add</span><span style="color: #666666;">(</span>thread<span style="color: #666666;">);</span>
<span style="color: #666666;">}</span>
<span style="color: #aa22ff; font-weight: bold;">for</span> <span style="color: #666666;">(</span>Thread thread <span style="color: #666666;">:</span> threads<span style="color: #666666;">)</span> <span style="color: #666666;">{</span>
<span style="color: #aa22ff; font-weight: bold;">try</span> <span style="color: #666666;">{</span>
thread<span style="color: #666666;">.</span><span style="color: #bb4444;">join</span><span style="color: #666666;">();</span>
<span style="color: #666666;">}</span> <span style="color: #aa22ff; font-weight: bold;">catch</span> <span style="color: #666666;">(</span>InterruptedException e<span style="color: #666666;">)</span> <span style="color: #666666;">{</span>
e<span style="color: #666666;">.</span><span style="color: #bb4444;">printStackTrace</span><span style="color: #666666;">();</span>
<span style="color: #666666;">}</span>
<span style="color: #666666;">}</span>
<span style="color: #666666;">}</span>
</pre>
</div>
The logs are where it gets interesting:<br />
<div style="background: rgb(248, 248, 248); border-width: 0.1em 0.1em 0.1em 0.8em; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="line-height: 125%; margin: 0px;">Pie has been saved
Eater Eater0 ready to start eating
Eater Eater1 ready to start eating
Eater0 Starting up on StrawBerryPie
Eater Eater2 ready to start eating
Eater1 Starting up on StrawBerryPie
Eater2 Starting up on StrawBerryPie
Eater2 looking to eat. Available Pieces 10.Last eaten by null [Version] 1
Eater2 looking to eat. Available Pieces 9.Last eaten by Eater2 [Version] 2
Eater0 looking to eat. Available Pieces 8.Last eaten by Eater2 [Version] 3
Eater1 looking to eat. Available Pieces 8.Last eaten by Eater2 [Version] 3
Eater2 looking to eat. Available Pieces 8.Last eaten by Eater2 [Version] 3
Saving pie for Eater1 failed. <span style="color: red;">The conditional request failed (Service:
AmazonDynamoDBv2; Status Code: 400; Error Code: ConditionalCheckFailedException;
Request ID: D2DHJL8RIN0VG91JMN1PJ23FDNVV4KQNSO5AEMVJF66Q9ASUAAJG; Proxy: null)</span>
Saving pie for Eater2 failed. <span style="color: red;">The conditional request failed (Service:
AmazonDynamoDBv2; Status Code: 400; Error Code: ConditionalCheckFailedException;
Request ID: 70FQ1PMILO5VDH37V019ONDA8RVV4KQNSO5AEMVJF66Q9ASUAAJG; Proxy: null)</span>
Eater0 looking to eat. Available Pieces 7.Last eaten by Eater0 [Version] 4
Eater1 looking to eat. Available Pieces 7.Last eaten by Eater0 [Version] 4
Eater2 looking to eat. Available Pieces 7.Last eaten by Eater0 [Version] 4
Saving pie for Eater1 failed. <span style="color: red;">The conditional request failed (Service:
AmazonDynamoDBv2; Status Code: 400; Error Code: ConditionalCheckFailedException;
Request ID: Q6OJPQVPA9TO8PKGCNFCEDGU6RVV4KQNSO5AEMVJF66Q9ASUAAJG; Proxy: null)</span>
Saving pie for Eater2 failed. <span style="color: red;">The conditional request failed (Service:
AmazonDynamoDBv2; Status Code: 400; Error Code: ConditionalCheckFailedException;
Request ID: 05RMKQ3UIE3I27U34DNPJUNF7BVV4KQNSO5AEMVJF66Q9ASUAAJG; Proxy: null)</span>
Eater0 looking to eat. Available Pieces 6.Last eaten by Eater0 [Version] 5
Eater1 looking to eat. Available Pieces 6.Last eaten by Eater0 [Version] 5
Eater2 looking to eat. Available Pieces 6.Last eaten by Eater0 [Version] 5
Saving pie for Eater1 failed. <span style="color: red;">The conditional request failed (Service:
AmazonDynamoDBv2; Status Code: 400; Error Code: ConditionalCheckFailedException;
Request ID: 9I5RF7P75UHCT8Q836ENQ8UHDFVV4KQNSO5AEMVJF66Q9ASUAAJG; Proxy: null)</span>
Saving pie for Eater2 failed. <span style="color: red;">The conditional request failed (Service:
AmazonDynamoDBv2; Status Code: 400; Error Code: ConditionalCheckFailedException;
Request ID: JQIFHNBA6O5J4V0CTIQ1LBQCJ3VV4KQNSO5AEMVJF66Q9ASUAAJG; Proxy: null)</span>
Eater0 looking to eat. Available Pieces 5.Last eaten by Eater0 [Version] 6
Eater1 looking to eat. Available Pieces 5.Last eaten by Eater0 [Version] 6
Eater2 looking to eat. Available Pieces 5.Last eaten by Eater0 [Version] 6
Saving pie for Eater1 failed. <span style="color: red;">The conditional request failed (Service:
AmazonDynamoDBv2; Status Code: 400; Error Code: ConditionalCheckFailedException;
Request ID: V7KE115NFLH2HJ6DCU2J5JHP33VV4KQNSO5AEMVJF66Q9ASUAAJG; Proxy: null)</span>
Saving pie for Eater2 failed. <span style="color: red;">The conditional request failed (Service:
AmazonDynamoDBv2; Status Code: 400; Error Code: ConditionalCheckFailedException;
Request ID: HA1P2B0SL1GV210AD5ROTCKE07VV4KQNSO5AEMVJF66Q9ASUAAJG; Proxy: null)
Eater0 looking to eat. Available Pieces 4.Last eaten by Eater0 [Version] 7</span>
Eater1 looking to eat. Available Pieces 4.Last eaten by Eater0 [Version] 7
Eater2 looking to eat. Available Pieces 4.Last eaten by Eater0 [Version] 7
Saving pie for Eater1 failed. <span style="color: red;">The conditional request failed (Service:
AmazonDynamoDBv2; Status Code: 400; Error Code: ConditionalCheckFailedException;
Request ID: CP0834PJJ5G5TQ21I9PJV6JMRJVV4KQNSO5AEMVJF66Q9ASUAAJG; Proxy: null)</span>
Saving pie for Eater2 failed. <span style="color: red;">The conditional request failed (Service:
AmazonDynamoDBv2; Status Code: 400; Error Code: ConditionalCheckFailedException;
Request ID: LASU1G7US4B5HVK3D8KBNUKEQ3VV4KQNSO5AEMVJF66Q9ASUAAJG; Proxy: null)</span>
Eater0 looking to eat. Available Pieces 3.Last eaten by Eater0 [Version] 8
Eater1 looking to eat. Available Pieces 3.Last eaten by Eater0 [Version] 8
Eater2 looking to eat. Available Pieces 3.Last eaten by Eater0 [Version] 8
Saving pie for Eater1 failed. <span style="color: red;">The conditional request failed (Service:
AmazonDynamoDBv2; Status Code: 400; Error Code: ConditionalCheckFailedException;
Request ID: 98LIVGVP3MET3C8TRNIT5AQT7FVV4KQNSO5AEMVJF66Q9ASUAAJG; Proxy: null)</span>
Saving pie for Eater2 failed. <span style="color: red;">The conditional request failed (Service:
AmazonDynamoDBv2; Status Code: 400; Error Code: ConditionalCheckFailedException;
Request ID: VT6O08ORKITUJTPFS0A9LC6QN7VV4KQNSO5AEMVJF66Q9ASUAAJG; Proxy: null)</span>
Eater0 looking to eat. Available Pieces 2.Last eaten by Eater0 [Version] 9
Eater1 looking to eat. Available Pieces 2.Last eaten by Eater0 [Version] 9
Eater2 looking to eat. Available Pieces 2.Last eaten by Eater0 [Version] 9
Saving pie for Eater1 failed. <span style="color: red;">The conditional request failed (Service:
AmazonDynamoDBv2; Status Code: 400; Error Code: ConditionalCheckFailedException;
Request ID: TVQDNGK3MTOQJAMIL7MJEUS9DNVV4KQNSO5AEMVJF66Q9ASUAAJG; Proxy: null)</span>
Saving pie for Eater2 failed. <span style="color: red;">The conditional request failed (Service:
AmazonDynamoDBv2; Status Code: 400; Error Code: ConditionalCheckFailedException;
Request ID: 8TABGOACGFN2TFP11GT02VD0L7VV4KQNSO5AEMVJF66Q9ASUAAJG; Proxy: null)</span>
Eater0 looking to eat. Available Pieces 1.Last eaten by Eater0 [Version] 10
Eater1 looking to eat. Available Pieces 1.Last eaten by Eater0 [Version] 10
Eater2 looking to eat. Available Pieces 1.Last eaten by Eater0 [Version] 10
Saving pie for Eater1 failed. <span style="color: red;">The conditional request failed (Service:
AmazonDynamoDBv2; Status Code: 400; Error Code: ConditionalCheckFailedException;
Request ID: M814TCUGGRMV8LCIH37A3C30A3VV4KQNSO5AEMVJF66Q9ASUAAJG; Proxy: null)</span>
Saving pie for Eater2 failed. <span style="color: red;">The conditional request failed (Service:
AmazonDynamoDBv2; Status Code: 400; Error Code: ConditionalCheckFailedException;
Request ID: FF6O9NCI1AQ9A48AL83B4RLFTVVV4KQNSO5AEMVJF66Q9ASUAAJG; Proxy: null)</span>
Pie is over. Eater0 stopping
Pie is over. Eater1 stopping
Pie is over. Eater2 stopping
</pre>
</div>
The Bakery item looks as below:<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgcDt1cw_PSjnnuLEPiKS9G1FMpaPWJNeSHO6fJTtJO-OTUuQaGqNFE4UIw7-lNWWcnnCCR7wWYscNXX26vQb0qnY-xNe-wkKmm6dVqAEoKhkUopAb30nMJixyDxKEk34fpAlB6OiC0VGE/s1600/Entry.png" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="91" data-original-width="573" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgcDt1cw_PSjnnuLEPiKS9G1FMpaPWJNeSHO6fJTtJO-OTUuQaGqNFE4UIw7-lNWWcnnCCR7wWYscNXX26vQb0qnY-xNe-wkKmm6dVqAEoKhkUopAb30nMJixyDxKEk34fpAlB6OiC0VGE/s1600/Entry.png" /></a></div>
<br />
As seen whenever a Thread attempts to save the Pie Record and its 'read version' does not match what version exists in Dynamo, the code throws a ConditionalCheckFailedException.<br />
When the save succeeds, the mapper updates the record along with a new versionNumber.<br />
DynamoDb actually provides a graph of the exception count<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhP2YcojyVmUkt7Otwcxa5eG8Ufb9IPWOQa_wWAvtCRNk5ixtPxM1UUGeabsYELR4ObybJwdzfZBsQ76uI8JMRSWtlex0QmB2Mpci_mmqdodPI66Wdx5-A_hzVpQZ2VEv4TgF9NeG4dXXQ/s1600/ConditionalCheckFailedException.png" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="325" data-original-width="603" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhP2YcojyVmUkt7Otwcxa5eG8Ufb9IPWOQa_wWAvtCRNk5ixtPxM1UUGeabsYELR4ObybJwdzfZBsQ76uI8JMRSWtlex0QmB2Mpci_mmqdodPI66Wdx5-A_hzVpQZ2VEv4TgF9NeG4dXXQ/s1600/ConditionalCheckFailedException.png" /></a></div>
<br />
In the next post, Ill look to use pessimistic locking
</div>
Robin Varghesehttp://www.blogger.com/profile/00841917517097216645noreply@blogger.com0tag:blogger.com,1999:blog-6262128735818493674.post-56990750050889798012020-06-18T09:10:00.002+05:302020-06-18T09:10:57.777+05:30Dynamo Db - Good To Remember stuff<div dir="ltr" style="text-align: left;" trbidi="on">
Post is a collection of interesting pointers on the working of Dynamo Db. I wanted to have them in a single place as a ready reference<br />
<a name='more'></a><span style="font-size: small; font-weight: 400;">DynamoDB is a </span><span style="font-size: small;"><b>key value database.</b></span><br />
<div style="background: #f8f8f8; border-width: 0.1em 0.1em 0.1em 0.8em; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="line-height: 125%; margin: 0;"><span style="color: blue;">A key-value database stores data as a collection of key-value pairs in which a key
serves as a unique identifier.</span> Both keys and values can be anything, ranging from
simple objects to complex compound objects. <span style="color: blue;">Key-value databases are highly
partitionable and allow horizontal scaling at scales that other types of databases
cannot achieve</span>
</pre>
</div>
<h3 style="text-align: left;">
Tables in DynamoDb</h3>
<ul style="text-align: left;">
<li>No schema unlike a RDBMS table - you only provide the primary key. (You can define all attributes if you want - using the CreateTable action, but its not needed.)</li>
<li>DynamoDb "Item" is the equivalent of a table row.</li>
<li>There is no need for a commit after adding/updating/deleting an item. They are permanent</li>
</ul>
<h3 style="text-align: left;">
What's inside the Item ?</h3>
<div>
<ul style="text-align: left;">
<li>Each Item has the primary key and zero or more attributes. No Limit on number of attributes( Although Item size cannot exceed 400KB)</li>
<li>Primary Key can be composed of a single attribute called Partition Key or of two attributes (Partition Key and Range Key)</li>
</ul>
</div>
<div style="background: #f8f8f8; border-width: 0.1em 0.1em 0.1em 0.8em; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="line-height: 125%; margin: 0px; text-align: left;">A simple primary key or hash key, composed of one attribute known as the partition key.
<span style="color: blue;">DynamoDB uses the partition key's value as input to an internal hash
function. The output from the hash function determines the partition (physical
storage internal to DynamoDB) in which the item will be stored.</span>
In a table that has only a partition key, no two items can have the same
partition key value.</pre>
</div>
<div style="text-align: left;">
</div>
<ul style="text-align: left;">
<li>When two attributes form the primary key, two items can have same hash key, but the range (or sort key) has to be different. If sort key exists, then items with same partition key are stored sorted by the range key. </li>
<li>What are the benefit of having a sort key ? As Dynamo Db is a key value store, all its queries are of the form "key=value". So if we have only Partition key, we can only do equality based lookups on it. The sort key however does not have this constraint. With Sort Key we can apply non equals constraints</li>
</ul>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhh6as1zzB20SWZOKgpLv7wZhoeTa0WHUk7GINX-GW8Zi8glMbGXCy756h9dX37RCBUsJNyqS6sw4iAVoZlvrzi7feL6zH8MvJ1lHmBFmeg_N86lL17v52SgGfEqGpmj13kVrVb5F8T1rM/s1600/DbSortKey.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="265" data-original-width="826" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhh6as1zzB20SWZOKgpLv7wZhoeTa0WHUk7GINX-GW8Zi8glMbGXCy756h9dX37RCBUsJNyqS6sw4iAVoZlvrzi7feL6zH8MvJ1lHmBFmeg_N86lL17v52SgGfEqGpmj13kVrVb5F8T1rM/s1600/DbSortKey.png" /></a></div>
<div>
As seen here, the sort key allows us to efficiently retrieve records (I personally love 'Begins with')</div>
<h3 style="text-align: left;">
Storing of Data and Retrieval</h3>
<div>
<ul style="text-align: left;">
<li>The data in DynamoDb is stored in partitions. (10 GB storage units (<a href="https://aws.amazon.com/blogs/database/choosing-the-right-dynamodb-partition-key/" target="_blank">Link</a>) - but that could have changed)</li>
</ul>
<div style="background: #f8f8f8; border-width: 0.1em 0.1em 0.1em 0.8em; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="line-height: 125%; margin: 0;">A partition is an allocation of storage for a table, backed by <span style="color: blue;">solid state drives
(SSDs)</span> and <span style="color: blue;">automatically replicated</span> across multiple Availability Zones within an
AWS Region. Partition management is handled entirely by DynamoDB — you never have
to manage partitions yourself.</pre>
</div>
</div>
<ul style="text-align: left;">
<li>DynamoDb allocates additional partitions - if an existing partition fills up or existing partitions cannot support any changes to table throughput.</li>
<li>DynamoDb's performance is closely tied to the partitions - Single Partition in Dynamo supports a maximum of 3000 read capacity units (RCUs) or 1000 write capacity units (WCUs).</li>
<li>The Read and Write capacity of DynamoDb is evenly divided among the table's physical partitions. <i>Therefore, a partition key design that doesn't distribute I/O requests evenly can create "hot" partitions that result in throttling and use your provisioned I/O capacity inefficiently (<a href="https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/bp-partition-key-uniform-load.html" target="_blank">Doc Link</a>).</i></li>
</ul>
<div style="background: #f8f8f8; border-width: 0.1em 0.1em 0.1em 0.8em; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="line-height: 125%; margin: 0;">The optimal usage of a table's provisioned throughput depends not only on the
workload patterns of individual items, but also on the partition key design.
This doesn't mean that you must access all partition key values to achieve an
efficient throughput level, or even that the percentage of accessed partition
key values must be high. It does mean that the more distinct partition key
values that your workload accesses, the more those requests will be spread
across the partitioned space. <b>In general, you will use your provisioned throughput
more efficiently as the ratio of partition key values accessed to the total
number of partition key values increases.</b></pre>
</div>
<h3 style="text-align: left;">
Local Secondary Indexes</h3>
<ul style="text-align: left;">
<li><span style="color: blue;">Dynamo Db equivalent of an SQL Index.</span> The goal of creating an index on a particular table in a database is to make it faster to search through the table and find the row or rows that we want.</li>
</ul>
<div style="background: #f8f8f8; border-width: 0.1em 0.1em 0.1em 0.8em; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="line-height: 125%; margin: 0;">A secondary index lets you query the data in the table using an alternate key,
in addition to queries against the primary key.
In case of Local Secondary Index, it has the same partition key as the table,
but a different sort key.
A local secondary index is "local" in the sense that every partition of a
local secondary index is scoped to a base table partition that has the same
partition key value.
Local secondary indexes are created at the same time that you create a table.
You cannot add a local secondary index to an existing table, nor can you
delete any local secondary indexes that currently exist.</pre>
</div>
<ul style="text-align: left;">
<li><span style="font-weight: normal;">
With Indexes you get pointers to the data, so DynamoDb will need to do additional read to fetch the data from the partition. </span></li>
</ul>
<div style="background: #f8f8f8; border-width: 0.1em 0.1em 0.1em 0.8em; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="line-height: 125%; margin: 0px; text-align: left;">to retrieve any additional attributes, DynamoDB must perform additional read operations
against the Thread table. These extra reads are known as fetches, and <span style="color: blue;">they can increase
</span><div style="text-align: left;">
<span style="color: blue;">the total amount of provisioned throughput required for a query.</span>
From an application's point of view, fetching additional attributes from the base table is
automatic and transparent, so there is no need to rewrite any application logic. However,
such fetching can greatly reduce the performance advantage of using a local secondary index.</div>
</pre>
</div>
To avoid extra read costs, DynamoDb can copy additional attributes to the index. This ensures Index read will return all values needed from this record. The attributes to be copied can also be configured to be none/some/all attributes. (The partition key and sort key of the table are always projected into the index)<br />
<br />
<ul style="text-align: left;">
<li>Local secondary indexes on a table are created when the table is created. When you delete a table, any local secondary indexes on that table are also deleted.</li>
</ul>
<br />
<h3 style="text-align: left;">
Global Secondary Index
</h3>
<div>
<ul style="text-align: left;">
<li><span style="color: blue;">Dynamo Db equivalent of a single table view in SQL</span>. With this index you have a hash key and a range key.</li>
<li>The partition key and sort key of the table are always projected into the index. Additional attributes can be included as needed.</li>
<li>The updates to GSI happen in an async manner. Updates are eventually consistent.</li>
</ul>
</div>
</div>
Robin Varghesehttp://www.blogger.com/profile/00841917517097216645noreply@blogger.com0tag:blogger.com,1999:blog-6262128735818493674.post-25865090208940141092020-06-15T05:25:00.000+05:302020-06-15T05:25:33.308+05:30Container Deployments - some basic concepts and EKS<div dir="ltr" style="text-align: left;" trbidi="on">
I came across AWS's container solutions - EKS. But I realized that to move forward, I actually needed a primer on Containers. The post is a revision on Containerization. I have liberally copied of the <a href="https://kubernetes.io/docs" target="_blank">Kubernetes Website</a> and from <a href="https://docs.docker.com/get-started/overview/" target="_blank">Docker website</a> - as I did my studies from there.<br />
<a name='more'></a>To start with Containers are a terminology associated with the world of application runtime. We run our applications on physical servers on premise. Or we run it on cloud. So where does the Container model of execution fit in ?<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj-U_ESSVkJKyWyB_rVcDKlz2oNO5grWcuOb2IVXFLkSHI0fxHuw0JY_YWVIQKaDuvqug_Q5rHHYsH8g2vV_MIVh_-UaVpfYBTHWD_PX8Waao7sY2kLROGmiQp9-suiV-D_WXoudKXXFew/s1600/ContainerVsVmsVsTraditional.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1600" data-original-width="757" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj-U_ESSVkJKyWyB_rVcDKlz2oNO5grWcuOb2IVXFLkSHI0fxHuw0JY_YWVIQKaDuvqug_Q5rHHYsH8g2vV_MIVh_-UaVpfYBTHWD_PX8Waao7sY2kLROGmiQp9-suiV-D_WXoudKXXFew/s1600/ContainerVsVmsVsTraditional.png" /></a></div>
As seen both Container deployment and Virtualized deployment offer significant benefits over the traditional model. The growth of cloud platforms led to increase in popularity of Virtual machines and this kind of an execution environment. Container Deployment is the next stage in this evolution.
So what makes Containers popular ?<br />
<ol style="text-align: left;">
<li><span style="color: red;">Virtual machines may take up a lot of system resources of the host machine</span>, being many GBs in size. <span style="color: red;">Running a single app on a virtual server means running a copy of an operating system as well as a virtual copy of all the hardware required for the system to run.</span> This quickly adds up to a lot of RAM and CPU cycles. <span style="color: blue;">Containers can be as small as 10MB.</span> This makes containers remarkably <span style="color: blue;">lightweight and fast to launch</span> as opposed to deploying virtual machines, where the entire operating system needs to be deployed.</li>
<li>The process of <span style="color: red;">relocating an app running on a virtual machine can also be complicated</span> as it is always attached to the operating system. Hence, you have to migrate the app as well as the OS with it. Also, <span style="color: red;">when creating a virtual machine, the hypervisor allocates hardware resources dedicated to the VM. </span><span style="color: blue;">As Containers are small, you can quickly scale in and out of containers and add identical containers.</span></li>
</ol>
This makes the guest OS sound like a bad thing. But it also gives some benefits that containers do not have
<br />
<br />
<ol style="text-align: left;">
<li>Each VM is completely isolated from the host operating system. Containers share the OS, so they are process-level isolated. One container could potentially affect others by compromising the stability of the operating system.</li>
<li>Due to the reduced isolation, containers do not offer the same security and stability that VMs can.</li>
</ol>
<div>
From what I read, Container technology is working on ever improving solutions to minimize the above factors.<br />
<br />
<b>But what if we could combine VMs and Containers ?</b><br />
Consider that we have multiple VMs and we want to deploy containers on each of them. With this approach you get all the container benefits and Virtual machines benefits.<br />
<br /></div>
<div>
Now that we know the difference between the two, the next question would be <b>what is Kubernetes</b> ?</div>
<div style="background: #f8f8f8; border-width: 0.1em 0.1em 0.1em 0.8em; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="line-height: 125%; margin: 0;">Kubernetes is a portable, extensible, open-source platform for managing
containerized workloads and services, that facilitates both declarative
configuration and automation.
</pre>
</div>
Of course this definition does not explain much. Sounds more like propaganda (It is from the Kubernetes website). From the same website however, this statement is more relevant:<!-- HTML generated using hilite.me --><br />
<div style="background: #f8f8f8; border-width: 0.1em 0.1em 0.1em 0.8em; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="line-height: 125%; margin: 0px; text-align: left;"><span style="color: blue;">
Kubernetes provides you with a framework to run distributed systems
resiliently.</span> It takes care of scaling and failover for your application,
provides deployment patterns, and more.
</pre>
</div>
Kubernetes is a software that allows you to create containers on a machine and run them. It performs<br />
<ul style="text-align: left;">
<li>Container management (health checks, container replacement), </li>
<li>efficient utilization of systems(allocating the right containers to right instances based on container memory, CPU requirement), </li>
<li>scaling (load balancer behavior if the container application is a web service), </li>
<li>integration with storage systems,</li>
<li> configuration management etc.</li>
</ul>
<br />
So thats where we stand. <b>How do we combine this with AWS ?</b><br />
<br />
AWS provides Virtual Machines - EC2 instances that we can setup within our AWS account. AWS also provides Kuberenetes - <b>Amazon Elastic Kubernetes Service (EKS)</b>
<br />
<div style="background: #f8f8f8; border-width: 0.1em 0.1em 0.1em 0.8em; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="line-height: 125%; margin: 0;">AWS makes it easy to run Kubernetes. You can choose to manage Kubernetes
infrastructure yourself with Amazon EC2 or get an automatically provisioned,
managed Kubernetes control plane with Amazon EKS. Either way, you get
powerful, community-backed integrations to AWS services like VPC, IAM, and
service discovery as well as the security, scalability, and
high-availability of AWS.
</pre>
</div>
<br />
And with this brief knowledge begins my journey into EKS.</div>
Robin Varghesehttp://www.blogger.com/profile/00841917517097216645noreply@blogger.com1tag:blogger.com,1999:blog-6262128735818493674.post-11945247394411348962020-06-15T02:04:00.000+05:302020-06-15T02:04:06.114+05:30EC2 and security groups<div dir="ltr" style="text-align: left;" trbidi="on">
In the previous post we setup an EC2 instance and tried to connect it using SSH:
<br />
<div style="background: #f8f8f8; border-width: 0.1em 0.1em 0.1em 0.8em; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="line-height: 125%; margin: 0;"><span style="color: #888888;">
ssh -i ~/Desktop/test-burner-pair.pem ec2-54-237-224-254.compute-1.amazonaws.com
ssh: connect to host ec2-54-237-224-254.compute-1.amazonaws.com port 22:
Operation timed out</span>
</pre>
</div>
My host has a public address, the subnet has a gateway that allows connection to internet, so ideally it should have worked. However there is additional setting with EC2 - <b>security groups</b><br />
<a name='more'></a><!-- HTML generated using hilite.me --><br />
<div style="background: #f8f8f8; border-width: 0.1em 0.1em 0.1em 0.8em; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="line-height: 125%; margin: 0;"><span style="color: blue;">A security group acts as a virtual firewall for your instance to control incoming
and outgoing traffic.</span> Inbound rules control the incoming traffic to your instance,
and outbound rules control the outgoing traffic from your instance.
</pre>
</div>
The current security group is<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiz9D2zawEuWFocEpojUZXR-YCrMPaejY3_jdKtTYr-OS6M209i6LZCNgj7h7S3FzoY9E4LVMn0Os49QqchdooP64KhYS3fviGJLIpkwo8Ev5Kv4l-22sAEOkH868_a8KxnOa87j648xfM/s1600/EC2SecurityGroup.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" data-original-height="521" data-original-width="761" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiz9D2zawEuWFocEpojUZXR-YCrMPaejY3_jdKtTYr-OS6M209i6LZCNgj7h7S3FzoY9E4LVMn0Os49QqchdooP64KhYS3fviGJLIpkwo8Ev5Kv4l-22sAEOkH868_a8KxnOa87j648xfM/s1600/EC2SecurityGroup.png" /></a></div>
For my SSH to work I need to define a security group inbound rule.<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjtWjpRbcZnTmWQKAAjkVYqCALDkC-pXSMpwY8ICSK257ckZElVV7VcZDyJlMknfIS9p12NDG274FM6ERQvUEJaBuo5wY2N7qDatIAIvM8Og71TPdPnTAw7GGRDSNJ5DzTZpuBs07Szpcw/s1600/ec2-securitygroup.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="265" data-original-width="807" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjtWjpRbcZnTmWQKAAjkVYqCALDkC-pXSMpwY8ICSK257ckZElVV7VcZDyJlMknfIS9p12NDG274FM6ERQvUEJaBuo5wY2N7qDatIAIvM8Og71TPdPnTAw7GGRDSNJ5DzTZpuBs07Szpcw/s1600/ec2-securitygroup.png" /></a><span style="text-align: left;">This is basically allowing my laptop to connect to my ec2 host using the security key (my pem file).</span></div>
<br /></div>
Robin Varghesehttp://www.blogger.com/profile/00841917517097216645noreply@blogger.com1tag:blogger.com,1999:blog-6262128735818493674.post-80966340221629314262020-06-07T02:02:00.001+05:302020-06-07T02:02:38.833+05:30EC2 and networking<div dir="ltr" style="text-align: left;" trbidi="on">
In the <a href="https://learningviacode.blogspot.com/2020/06/aws-ec2-setting-up-instance.html" target="_blank">previous post</a>, we setup an EC2 instance. I wanted to play around with the configurations available here.<br />
<a name='more'></a><div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhg0xOg3ebHlu8WBK7Nk2ZfkxlVey-I_tpmjPDwKRcNhJ994VYOYyeOH4COl3HDfQzM7uvOUpMcdUQY9dD-JoNqKVb8zptgDVCaxeftDAQwT0kQ3SdyufVk59t8EVDv3yyWxw0E8elrZ7o/s1600/EC2Host.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" data-original-height="225" data-original-width="816" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhg0xOg3ebHlu8WBK7Nk2ZfkxlVey-I_tpmjPDwKRcNhJ994VYOYyeOH4COl3HDfQzM7uvOUpMcdUQY9dD-JoNqKVb8zptgDVCaxeftDAQwT0kQ3SdyufVk59t8EVDv3yyWxw0E8elrZ7o/s1600/EC2Host.png" /></a></div>
This is the host we created in the last post.
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiGST56enE8ek6vz0IJ4fRECfNty2foYYFV804bHcMmJQCd6XSHXssRIY40E-9uva3gX5O40_s2PWFVShxIVXWW4afU_Dk7jKA82kAeVP3URPb0glFml7dLW1BM0-FjPp12NpVA8nwOqJE/s1600/EC2HostDetails.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" data-original-height="838" data-original-width="810" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiGST56enE8ek6vz0IJ4fRECfNty2foYYFV804bHcMmJQCd6XSHXssRIY40E-9uva3gX5O40_s2PWFVShxIVXWW4afU_Dk7jKA82kAeVP3URPb0glFml7dLW1BM0-FjPp12NpVA8nwOqJE/s1600/EC2HostDetails.png" /></a></div>
The instance type and AMI is as selected during creation. The ec2 instance has what is called an <b>elastic network interface </b>(eth0) - it represents a virtual network card associated with the ec2 instance.<br />
<!-- HTML generated using hilite.me --><br />
<div style="background: #f8f8f8; border-width: 0.1em 0.1em 0.1em 0.8em; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="line-height: 125%; margin: 0;">This AWS resource is referred to as a network interface in the AWS Management
Console and the Amazon EC2 API. <span style="color: blue;">Every instance in a VPC has a default network
interface, called the primary network interface.</span> You cannot detach a primary
network interface from an instance. You can create and attach additional network
interfaces. The maximum number of network interfaces that you can use varies by
instance type.
A Network Interface can include
<span style="color: blue;">A primary private IPv4 address</span> from the IPv4 address range of your VPC
One or more secondary private IPv4 addresses from the IPv4 address range of your VPC
One Elastic IP address (IPv4) per private IPv4 address
<span style="color: blue;">One public IPv4 address</span>
One or more IPv6 addresses
One or more security groups
A MAC address
A source/destination check flag
A description
</pre>
</div>
We can modify subnet settings to ensure that instances launched in a subnet do not include a public IPv4 address.<br />
<div style="background: #f8f8f8; border-width: 0.1em 0.1em 0.1em 0.8em; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="line-height: 125%; margin: 0;"><span style="color: blue;">The public IPv4 address is assigned from Amazon's pool of public IPv4 addresses.
</span>When you launch an instance, the IP address is assigned to the primary network
interface that's created.
When you create a network interface, it inherits the public IPv4 addressing
attribute from the subnet. If you later modify the public IPv4 addressing attribute
of the subnet, the network interface keeps the setting that was in effect when
it was created.
</pre>
</div>
There are additional details about VPC, subnet and so on. Let's dive into what they mean.<br />
The EC2 instance is created in the default VPC. <b>All EC2 instances are created by default in the default VPC</b>.<br />
<div>
<b>AWS provides a default VPC for each region.</b> The host was created in 1e availability zone.<br />
<ul>
</ul>
<div style="background: #f8f8f8; border-width: 0.1em 0.1em 0.1em 0.8em; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="line-height: 125%; margin: 0px; text-align: left;">Amazon Virtual Private Cloud (Amazon VPC) enables you to launch AWS resources
into a virtual network that you've defined. This virtual network closely
resembles a traditional network that you'd operate in your own data center
<span style="color: blue;">VPC is a virtual network dedicated to your AWS account.</span> It is logically isolated
from other virtual networks in the AWS Cloud. You can launch your AWS resources,
such as Amazon EC2 instances, into your VPC
</pre>
</div>
The subnet is created under the default VPC. Each subnet is associated with an AZ. This once is for instance associated with us-east-1e.<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEheyk5li8FtV4E3SF0wLAaujCoRbDU5cnZnEUs76YkH0gCwT8GPwAUJU7Kqg3QpDFOK-1Y_ZVHPxHusSey9av5zCnxjLpIVFd_c1LgmXAKla152zSthGIT4CUtlYdWXWIRhWgAU14wZfTk/s1600/Subnet.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="302" data-original-width="728" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEheyk5li8FtV4E3SF0wLAaujCoRbDU5cnZnEUs76YkH0gCwT8GPwAUJU7Kqg3QpDFOK-1Y_ZVHPxHusSey9av5zCnxjLpIVFd_c1LgmXAKla152zSthGIT4CUtlYdWXWIRhWgAU14wZfTk/s1600/Subnet.png" /></a></div>
<div>
The subnets are all default subnets (associated with default vpc)</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiB8aAJpQDIrgmqFvBtX5FklcIPiOCriADT-60uwTKqKBqg2BBAjAKTT5JALgxSR09B9x0g2LZ9RVRzcUsf-K8dHXSP6DAzdic7VBlgkC9-KEfb6dygZHBkS9hmaqQthEsnnmhuH01tbNA/s1600/DefaultSubnets.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" data-original-height="211" data-original-width="1091" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiB8aAJpQDIrgmqFvBtX5FklcIPiOCriADT-60uwTKqKBqg2BBAjAKTT5JALgxSR09B9x0g2LZ9RVRzcUsf-K8dHXSP6DAzdic7VBlgkC9-KEfb6dygZHBkS9hmaqQthEsnnmhuH01tbNA/s1600/DefaultSubnets.png" /></a></div>
A <b>subnet or subnetwork </b>is a sub-section of a network. In most cases, the subnet includes all the computers setup in a specific location. In case of AWS, all the EC2 instances setup in the same Availability zone under the VPC would associate with one subnet.
The default subnets are all public subnets - A public subnet is a sub-network that has access to the internet.<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEglWkTYBMzO1IEHa1sIFbSuxNLtgyw0SRxNiTHmGMy-reLTzvlSDifCAlCoudbRivxTRmxEWwGWg0rNmv31Lv3hWe0jj1IIzoU6Fql-gwLp3c3jiDDkB8nNMVlrQJdYOEAw5EGAjS54pgI/s1600/SubnetPublic.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" data-original-height="468" data-original-width="700" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEglWkTYBMzO1IEHa1sIFbSuxNLtgyw0SRxNiTHmGMy-reLTzvlSDifCAlCoudbRivxTRmxEWwGWg0rNmv31Lv3hWe0jj1IIzoU6Fql-gwLp3c3jiDDkB8nNMVlrQJdYOEAw5EGAjS54pgI/s1600/SubnetPublic.png" /></a></div>
What we see here is the <b>Route table</b>. Every Subnet has a route table. The route table defines the routes to determine where the traffic is to be routed. We can think of it as something similar to the router we have at home. The Internet Gateway on the other hand knows how to communicate with the internet. We can think of it as something similar to the modem we use to connect to the internet.<br />
<br />
We could setup a private subnet. That would be a subnet without internet access. This subnet would be one without an Internet Gateway mapping in its route table. Such a subnet would allow the instances within it to communicate with other hosts in the VPC. But it wouldn't be able to connect to anything outside the VPC.<br />
<br />
The setup could be explained with this diagram:<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhaopC5iy-4Ij1OE-mB1jgV8Tq3GuiK7ajFBhCMilCCr71gyCAnw9htemb9KrVxXnoDjFilYiaeIv7IlE0csT1hFZ6znHemhPmHyA_I1H752BXn5EifiPIQQDIKqS8sdrS5g6x9lryUKIU/s1600/VpcBlockDiag.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" data-original-height="515" data-original-width="1139" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhaopC5iy-4Ij1OE-mB1jgV8Tq3GuiK7ajFBhCMilCCr71gyCAnw9htemb9KrVxXnoDjFilYiaeIv7IlE0csT1hFZ6znHemhPmHyA_I1H752BXn5EifiPIQQDIKqS8sdrS5g6x9lryUKIU/s1600/VpcBlockDiag.png" /></a></div>
<br />
<br />
<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
</div>
<br />
<ol style="text-align: left;">
<li>AWS allows the creation of VPC similar to what you would get in a datacenter (or similar to the private network you have at your home)</li>
<li>Every AWS Account comes with a default VPS setup for each region. This default VPC spans across all the Availability Zones for that region. So if you create an application and deploy it across multiple AZs, the EC2 instances would have no problem communicating with each other.</li>
<li>The networking setup that spans across an AZ in a VPC is called a subnet or a sub network. Each subnet consists of a Route Table. The route table tells how to route messages to any destination.</li>
<li>So Instance 2 in above diagram is capable of communicating with Instance 1 (same subnet) and Instance 3 (different subnet within same VPC). The route table includes this information</li>
<li>Instance 1 and Instance 2 can also communicate across the internet. This is possible because the hosts are in a Subnet that is connected to an Internet Gateway. (This information is also present in the Route table).</li>
<li>Instance 3 cannot communicate with internet. This is because Subnet 2 is a private subnet. It is not connected to an Internet Gateway.</li>
</ol>
</div>
</div>
Robin Varghesehttp://www.blogger.com/profile/00841917517097216645noreply@blogger.com1tag:blogger.com,1999:blog-6262128735818493674.post-55350553205188401592020-06-06T23:04:00.001+05:302020-06-06T23:04:29.452+05:30AWS EC2 - setting up an instance<div dir="ltr" style="text-align: left;" trbidi="on">
Everything that can be written about EC2 has already been written. I however like to make my own short notes and will use this entry as my personal notebook on EC2.<br />
<a name='more'></a>The definition for EC2 from <a href="https://aws.amazon.com/ec2/faqs/" target="_blank">aws docs</a>:
<br />
<div style="background: #f8f8f8; border-width: 0.1em 0.1em 0.1em 0.8em; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="line-height: 125%; margin: 0;">Amazon Elastic Compute Cloud (Amazon EC2) is a web service that provides
resizable compute capacity in the cloud
</pre>
</div>
OK, why should I use a Cloud offering ?<br />
<!-- HTML generated using hilite.me --><br />
<div style="background: #f8f8f8; border-width: 0.1em 0.1em 0.1em 0.8em; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="line-height: 125%; margin: 0;">1. EC2 comes with a web service interface to obtain and configure EC2
capacity with minimal friction.
2. EC2 allows users tp obtain and boot new server instances to minutes,
this means users can quickly scale capacity, both up and down, as
computing requirements change.
3. This means with EC2 you only pay for what you use.
</pre>
</div>
Lets start by setting up an EC2 instance.<br />
To create an EC2 instance, we need to select the AMI or <b>Amazon Machine Image</b>
<!-- HTML generated using hilite.me --><br />
<div style="background: #f8f8f8; border-width: 0.1em 0.1em 0.1em 0.8em; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="line-height: 125%; margin: 0;">An AMI is a template that contains the software configuration (operating system,
application server, and applications) required to launch your instance.
</pre>
</div>
The AMI page list AMIs from Amazon Linux, SUSE Linux, Red Hat Linux, Windows Server etc. I decided to setup one with a Windows Server and later one with Amazon Linux.<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh9xnaZ7ZIHLt5vllMjvaU6jrXWJVp97O8AlPStgJF8E1HKTiat69lk87hSJQYbZGrpIJsK0fuAjvjjc2EaEfJljd895ULMkHRLtFpQJlKqJyryjPOnrNNIbTEjTDDw1UI_9w-6XplJEvo/s1600/Ec2CreateStep1.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="169" data-original-width="692" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh9xnaZ7ZIHLt5vllMjvaU6jrXWJVp97O8AlPStgJF8E1HKTiat69lk87hSJQYbZGrpIJsK0fuAjvjjc2EaEfJljd895ULMkHRLtFpQJlKqJyryjPOnrNNIbTEjTDDw1UI_9w-6XplJEvo/s1600/Ec2CreateStep1.png" /></a></div>
<div class="separator" style="clear: both; text-align: center;">
</div>
<br />
The next step was to choose the instance type - or the hardware configurations
<!-- HTML generated using hilite.me --><br />
<div style="background: #f8f8f8; border-width: 0.1em 0.1em 0.1em 0.8em; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="line-height: 125%; margin: 0;">Instance types comprise varying combinations of CPU, memory, storage, and
networking capacity.
Each instance type includes one or more instance sizes, allowing you to
scale your resources to the requirements of your target workload.
</pre>
</div>
<a href="https://aws.amazon.com/ec2/instance-types/">Amazon Docs</a> indicate the variety of instance types - General Purpose, Compute Optimized, Memory Optimized etc. Each of these instance types include varied instance sizes.
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg-sa1UQsB_sFDkjzUBlGE6XrGeXKzqdYmSpc-w8PIlWrdMlCo5-8Y_hsYWv4VjI1pv48rcVl06jbUl19I5QMqH8pbRpfUeQSRSvPRddW-DKtsCj4XJace2jk3xYKyAJUeJanQUTtw08DU/s1600/Ec2CreateStep2.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="279" data-original-width="990" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg-sa1UQsB_sFDkjzUBlGE6XrGeXKzqdYmSpc-w8PIlWrdMlCo5-8Y_hsYWv4VjI1pv48rcVl06jbUl19I5QMqH8pbRpfUeQSRSvPRddW-DKtsCj4XJace2jk3xYKyAJUeJanQUTtw08DU/s1600/Ec2CreateStep2.png" /></a></div>
We can either launch from this step - the Ec2 host will be created with default configurations or we can do further configurations. Here I am opening up the detailed configurations screen for a look at the options:<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjbLElvc4a8_D25s1nXx4fiE3BvtZQNBQLgX6vPcTzn7u0H1yttduzkqdOWAUjh4FHOGf6wSTiKRGZL0UquLMYlrQVRr-xk99LwLP_w8dhFwyMX2zgf6fcbmoTsiNkQvnMza7hvL56uF3c/s1600/Ec2CreateStep3.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="772" data-original-width="861" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjbLElvc4a8_D25s1nXx4fiE3BvtZQNBQLgX6vPcTzn7u0H1yttduzkqdOWAUjh4FHOGf6wSTiKRGZL0UquLMYlrQVRr-xk99LwLP_w8dhFwyMX2zgf6fcbmoTsiNkQvnMza7hvL56uF3c/s1600/Ec2CreateStep3.png" /></a></div>
These are the default selections. The new EC2 host will be created in a VPC by default (more on VPCs later). The EC2 host will stop when shutdown is executed.<br />
Next is storage selection.<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgF6xbo_meBLdMp3lZXybMMHt0sQOJwyYE1SEa8iCV_zlgkZpO3WfEVf7bD0CYRk-VynC6_LqhyUuOk5pnTjS9rrp4s5VO4SAbW41KX-H2j3DnGabbsnCigKzkM2V7FCt8S2KCWAITEzFQ/s1600/Ec2CreateStep4.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="278" data-original-width="847" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgF6xbo_meBLdMp3lZXybMMHt0sQOJwyYE1SEa8iCV_zlgkZpO3WfEVf7bD0CYRk-VynC6_LqhyUuOk5pnTjS9rrp4s5VO4SAbW41KX-H2j3DnGabbsnCigKzkM2V7FCt8S2KCWAITEzFQ/s1600/Ec2CreateStep4.png" /></a></div>
The default storage is 30 GB of EBS SSD. The storage is released when the host is terminated. The next is Add Tags, which I skipped past for now. The last step is configuring security groups<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjbG0weiNafPA-k2iRpX9gTqHJZydcxLQhdSSOaHznRN2PkXVM88CVPz4xUPoG2sRMRJLd91_XOOfvUo4MetIYqc__G8j2kQW2iI0GM6qi1_Y7vi77FQJuONgew_X8ZGT7HA2Bf6n9tN8Q/s1600/Ec2CreateStep5.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="339" data-original-width="927" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjbG0weiNafPA-k2iRpX9gTqHJZydcxLQhdSSOaHznRN2PkXVM88CVPz4xUPoG2sRMRJLd91_XOOfvUo4MetIYqc__G8j2kQW2iI0GM6qi1_Y7vi77FQJuONgew_X8ZGT7HA2Bf6n9tN8Q/s1600/Ec2CreateStep5.png" /></a></div>
However to create the EC2 instance, there is one more step - We need an EC2 Key value pair to access the host.<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhvzyu44id9nzWmLlquyD8-RYW3m8Q2D5sq3wyyittLB8cc9BpGj0Om44zT6enhV4evIIM-HYp3sgGF-tN8uv2dbQbXdus2C9tZ0gsBs_eeRFaR6L7K5p5hQSSGKlkyBlB96ERLgAEPPIk/s1600/Ec2CreateStep6.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="511" data-original-width="682" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhvzyu44id9nzWmLlquyD8-RYW3m8Q2D5sq3wyyittLB8cc9BpGj0Om44zT6enhV4evIIM-HYp3sgGF-tN8uv2dbQbXdus2C9tZ0gsBs_eeRFaR6L7K5p5hQSSGKlkyBlB96ERLgAEPPIk/s1600/Ec2CreateStep6.png" /></a></div>
I saved the .pem file (which is the private key part of the key pair). This is the last step. The instance is now launched and can be connected to.<br />
The next post will look into further configuring the EC2 I setup.</div>
Robin Varghesehttp://www.blogger.com/profile/00841917517097216645noreply@blogger.com3tag:blogger.com,1999:blog-6262128735818493674.post-56205149577775489822020-06-04T09:32:00.001+05:302020-06-04T09:32:51.893+05:30Athena through API<div dir="ltr" style="text-align: left;" trbidi="on">
In the last post, we setup tables and database for our s3 data on Athena and queries it through the AWS console. In this post, I will attempt the same thing through Java SDK<br />
<a name='more'></a>The post follows Amazon sample code given in AWS Docs <a href="https://docs.aws.amazon.com/athena/latest/ug/code-samples.html#constants" target="_blank">here</a>.<br />
To connect to Athena from Java, we use the Athena Client.
<!-- HTML generated using hilite.me --><br />
<div style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="line-height: 125%; margin: 0px;"><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #aa22ff; font-weight: bold;">private</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #aa22ff; font-weight: bold;">static</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: yellow;">AthenaClient</span><span style="background-color: #f8f8f8;"> athenaClient</span><span style="background-color: #f8f8f8; color: #666666;">;</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #aa22ff; font-weight: bold;">public</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #aa22ff; font-weight: bold;">static</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #00bb00; font-weight: bold;">void</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #00a000;">main</span><span style="background-color: #f8f8f8; color: #666666;">(</span><span style="background-color: #f8f8f8;">String</span><span style="background-color: #f8f8f8; color: #666666;">[]</span><span style="background-color: #f8f8f8;"> args</span><span style="background-color: #f8f8f8; color: #666666;">)</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #aa22ff; font-weight: bold;">throws</span><span style="background-color: #f8f8f8;"> InterruptedException </span><span style="background-color: #f8f8f8; color: #666666;">{</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #aa22ff; font-weight: bold;">final</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: yellow;">AthenaClientBuilder</span><span style="background-color: #f8f8f8;"> builder </span><span style="background-color: #f8f8f8; color: #666666;">=</span><span style="background-color: #f8f8f8;"> AthenaClient</span><span style="background-color: #f8f8f8; color: #666666;">.</span><span style="background-color: #f8f8f8; color: #bb4444;">builder</span><span style="background-color: #f8f8f8; color: #666666;">()</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #666666;">.</span><span style="background-color: #f8f8f8; color: #bb4444;">region</span><span style="background-color: #f8f8f8; color: #666666;">(</span><span style="background-color: #f8f8f8;">Region</span><span style="background-color: #f8f8f8; color: #666666;">.</span><span style="background-color: #f8f8f8; color: #bb4444;">US_EAST_1</span><span style="background-color: #f8f8f8; color: #666666;">)</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #666666;">.</span><span style="background-color: #f8f8f8; color: #bb4444;">credentialsProvider</span><span style="background-color: #f8f8f8; color: #666666;">(</span><span style="background-color: #f8f8f8;">
StaticCredentialsProvider</span><span style="background-color: #f8f8f8; color: #666666;">.</span><span style="background-color: #f8f8f8; color: #bb4444;">create</span><span style="background-color: #f8f8f8; color: #666666;">(</span><span style="background-color: #f8f8f8;">
AwsBasicCredentials</span><span style="background-color: #f8f8f8; color: #666666;">.</span><span style="background-color: #f8f8f8; color: #bb4444;">create</span><span style="background-color: #f8f8f8; color: #666666;">(</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #bb4444;">"AccessKey"</span><span style="background-color: #f8f8f8; color: #666666;">,</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #bb4444;">"SecretKey"</span><span style="background-color: #f8f8f8; color: #666666;">)));</span><span style="background-color: #f8f8f8;">
athenaClient </span><span style="background-color: #f8f8f8; color: #666666;">=</span><span style="background-color: #f8f8f8;"> builder</span><span style="background-color: #f8f8f8; color: #666666;">.</span><span style="background-color: #f8f8f8; color: #bb4444;">build</span><span style="background-color: #f8f8f8; color: #666666;">();</span><span style="background-color: #f8f8f8;">
String queryExecutionId </span><span style="background-color: #f8f8f8; color: #666666;">=</span><span style="background-color: #f8f8f8;"> startQueryExecution</span><span style="background-color: #f8f8f8; color: #666666;">();</span><span style="background-color: #f8f8f8;">
completeQueryExecution</span><span style="background-color: #f8f8f8; color: #666666;">(</span><span style="background-color: #f8f8f8;">queryExecutionId</span><span style="background-color: #f8f8f8; color: #666666;">);</span><span style="background-color: #f8f8f8;">
processQueryResult</span><span style="background-color: #f8f8f8; color: #666666;">(</span><span style="background-color: #f8f8f8;">queryExecutionId</span><span style="background-color: #f8f8f8; color: #666666;">);</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #666666;">}</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #666666;">}</span><span style="background-color: #f8f8f8;">
</span></pre>
</div>
The method shows the three steps to be performed -<br />
<ol style="text-align: left;">
<li>Start a query execution on Athena.</li>
<li>Wait for execution to complete</li>
<li>Process the results of the query</li>
</ol>
The first part was to <b>execute the query</b>. I am using the same dataset from the previous post:
<br />
<div style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="line-height: 125%; margin: 0px;"><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #aa22ff; font-weight: bold;">private</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #aa22ff; font-weight: bold;">static</span><span style="background-color: #f8f8f8;"> String </span><span style="background-color: #f8f8f8; color: #00a000;">startQueryExecution</span><span style="background-color: #f8f8f8; color: #666666;">()</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #666666;">{</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #008800; font-style: italic;">//The database and data catalog context in which the query execution occurs.</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: yellow;">QueryExecutionContext</span><span style="background-color: #f8f8f8;"> queryExecutionContext </span><span style="background-color: #f8f8f8; color: #666666;">=</span><span style="background-color: #f8f8f8;"> QueryExecutionContext</span><span style="background-color: #f8f8f8; color: #666666;">.</span><span style="background-color: #f8f8f8; color: #bb4444;">builder</span><span style="background-color: #f8f8f8; color: #666666;">()</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #666666;">.</span><span style="background-color: #f8f8f8; color: #bb4444;">database</span><span style="background-color: #f8f8f8; color: #666666;">(</span><span style="background-color: #f8f8f8; color: #bb4444;">"users_db"</span><span style="background-color: #f8f8f8; color: #666666;">)</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #666666;">.</span><span style="background-color: #f8f8f8; color: #bb4444;">build</span><span style="background-color: #f8f8f8; color: #666666;">();</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #008800; font-style: italic;">// The result configuration specifies where the results of the query</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #008800; font-style: italic;">// should go in S3 and encryption options</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: yellow;">ResultConfiguration</span><span style="background-color: #f8f8f8;"> resultConfiguration </span><span style="background-color: #f8f8f8; color: #666666;">=</span><span style="background-color: #f8f8f8;"> ResultConfiguration</span><span style="background-color: #f8f8f8; color: #666666;">.</span><span style="background-color: #f8f8f8; color: #bb4444;">builder</span><span style="background-color: #f8f8f8; color: #666666;">()</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #008800; font-style: italic;">// You can provide encryption options for the output that is written.</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #008800; font-style: italic;">// .withEncryptionConfiguration(encryptionConfiguration)</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #666666;">.</span><span style="background-color: #f8f8f8; color: #bb4444;">outputLocation</span><span style="background-color: #f8f8f8; color: #666666;">(</span><span style="background-color: #f8f8f8; color: #bb4444;">"s3://athena-query-output-locn"</span><span style="background-color: #f8f8f8; color: #666666;">)</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #666666;">.</span><span style="background-color: #f8f8f8; color: #bb4444;">build</span><span style="background-color: #f8f8f8; color: #666666;">();</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #008800; font-style: italic;">// Create the StartQueryExecutionRequest to send to Athena which will start the query.</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: yellow;">StartQueryExecutionRequest</span><span style="background-color: #f8f8f8;"> startQueryExecutionRequest </span><span style="background-color: #f8f8f8; color: #666666;">=</span><span style="background-color: #f8f8f8;"> StartQueryExecutionRequest</span><span style="background-color: #f8f8f8; color: #666666;">.</span><span style="background-color: #f8f8f8; color: #bb4444;">builder</span><span style="background-color: #f8f8f8; color: #666666;">()</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #666666;">.</span><span style="background-color: #f8f8f8; color: #bb4444;">queryString</span><span style="background-color: #f8f8f8; color: #666666;">(</span><span style="background-color: #f8f8f8; color: #bb4444;">"SELECT * from users_partitioned LIMIT 5;"</span><span style="background-color: #f8f8f8; color: #666666;">)</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #666666;">.</span><span style="background-color: #f8f8f8; color: #bb4444;">queryExecutionContext</span><span style="background-color: #f8f8f8; color: #666666;">(</span><span style="background-color: #f8f8f8;">queryExecutionContext</span><span style="background-color: #f8f8f8; color: #666666;">)</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #666666;">.</span><span style="background-color: #f8f8f8; color: #bb4444;">resultConfiguration</span><span style="background-color: #f8f8f8; color: #666666;">(</span><span style="background-color: #f8f8f8;">resultConfiguration</span><span style="background-color: #f8f8f8; color: #666666;">)</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #666666;">.</span><span style="background-color: #f8f8f8; color: #bb4444;">build</span><span style="background-color: #f8f8f8; color: #666666;">();</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #008800; font-style: italic;">//Start executing the query</span><span style="background-color: #f8f8f8;">
StartQueryExecutionResponse startQueryExecutionResponse </span><span style="background-color: #f8f8f8; color: #666666;">=</span><span style="background-color: #f8f8f8;"> athenaClient</span><span style="background-color: #f8f8f8; color: #666666;">.</span><span style="background-color: #f8f8f8; color: #bb4444;">startQueryExecution</span><span style="background-color: #f8f8f8; color: #666666;">(</span><span style="background-color: #f8f8f8;">startQueryExecutionRequest</span><span style="background-color: #f8f8f8; color: #666666;">);</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #aa22ff; font-weight: bold;">return</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: yellow;">startQueryExecutionResponse<span style="color: #666666;">.</span><span style="color: #bb4444;">queryExecutionId</span><span style="color: #666666;">();</span></span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #666666;">}</span><span style="background-color: #f8f8f8;">
</span></pre>
</div>
The first step was to setup the QueryExecutionContext which includes the name of the database I setup perviously.<br />
Next was creating the ResultConfiguration which indicates where the output of the query is to be stored in S3. Athena wrote the query result as csv files to that location.<br />
The last step was to create StartQueryExecutionRequest instance and execute the query. AthenaClient returns a query Execution id that can be used to track the query execution.<!-- HTML generated using hilite.me --><br />
<br />
Now we <b>wait for the query to complete execution.</b><br />
<div style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="background-color: #f8f8f8; line-height: 125%; margin: 0px;"> <span style="color: #aa22ff; font-weight: bold;">private</span> <span style="color: #aa22ff; font-weight: bold;">static</span> <span style="color: #00bb00; font-weight: bold;">void</span> <span style="color: #00a000;">completeQueryExecution</span><span style="color: #666666;">(</span>String queryExecutionId<span style="color: #666666;">)</span> </pre>
<pre style="line-height: 125%; margin: 0px;"><span style="background-color: #f8f8f8; color: #aa22ff; font-weight: bold;"> throws</span><span style="background-color: #f8f8f8;"> InterruptedException </span><span style="background-color: #f8f8f8; color: #666666;">{</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: yellow;">GetQueryExecutionRequest</span><span style="background-color: #f8f8f8;"> getQueryExecutionRequest </span><span style="background-color: #f8f8f8; color: #666666;">=</span><span style="background-color: #f8f8f8;"> </span></pre>
<pre style="line-height: 125%; margin: 0px;"><span style="background-color: #f8f8f8;"> GetQueryExecutionRequest<span style="color: #666666;">.</span><span style="color: #bb4444;">builder</span><span style="color: #666666;">()</span>
<span style="color: #666666;">.</span><span style="color: #bb4444;">queryExecutionId</span><span style="color: #666666;">(</span>queryExecutionId<span style="color: #666666;">).</span><span style="color: #bb4444;">build</span><span style="color: #666666;">();</span>
</span><span style="background-color: yellow;">GetQueryExecutionResponse</span><span style="background-color: #f8f8f8;"> getQueryExecutionResponse</span><span style="background-color: #f8f8f8; color: #666666;">;</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #00bb00; font-weight: bold;">boolean</span><span style="background-color: #f8f8f8;"> isQueryStillRunning </span><span style="background-color: #f8f8f8; color: #666666;">=</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #aa22ff; font-weight: bold;">true</span><span style="background-color: #f8f8f8; color: #666666;">;</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #00bb00; font-weight: bold;">int</span><span style="background-color: #f8f8f8;"> i </span><span style="background-color: #f8f8f8; color: #666666;">=</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #666666;">0;</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #aa22ff; font-weight: bold;">while</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #666666;">(</span><span style="background-color: #f8f8f8;">isQueryStillRunning</span><span style="background-color: #f8f8f8; color: #666666;">)</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #666666;">{</span><span style="background-color: #f8f8f8;">
System</span><span style="background-color: #f8f8f8; color: #666666;">.</span><span style="background-color: #f8f8f8; color: #bb4444;">out</span><span style="background-color: #f8f8f8; color: #666666;">.</span><span style="background-color: #f8f8f8; color: #bb4444;">println</span><span style="background-color: #f8f8f8; color: #666666;">(</span><span style="background-color: #f8f8f8; color: #bb4444;">"Checking if query is running. Check No "</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #666666;">+</span><span style="background-color: #f8f8f8;"> i</span><span style="background-color: #f8f8f8; color: #666666;">);</span><span style="background-color: #f8f8f8;">
getQueryExecutionResponse </span><span style="background-color: #f8f8f8; color: #666666;">=</span><span style="background-color: #f8f8f8;"> athenaClient</span><span style="background-color: #f8f8f8; color: #666666;">.</span><span style="background-color: #f8f8f8; color: #bb4444;">getQueryExecution</span><span style="background-color: #f8f8f8; color: #666666;">(</span></pre>
<pre style="line-height: 125%; margin: 0px;"><span style="background-color: #f8f8f8;"> getQueryExecutionRequest<span style="color: #666666;">);</span>
String queryState <span style="color: #666666;">=</span> </span><span style="background-color: yellow;">getQueryExecutionResponse<span style="color: #666666;">.</span><span style="color: #bb4444;">queryExecution</span><span style="color: #666666;">()</span></span></pre>
<pre style="line-height: 125%; margin: 0px;"><span style="color: #666666;"><span style="background-color: white;"> </span><span style="background-color: yellow;">.</span></span><span style="background-color: yellow; color: #bb4444;">status</span><span style="background-color: yellow; color: #666666;">().</span><span style="background-color: yellow; color: #bb4444;">state</span><span style="background-color: yellow; color: #666666;">().</span><span style="background-color: yellow; color: #bb4444;">toString</span><span style="color: #666666;"><span style="background-color: yellow;">()</span><span style="background-color: #f8f8f8;">;</span></span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #aa22ff; font-weight: bold;">if</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #666666;">(</span><span style="background-color: #f8f8f8;">queryState</span><span style="background-color: #f8f8f8; color: #666666;">.</span><span style="background-color: #f8f8f8; color: #bb4444;">equals</span><span style="background-color: #f8f8f8; color: #666666;">(</span><span style="background-color: #f8f8f8;">QueryExecutionState</span><span style="background-color: #f8f8f8; color: #666666;">.</span><span style="background-color: yellow; color: #bb4444;">FAILED</span><span style="background-color: #f8f8f8; color: #666666;">.</span><span style="background-color: #f8f8f8; color: #bb4444;">toString</span><span style="background-color: #f8f8f8; color: #666666;">()))</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #666666;">{</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #aa22ff; font-weight: bold;">throw</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #aa22ff; font-weight: bold;">new</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #00a000;">RuntimeException</span><span style="background-color: #f8f8f8; color: #666666;">(</span><span style="background-color: #f8f8f8; color: #bb4444;">"Query Failed to run with Error " + </span></pre>
<pre style="line-height: 125%; margin: 0px;"><span style="background-color: #f8f8f8; color: #bb4444;"> "Message: "</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #666666;">+</span><span style="background-color: #f8f8f8;"> getQueryExecutionResponse
</span><span style="background-color: #f8f8f8; color: #666666;">.</span><span style="background-color: #f8f8f8; color: #bb4444;">queryExecution</span><span style="background-color: #f8f8f8; color: #666666;">().</span><span style="background-color: #f8f8f8; color: #bb4444;">status</span><span style="background-color: #f8f8f8; color: #666666;">().</span><span style="background-color: #f8f8f8; color: #bb4444;">stateChangeReason</span><span style="background-color: #f8f8f8; color: #666666;">());</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #666666;">}</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #aa22ff; font-weight: bold;">else</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #aa22ff; font-weight: bold;">if</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #666666;">(</span><span style="background-color: #f8f8f8;">queryState</span><span style="background-color: #f8f8f8; color: #666666;">.</span><span style="background-color: #f8f8f8; color: #bb4444;">equals</span><span style="background-color: #f8f8f8; color: #666666;">(</span><span style="background-color: #f8f8f8;">QueryExecutionState</span><span style="background-color: #f8f8f8; color: #666666;">.</span><span style="background-color: yellow; color: #bb4444;">CANCELLED</span><span style="background-color: #f8f8f8; color: #666666;">.</span><span style="background-color: #f8f8f8; color: #bb4444;">toString</span><span style="background-color: #f8f8f8; color: #666666;">()))</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #666666;">{</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #aa22ff; font-weight: bold;">throw</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #aa22ff; font-weight: bold;">new</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #00a000;">RuntimeException</span><span style="background-color: #f8f8f8; color: #666666;">(</span><span style="background-color: #f8f8f8; color: #bb4444;">"Query was cancelled."</span><span style="background-color: #f8f8f8; color: #666666;">);</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #666666;">}</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #aa22ff; font-weight: bold;">else</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #aa22ff; font-weight: bold;">if</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #666666;">(</span><span style="background-color: #f8f8f8;">queryState</span><span style="background-color: #f8f8f8; color: #666666;">.</span><span style="background-color: #f8f8f8; color: #bb4444;">equals</span><span style="background-color: #f8f8f8; color: #666666;">(</span><span style="background-color: #f8f8f8;">QueryExecutionState</span><span style="background-color: #f8f8f8; color: #666666;">.</span><span style="background-color: yellow; color: #bb4444;">SUCCEEDED</span><span style="background-color: #f8f8f8; color: #666666;">.</span><span style="background-color: #f8f8f8; color: #bb4444;">toString</span><span style="background-color: #f8f8f8; color: #666666;">()))</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #666666;">{</span><span style="background-color: #f8f8f8;">
isQueryStillRunning </span><span style="background-color: #f8f8f8; color: #666666;">=</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #aa22ff; font-weight: bold;">false</span><span style="background-color: #f8f8f8; color: #666666;">;</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #666666;">}</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #aa22ff; font-weight: bold;">else</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #666666;">{</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #008800; font-style: italic;">// Sleep an amount of time before retrying again.</span><span style="background-color: #f8f8f8;">
Thread</span><span style="background-color: #f8f8f8; color: #666666;">.</span><span style="background-color: #f8f8f8; color: #bb4444;">sleep</span><span style="background-color: #f8f8f8; color: #666666;">(1000);</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #666666;">}</span><span style="background-color: #f8f8f8;">
i</span><span style="background-color: #f8f8f8; color: #666666;">++;</span><span style="background-color: #f8f8f8;">
System</span><span style="background-color: #f8f8f8; color: #666666;">.</span><span style="background-color: #f8f8f8; color: #bb4444;">out</span><span style="background-color: #f8f8f8; color: #666666;">.</span><span style="background-color: #f8f8f8; color: #bb4444;">println</span><span style="background-color: #f8f8f8; color: #666666;">(</span><span style="background-color: #f8f8f8; color: #bb4444;">"Current Status is: "</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #666666;">+</span><span style="background-color: #f8f8f8;"> queryState</span><span style="background-color: #f8f8f8; color: #666666;">);</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #666666;">}</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #666666;">}</span><span style="background-color: #f8f8f8;">
</span></pre>
</div>
The code uses the query execution id returned in the StartQueryExecutionResponse to check the status of query execution. The code simply sleeps until it receives one of FAILED, CANCELLED or SUCCEEDED status.<br />
The last step is to <b>process the response from query execution</b>:<!-- HTML generated using hilite.me --><br />
<div style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="line-height: 125%; margin: 0px;"><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #aa22ff; font-weight: bold;">private</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #aa22ff; font-weight: bold;">static</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #00bb00; font-weight: bold;">void</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #00a000;">processQueryResult</span><span style="background-color: #f8f8f8; color: #666666;">(</span><span style="background-color: #f8f8f8;">String queryExecutionId</span><span style="background-color: #f8f8f8; color: #666666;">)</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #666666;">{</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: yellow;">GetQueryResultsRequest</span><span style="background-color: #f8f8f8;"> getQueryResultsRequest </span><span style="background-color: #f8f8f8; color: #666666;">=</span><span style="background-color: #f8f8f8;"> GetQueryResultsRequest</span><span style="background-color: #f8f8f8; color: #666666;">.</span><span style="background-color: #f8f8f8; color: #bb4444;">builder</span><span style="background-color: #f8f8f8; color: #666666;">()</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #008800; font-style: italic;">// Max Results can be set but if its not set,</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #008800; font-style: italic;">// it will choose the maximum page size</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #008800; font-style: italic;">// As of the writing of this code, the maximum value is 1000</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #008800; font-style: italic;">// .withMaxResults(1000)</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #666666;">.</span><span style="background-color: #f8f8f8; color: #bb4444;">queryExecutionId</span><span style="background-color: #f8f8f8; color: #666666;">(</span><span style="background-color: #f8f8f8;">queryExecutionId</span><span style="background-color: #f8f8f8; color: #666666;">).</span><span style="background-color: #f8f8f8; color: #bb4444;">build</span><span style="background-color: #f8f8f8; color: #666666;">();</span><span style="background-color: #f8f8f8;">
GetQueryResultsIterable getQueryResultsResults </span><span style="background-color: #f8f8f8; color: #666666;">=</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: yellow;">athenaClient
</span><span style="background-color: white;"> </span><span style="background-color: yellow; color: #666666;">.</span><span style="background-color: yellow; color: #bb4444;">getQueryResultsPaginator</span><span style="background-color: yellow; color: #666666;">(</span><span style="background-color: yellow;">getQueryResultsRequest</span><span style="background-color: yellow; color: #666666;">);</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #aa22ff; font-weight: bold;">for</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #666666;">(</span><span style="background-color: #f8f8f8;">GetQueryResultsResponse result </span><span style="background-color: #f8f8f8; color: #666666;">:</span><span style="background-color: #f8f8f8;"> getQueryResultsResults</span><span style="background-color: #f8f8f8; color: #666666;">)</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #666666;">{</span><span style="background-color: #f8f8f8;">
List</span><span style="background-color: #f8f8f8; color: #666666;"><</span><span style="background-color: #f8f8f8;">ColumnInfo</span><span style="background-color: #f8f8f8; color: #666666;">></span><span style="background-color: #f8f8f8;"> columnInfoList </span><span style="background-color: #f8f8f8; color: #666666;">=</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: yellow;">result<span style="color: #666666;">.</span><span style="color: #bb4444;">resultSet</span><span style="color: #666666;">()</span></span></pre>
<pre style="line-height: 125%; margin: 0px;"><span style="background-color: #f8f8f8; color: #666666;"> </span><span style="background-color: yellow;"><span style="color: #666666;">.</span><span style="color: #bb4444;">resultSetMetadata</span><span style="color: #666666;">().</span><span style="color: #bb4444;">columnInfo</span><span style="color: #666666;">();</span></span><span style="background-color: #f8f8f8;">
System</span><span style="background-color: #f8f8f8; color: #666666;">.</span><span style="background-color: #f8f8f8; color: #bb4444;">out</span><span style="background-color: #f8f8f8; color: #666666;">.</span><span style="background-color: #f8f8f8; color: #bb4444;">println</span><span style="background-color: #f8f8f8; color: #666666;">(</span><span style="background-color: #f8f8f8; color: #bb4444;">"ColumnInfo received is "</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #666666;">+</span><span style="background-color: #f8f8f8;"> columnInfoList</span><span style="background-color: #f8f8f8; color: #666666;">);</span><span style="background-color: #f8f8f8;">
List</span><span style="background-color: #f8f8f8; color: #666666;"><</span><span style="background-color: #f8f8f8;">Row</span><span style="background-color: #f8f8f8; color: #666666;">></span><span style="background-color: #f8f8f8;"> results </span><span style="background-color: #f8f8f8; color: #666666;">=</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: yellow;">result<span style="color: #666666;">.</span><span style="color: #bb4444;">resultSet</span><span style="color: #666666;">().</span><span style="color: #bb4444;">rows</span><span style="color: #666666;">()</span></span><span style="background-color: #f8f8f8; color: #666666;">;</span><span style="background-color: #f8f8f8;">
processRow</span><span style="background-color: #f8f8f8; color: #666666;">(</span><span style="background-color: #f8f8f8;">results</span><span style="background-color: #f8f8f8; color: #666666;">);</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #666666;">}</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #666666;">}</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #aa22ff; font-weight: bold;">private</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #aa22ff; font-weight: bold;">static</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #00bb00; font-weight: bold;">void</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #00a000;">processRow</span><span style="background-color: #f8f8f8; color: #666666;">(</span><span style="background-color: #f8f8f8;">List</span><span style="background-color: #f8f8f8; color: #666666;"><</span><span style="background-color: #f8f8f8;">Row</span><span style="background-color: #f8f8f8; color: #666666;">></span><span style="background-color: #f8f8f8;"> row</span><span style="background-color: #f8f8f8; color: #666666;">)</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #666666;">{</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #008800; font-style: italic;">//Write out the data</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #aa22ff; font-weight: bold;">for</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #666666;">(</span><span style="background-color: #f8f8f8;">Row myRow </span><span style="background-color: #f8f8f8; color: #666666;">:</span><span style="background-color: #f8f8f8;"> row</span><span style="background-color: #f8f8f8; color: #666666;">)</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #666666;">{</span><span style="background-color: #f8f8f8;">
List</span><span style="background-color: #f8f8f8; color: #666666;"><</span><span style="background-color: #f8f8f8;">Datum</span><span style="background-color: #f8f8f8; color: #666666;">></span><span style="background-color: #f8f8f8;"> allData </span><span style="background-color: #f8f8f8; color: #666666;">=</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: yellow;">myRow<span style="color: #666666;">.</span><span style="color: #bb4444;">data</span><span style="color: #666666;">()</span></span><span style="background-color: #f8f8f8; color: #666666;">;</span><span style="background-color: #f8f8f8;">
System</span><span style="background-color: #f8f8f8; color: #666666;">.</span><span style="background-color: #f8f8f8; color: #bb4444;">out</span><span style="background-color: #f8f8f8; color: #666666;">.</span><span style="background-color: #f8f8f8; color: #bb4444;">println</span><span style="background-color: #f8f8f8; color: #666666;">();</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #aa22ff; font-weight: bold;">for</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #666666;">(</span><span style="background-color: #f8f8f8;">Datum data </span><span style="background-color: #f8f8f8; color: #666666;">:</span><span style="background-color: #f8f8f8;"> allData</span><span style="background-color: #f8f8f8; color: #666666;">)</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #666666;">{</span><span style="background-color: #f8f8f8;">
System</span><span style="background-color: #f8f8f8; color: #666666;">.</span><span style="background-color: #f8f8f8; color: #bb4444;">out</span><span style="background-color: #f8f8f8; color: #666666;">.</span><span style="background-color: #f8f8f8; color: #bb4444;">print</span><span style="background-color: #f8f8f8; color: #666666;">(</span><span style="background-color: #f8f8f8; color: #bb4444;">" "</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #666666;">+</span><span style="background-color: #f8f8f8;"> data</span><span style="background-color: #f8f8f8; color: #666666;">.</span><span style="background-color: #f8f8f8; color: #bb4444;">varCharValue</span><span style="background-color: #f8f8f8; color: #666666;">());</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #666666;">}</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #666666;">}</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #666666;">}</span><span style="background-color: #f8f8f8;">
</span></pre>
</div>
The code create an instance of GetQueryResultsRequest with the queryExecutionId from the StartQueryExecutionResponse. The response is <span style="background-color: #f8f8f8;">GetQueryResultsIterable</span>
<br />
<div style="background: #f8f8f8; border-width: 0.1em 0.1em 0.1em 0.8em; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="line-height: 125%; margin: 0;">This class is an iterable of GetQueryResultsResponse that can be used to iterate
through all the response pages of the operation.
When the operation is called, an instance of this class is returned. At this
point, no service calls are made yet and so there is no guarantee that the
request is valid. As you iterate through the iterable, SDK will start lazily
loading response pages by making service calls until there are no pages left or
your iteration stops. If there are errors in your request, you will see the failures
only after you start iterating through the iterable.
</pre>
</div>
The result of the query is placed by Athena in the S3 folder. Each page of the resultSet includes a list of <span style="background-color: #f8f8f8;">ColumnInfo and a list of Row.</span><br />
<span style="background-color: #f8f8f8;">ColumnInfo has </span><span style="background-color: #f8f8f8;">Information about the columns in a query execution result, while </span><span style="background-color: #f8f8f8;">Row represents each record from the result.</span><br />
<span style="background-color: #f8f8f8;"><br /></span>
<span style="background-color: #f8f8f8;">I executed my code and the output is as below:</span>
<!-- HTML generated using hilite.me --><br />
<div style="background: #f8f8f8; border-width: 0.1em 0.1em 0.1em 0.8em; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="line-height: 125%; margin: 0;">Checking if query is running. Check No 0
Current Status is: QUEUED
Checking if query is running. Check No 1
Current Status is: RUNNING
Checking if query is running. Check No 2
Current Status is: RUNNING
Checking if query is running. Check No 3
Current Status is: SUCCEEDED
ColumnInfo received is [</pre>
<pre style="line-height: 125%; margin: 0;"><span style="color: blue;">ColumnInfo</span>(CatalogName=hive, SchemaName=, TableName=, </pre>
<pre style="line-height: 125%; margin: 0;"><span style="color: blue;">Name=user_id, Label=user_id, Type=integer</span>, Precision=10, Scale=0, </pre>
<pre style="line-height: 125%; margin: 0;">Nullable=UNKNOWN, CaseSensitive=false), </pre>
<pre style="line-height: 125%; margin: 0;"><span style="color: blue;">ColumnInfo</span>(CatalogName=hive, SchemaName=, TableName=, <span style="color: blue;">Name=name, Label=name, </span></pre>
<pre style="line-height: 125%; margin: 0;"><span style="color: blue;">Type=varchar,</span> Precision=2147483647, Scale=0, Nullable=UNKNOWN, CaseSensitive=true),</pre>
<pre style="line-height: 125%; margin: 0;"><span style="color: blue;">ColumnInfo</span>(CatalogName=hive, SchemaName=, TableName=, <span style="color: blue;">Name=phone_no, </span></pre>
<pre style="line-height: 125%; margin: 0;"><span style="color: blue;">Label=phone_no, Type=varchar,</span> Precision=2147483647, Scale=0, Nullable=UNKNOWN, </pre>
<pre style="line-height: 125%; margin: 0;">CaseSensitive=true), </pre>
<pre style="line-height: 125%; margin: 0;"><span style="color: blue;">ColumnInfo</span>(CatalogName=hive, SchemaName=, TableName=, </pre>
<pre style="line-height: 125%; margin: 0;"><span style="color: blue;">Name=age, Label=age, Type=integer,</span> Precision=10, Scale=0, Nullable=UNKNOWN, </pre>
<pre style="line-height: 125%; margin: 0;">CaseSensitive=false), </pre>
<pre style="line-height: 125%; margin: 0;"><span style="color: blue;">ColumnInfo</span>(CatalogName=hive, SchemaName=, TableName=, <span style="color: blue;">Name=hobbies, Label=hobbies,</span></pre>
<pre style="line-height: 125%; margin: 0;"><span style="color: blue;"> Type=array</span>, Precision=0, Scale=0, Nullable=UNKNOWN, CaseSensitive=false), </pre>
<pre style="line-height: 125%; margin: 0;"><span style="color: blue;">ColumnInfo</span>(CatalogName=hive, SchemaName=, TableName=, <span style="color: blue;">Name=state, Label=state, </span></pre>
<pre style="line-height: 125%; margin: 0;"><span style="color: blue;">Type=varchar</span>, Precision=2147483647, Scale=0, Nullable=UNKNOWN, CaseSensitive=true),</pre>
<pre style="line-height: 125%; margin: 0;"><span style="color: blue;">ColumnInfo</span>(CatalogName=hive, SchemaName=, TableName=, N<span style="color: blue;">ame=country, Label=country,</span></pre>
<pre style="line-height: 125%; margin: 0;"><span style="color: blue;">Type=varchar</span>, Precision=2147483647, Scale=0, Nullable=UNKNOWN, CaseSensitive=true),</pre>
<pre style="line-height: 125%; margin: 0;"><span style="color: blue;">ColumnInfo</span>(CatalogName=hive, SchemaName=, TableName=, <span style="color: blue;">Name=dataset_date, </span></pre>
<pre style="line-height: 125%; margin: 0;"><span style="color: blue;">Label=dataset_date, Type=varchar,</span> Precision=2147483647, Scale=0, Nullable=UNKNOWN,</pre>
<pre style="line-height: 125%; margin: 0;">CaseSensitive=true)]
user_id name phone_no age hobbies state country dataset_date
4358786 UserName_4358786 phNo4358786 48 [] OR US 2020-05-28
4358787 UserName_4358787 phNo4358787 54 [] TE US 2020-05-28
4358788 UserName_4358788 phNo4358788 27 [piano, singing, painting, video games] CA US 2020-05-28
4358789 UserName_4358789 phNo4358789 34 [] OK US 2020-05-28
4358790 UserName_4358790 phNo4358790 25 [singing] NM US 2020-05-28
Process finished with exit code 0
</pre>
</div>
The S3 bucket had two files - one a csv with the 5 rows, the other a metadata file which had information about the columns
</div>
Robin Varghesehttp://www.blogger.com/profile/00841917517097216645noreply@blogger.com1tag:blogger.com,1999:blog-6262128735818493674.post-55539959930815296312020-06-04T07:21:00.001+05:302021-06-13T00:31:56.320+05:30Dynamo Db's API Model<div dir="ltr" style="text-align: left;" trbidi="on">
While doing a search for Dynamo Db get Item code, I came across several different code samples (obviously) and also realized a not so obvious thing - there are multiple ways to access Dynamo Db through code.<br />
<a name='more'></a>No longer confident on what is the best way, I decided to check the java docs. Turns out, there are (as of today) <b>3 ways</b> to perform CRUD operations on a Dynamo Table. This <a href="https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Programming.SDKOverview.html" target="_blank">link</a> gives a very nice description of the same. I decided to mess around with the available options. As a prerequisite, I setup a simple users table with a hash key.<br />
<div>
<br />
<div>
<b>Technique 1: </b><b>Use low-level interface</b></div>
The low-level interface for Amazon DynamoDB most closely resemble the POST requests used for Dynamo operations. The below code will convert into a POST request and create an Item in the table.<br />
<div style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="line-height: 125%; margin: 0px;"><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #aa22ff; font-weight: bold;">private</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #aa22ff; font-weight: bold;">static</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #00bb00; font-weight: bold;">void</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #00a000;">createItemUsingLowLevelApi</span><span style="background-color: #f8f8f8; color: #666666;">(</span><span style="background-color: #f8f8f8;">List</span><span style="background-color: #f8f8f8; color: #666666;"><</span><span style="background-color: #f8f8f8;">String</span><span style="background-color: #f8f8f8; color: #666666;">></span><span style="background-color: #f8f8f8;"> hobbies</span><span style="background-color: #f8f8f8; color: #666666;">,</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #00bb00; font-weight: bold;">int</span><span style="background-color: #f8f8f8;"> suffix</span><span style="background-color: #f8f8f8; color: #666666;">,</span><span style="background-color: #f8f8f8;">
AWSCredentialsProvider awsCredentialsProvider</span><span style="background-color: #f8f8f8; color: #666666;">)</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #666666;">{</span><span style="background-color: #f8f8f8;">
AmazonDynamoDB client </span><span style="background-color: #f8f8f8; color: #666666;">=</span><span style="background-color: #f8f8f8;"> AmazonDynamoDBClientBuilder</span><span style="background-color: #f8f8f8; color: #666666;">.</span><span style="background-color: #f8f8f8; color: #bb4444;">standard</span><span style="background-color: #f8f8f8; color: #666666;">()</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #666666;">.</span><span style="background-color: #f8f8f8; color: #bb4444;">withRegion</span><span style="background-color: #f8f8f8; color: #666666;">(</span><span style="background-color: #f8f8f8; color: #bb4444;">"us-east-1"</span><span style="background-color: #f8f8f8; color: #666666;">)</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #666666;">.</span><span style="background-color: #f8f8f8; color: #bb4444;">withCredentials</span><span style="background-color: #f8f8f8; color: #666666;">(</span><span style="background-color: #f8f8f8;">awsCredentialsProvider</span><span style="background-color: #f8f8f8; color: #666666;">).</span><span style="background-color: #f8f8f8; color: #bb4444;">build</span><span style="background-color: #f8f8f8; color: #666666;">();</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: yellow;">PutItemRequest</span><span style="background-color: #f8f8f8;"> request </span><span style="background-color: #f8f8f8; color: #666666;">=</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #aa22ff; font-weight: bold;">new</span><span style="background-color: #f8f8f8;"> PutItemRequest</span><span style="background-color: #f8f8f8; color: #666666;">().</span><span style="background-color: #f8f8f8; color: #bb4444;">withTableName</span><span style="background-color: #f8f8f8; color: #666666;">(</span><span style="background-color: #f8f8f8; color: #bb4444;">"Users"</span><span style="background-color: #f8f8f8; color: #666666;">)</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #666666;">.</span><span style="background-color: #f8f8f8; color: #bb4444;">withReturnConsumedCapacity</span><span style="background-color: #f8f8f8; color: #666666;">(</span><span style="background-color: #f8f8f8; color: #bb4444;">"TOTAL"</span><span style="background-color: #f8f8f8; color: #666666;">);</span><span style="background-color: #f8f8f8;">
request</span><span style="background-color: #f8f8f8; color: #666666;">.</span><span style="background-color: #f8f8f8; color: #bb4444;">addItemEntry</span><span style="background-color: #f8f8f8; color: #666666;">(</span><span style="background-color: #f8f8f8; color: #bb4444;">"userId"</span><span style="background-color: #f8f8f8; color: #666666;">,</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #aa22ff; font-weight: bold;">new</span><span style="background-color: #f8f8f8;"> AttributeValue</span><span style="background-color: #f8f8f8; color: #666666;">(</span><span style="background-color: #f8f8f8; color: #bb4444;">"userId_"</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #666666;">+</span><span style="background-color: #f8f8f8;"> suffix</span><span style="background-color: #f8f8f8; color: #666666;">))</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #666666;">.</span><span style="background-color: #f8f8f8; color: #bb4444;">addItemEntry</span><span style="background-color: #f8f8f8; color: #666666;">(</span><span style="background-color: #f8f8f8; color: #bb4444;">"name"</span><span style="background-color: #f8f8f8; color: #666666;">,</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #aa22ff; font-weight: bold;">new</span><span style="background-color: #f8f8f8;"> AttributeValue</span><span style="background-color: #f8f8f8; color: #666666;">(</span><span style="background-color: #f8f8f8; color: #bb4444;">"User_"</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #666666;">+</span><span style="background-color: #f8f8f8;"> suffix</span><span style="background-color: #f8f8f8; color: #666666;">))</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #666666;">.</span><span style="background-color: #f8f8f8; color: #bb4444;">addItemEntry</span><span style="background-color: #f8f8f8; color: #666666;">(</span><span style="background-color: #f8f8f8; color: #bb4444;">"age"</span><span style="background-color: #f8f8f8; color: #666666;">,</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #aa22ff; font-weight: bold;">new</span><span style="background-color: #f8f8f8;"> AttributeValue</span><span style="background-color: #f8f8f8; color: #666666;">().</span><span style="background-color: #f8f8f8; color: #bb4444;">withN</span><span style="background-color: #f8f8f8; color: #666666;">(</span><span style="background-color: #f8f8f8; color: #bb4444;">"25"</span><span style="background-color: #f8f8f8; color: #666666;">))</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #666666;">.</span><span style="background-color: #f8f8f8; color: #bb4444;">addItemEntry</span><span style="background-color: #f8f8f8; color: #666666;">(</span><span style="background-color: #f8f8f8; color: #bb4444;">"hobbies"</span><span style="background-color: #f8f8f8; color: #666666;">,</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #aa22ff; font-weight: bold;">new</span><span style="background-color: #f8f8f8;"> AttributeValue</span><span style="background-color: #f8f8f8; color: #666666;">(</span><span style="background-color: #f8f8f8;">hobbies</span><span style="background-color: #f8f8f8; color: #666666;">));</span><span style="background-color: #f8f8f8;">
request</span><span style="background-color: #f8f8f8; color: #666666;">.</span><span style="background-color: #f8f8f8; color: #bb4444;">setConditionExpression</span><span style="background-color: #f8f8f8; color: #666666;">(</span><span style="background-color: #f8f8f8; color: #bb4444;">"attribute_not_exists(userId)"</span><span style="background-color: #f8f8f8; color: #666666;">);</span><span style="background-color: #f8f8f8;">
request</span><span style="background-color: #f8f8f8; color: #666666;">.</span><span style="background-color: #f8f8f8; color: #bb4444;">setReturnValues</span><span style="background-color: #f8f8f8; color: #666666;">(</span><span style="background-color: #f8f8f8;">ReturnValue</span><span style="background-color: #f8f8f8; color: #666666;">.</span><span style="background-color: #f8f8f8; color: #bb4444;">ALL_OLD</span><span style="background-color: #f8f8f8; color: #666666;">);</span><span style="background-color: #f8f8f8;">
request</span><span style="background-color: #f8f8f8; color: #666666;">.</span><span style="background-color: #f8f8f8; color: #bb4444;">withReturnItemCollectionMetrics</span><span style="background-color: #f8f8f8; color: #666666;">(</span></pre>
<pre style="line-height: 125%; margin: 0px;"><span style="background-color: #f8f8f8;"> ReturnItemCollectionMetrics</span><span style="background-color: #f8f8f8; color: #666666;">.</span><span style="background-color: #f8f8f8; color: #bb4444;">SIZE</span><span style="background-color: #f8f8f8; color: #666666;">);</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: yellow;">PutItemResult response <span style="color: #666666;">=</span> client<span style="color: #666666;">.</span><span style="color: #bb4444;">putItem</span><span style="color: #666666;">(</span>request<span style="color: #666666;">);</span></span><span style="background-color: #f8f8f8;">
System</span><span style="background-color: #f8f8f8; color: #666666;">.</span><span style="background-color: #f8f8f8; color: #bb4444;">out</span><span style="background-color: #f8f8f8; color: #666666;">.</span><span style="background-color: #f8f8f8; color: #bb4444;">println</span><span style="background-color: #f8f8f8; color: #666666;">(</span><span style="background-color: #f8f8f8;">response</span><span style="background-color: #f8f8f8; color: #666666;">);</span><span style="background-color: #f8f8f8;">
</span></pre>
</div>
<b>AmazonDynamoDB</b> class implements the DynamoDB low-level interface. The userId is the primary hash key here. The Interface assumes the data type is string unless explicitly specified.<br />
The response for the code is:
<br />
<div style="background: rgb(248, 248, 248); border-width: 0.1em 0.1em 0.1em 0.8em; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="line-height: 125%; margin: 0px;">{ConsumedCapacity: {TableName: Users,CapacityUnits: 1.0,},}</pre>
</div>
<div>
PutItem will override the entry if one exists in the table with same Range key. That is the reason the API returns the OLD value.<br />
If we do not want this behavior than we can add the condition expression that inserts record only if one with userId does not exists.</div><div><br />
<b>Technique 2: Use Document Interfaces </b></div>
<!--HTML generated using hilite.me--><br />
<div style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="line-height: 125%; margin: 0px;"><span style="background-color: #f8f8f8; color: #aa22ff; font-weight: bold;">private</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #aa22ff; font-weight: bold;">static</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #00bb00; font-weight: bold;">void</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #00a000;">createItemUsingDocumentApi</span><span style="background-color: #f8f8f8; color: #666666;">(</span><span style="background-color: #f8f8f8;">List</span><span style="background-color: #f8f8f8; color: #666666;"><</span><span style="background-color: #f8f8f8;">String</span><span style="background-color: #f8f8f8; color: #666666;">></span><span style="background-color: #f8f8f8;"> hobbies</span><span style="background-color: #f8f8f8; color: #666666;">,</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #00bb00; font-weight: bold;">int</span><span style="background-color: #f8f8f8;"> suffix</span><span style="background-color: #f8f8f8; color: #666666;">,</span><span style="background-color: #f8f8f8;">
AWSCredentialsProvider awsCredentialsProvider</span><span style="background-color: #f8f8f8; color: #666666;">)</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #666666;">{</span><span style="background-color: #f8f8f8;">
AmazonDynamoDB client </span><span style="background-color: #f8f8f8; color: #666666;">=</span><span style="background-color: #f8f8f8;"> AmazonDynamoDBClientBuilder</span><span style="background-color: #f8f8f8; color: #666666;">.</span><span style="background-color: #f8f8f8; color: #bb4444;">standard</span><span style="background-color: #f8f8f8; color: #666666;">()</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #666666;">.</span><span style="background-color: #f8f8f8; color: #bb4444;">withRegion</span><span style="background-color: #f8f8f8; color: #666666;">(</span><span style="background-color: #f8f8f8; color: #bb4444;">"us-east-1"</span><span style="background-color: #f8f8f8; color: #666666;">)</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #666666;">.</span><span style="background-color: #f8f8f8; color: #bb4444;">withCredentials</span><span style="background-color: #f8f8f8; color: #666666;">(</span><span style="background-color: #f8f8f8;">awsCredentialsProvider</span><span style="background-color: #f8f8f8; color: #666666;">).</span><span style="background-color: #f8f8f8; color: #bb4444;">build</span><span style="background-color: #f8f8f8; color: #666666;">();</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: yellow;">DynamoDB docClient <span style="color: #666666;">=</span> <span style="color: #aa22ff; font-weight: bold;">new</span> DynamoDB<span style="color: #666666;">(</span>client<span style="color: #666666;">);</span>
</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: yellow;">Table table <span style="color: #666666;">=</span> docClient<span style="color: #666666;">.</span><span style="color: #bb4444;">getTable</span><span style="color: #666666;">(</span><span style="color: #bb4444;">"Users"</span><span style="color: #666666;">);</span></span><span style="background-color: #f8f8f8;">
</span><span style="background-color: yellow;">Item item <span style="color: #666666;">=</span> <span style="color: #aa22ff; font-weight: bold;">new</span> Item<span style="color: #666666;">()</span></span><span style="background-color: #f8f8f8; color: #666666;">.</span><span style="background-color: #f8f8f8; color: #bb4444;">withPrimaryKey</span><span style="background-color: #f8f8f8; color: #666666;">(</span><span style="background-color: #f8f8f8; color: #bb4444;">"userId"</span><span style="background-color: #f8f8f8; color: #666666;">,</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #bb4444;">"userId_"</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #666666;">+</span><span style="background-color: #f8f8f8;"> suffix</span><span style="background-color: #f8f8f8; color: #666666;">)</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #666666;">.</span><span style="background-color: #f8f8f8; color: #bb4444;">withString</span><span style="background-color: #f8f8f8; color: #666666;">(</span><span style="background-color: #f8f8f8; color: #bb4444;">"name"</span><span style="background-color: #f8f8f8; color: #666666;">,</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #bb4444;">"User_"</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #666666;">+</span><span style="background-color: #f8f8f8;"> suffix</span><span style="background-color: #f8f8f8; color: #666666;">)</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #666666;">.</span><span style="background-color: #f8f8f8; color: #bb4444;">withStringSet</span><span style="background-color: #f8f8f8; color: #666666;">(</span><span style="background-color: #f8f8f8; color: #bb4444;">"hobbies"</span><span style="background-color: #f8f8f8; color: #666666;">,</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #aa22ff; font-weight: bold;">new</span><span style="background-color: #f8f8f8;"> HashSet</span><span style="background-color: #f8f8f8; color: #666666;"><>(</span><span style="background-color: #f8f8f8;">hobbies</span><span style="background-color: #f8f8f8; color: #666666;">))</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #666666;">.</span><span style="background-color: #f8f8f8; color: #bb4444;">withNumber</span><span style="background-color: #f8f8f8; color: #666666;">(</span><span style="background-color: #f8f8f8; color: #bb4444;">"age"</span><span style="background-color: #f8f8f8; color: #666666;">,</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #666666;">25);</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: yellow;">PutItemOutcome putItemOutcom</span><span style="background-color: #f8f8f8;">e </span><span style="background-color: #f8f8f8; color: #666666;">=</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: yellow;">table<span style="color: #666666;">.</span><span style="color: #bb4444;">putItem</span><span style="color: #666666;">(</span>item<span style="color: #666666;">,</span></span><span style="background-color: #f8f8f8;">
</span> <span style="color: #aa22ff; font-weight: bold;">new</span><span style="background-color: yellow;"> Expected</span><span style="color: #666666;">(</span><span style="color: #bb4444;">"userId"</span><span style="color: #666666;">).</span><span style="color: #bb4444;">notExist</span><span style="color: #666666;">());</span> </pre>
<pre style="line-height: 125%; margin: 0px;"><span style="background-color: #f8f8f8;"> System</span><span style="background-color: #f8f8f8; color: #666666;">.</span><span style="background-color: #f8f8f8; color: #bb4444;">out</span><span style="background-color: #f8f8f8; color: #666666;">.</span><span style="background-color: #f8f8f8; color: #bb4444;">println</span><span style="background-color: #f8f8f8; color: #666666;">(</span><span style="background-color: #f8f8f8;">putItemOutcome</span><span style="background-color: #f8f8f8; color: #666666;">);</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #666666;">}</span><span style="background-color: #f8f8f8;">
</span></pre>
</div>
The code uses <b>DynamoDB</b> instance which is a wrapper around the AmazonDynamoDB instance. Table and Item are representations of the dynamo db table and table entry.<br />
The not exists constraint is added through the Expected instance passed to putItem method. The datatype of the attributes is implied from the method used to add them (unlike in LowLevel where it needs to be specified)<br />
. The output to console came as blank
<br />
<div style="background: rgb(248, 248, 248); border-width: 0.1em 0.1em 0.1em 0.8em; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="line-height: 125%; margin: 0px;">{}
</pre>
</div>
From the AWS Docs:
<br />
<div style="background: rgb(248, 248, 248); border-width: 0.1em 0.1em 0.1em 0.8em; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="line-height: 125%; margin: 0px;">Many AWS SDKs provide a <span style="color: blue;">document interface</span>, allowing you to perform data plane</pre>
<pre style="line-height: 125%; margin: 0px;">operations (create, read, update, delete) on tables and indexes. </pre>
<pre style="line-height: 125%; margin: 0px;">With a document interface, you do not need to specify Data Type
Descriptors. The data types are implied by the semantics of the data itself.
These AWS SDKs also provide methods to easily convert JSON documents to
and from native Amazon DynamoDB data types.
</pre>
</div>
<b><div><b><br /></b></div>Technique 3: Use Object Persistence Interface</b></div>
<div style="background: rgb(248, 248, 248); border-width: 0.1em 0.1em 0.1em 0.8em; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="line-height: 125%; margin: 0px;">Some AWS SDKs provide an object persistence interface where you do not directly
perform data plane operations. Instead, you create objects that represent items in Amazon DynamoDB tables and indexes, and
interact only with those objects. This allows you to write object-centric code, rather than database-centric code.
</pre>
</div>
<!--HTML generated using hilite.me--><br />
<div style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="line-height: 125%; margin: 0px;"><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #aa22ff; font-weight: bold;">private</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #aa22ff; font-weight: bold;">static</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #00bb00; font-weight: bold;">void</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #00a000;">createItemUsingObjectPersistenceApi</span><span style="background-color: #f8f8f8; color: #666666;">(</span><span style="background-color: #f8f8f8;">
List</span><span style="background-color: #f8f8f8; color: #666666;"><</span><span style="background-color: #f8f8f8;">String</span><span style="background-color: #f8f8f8; color: #666666;">></span><span style="background-color: #f8f8f8;"> hobbies</span><span style="background-color: #f8f8f8; color: #666666;">,</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #00bb00; font-weight: bold;">int</span><span style="background-color: #f8f8f8;"> suffix</span><span style="background-color: #f8f8f8; color: #666666;">,</span><span style="background-color: #f8f8f8;">
AWSCredentialsProvider awsCredentialsProvider</span><span style="background-color: #f8f8f8; color: #666666;">)</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #666666;">{</span><span style="background-color: #f8f8f8;">
AmazonDynamoDB client </span><span style="background-color: #f8f8f8; color: #666666;">=</span><span style="background-color: #f8f8f8;"> AmazonDynamoDBClientBuilder</span><span style="background-color: #f8f8f8; color: #666666;">.</span><span style="background-color: #f8f8f8; color: #bb4444;">standard</span><span style="background-color: #f8f8f8; color: #666666;">()</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #666666;">.</span><span style="background-color: #f8f8f8; color: #bb4444;">withRegion</span><span style="background-color: #f8f8f8; color: #666666;">(</span><span style="background-color: #f8f8f8; color: #bb4444;">"us-east-1"</span><span style="background-color: #f8f8f8; color: #666666;">)</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #666666;">.</span><span style="background-color: #f8f8f8; color: #bb4444;">withCredentials</span><span style="background-color: #f8f8f8; color: #666666;">(</span><span style="background-color: #f8f8f8;">awsCredentialsProvider</span><span style="background-color: #f8f8f8; color: #666666;">).</span><span style="background-color: #f8f8f8; color: #bb4444;">build</span><span style="background-color: #f8f8f8; color: #666666;">();</span><span style="background-color: #f8f8f8;">
User user </span><span style="background-color: #f8f8f8; color: #666666;">=</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #aa22ff; font-weight: bold;">new</span><span style="background-color: #f8f8f8;"> User</span><span style="background-color: #f8f8f8; color: #666666;">();</span><span style="background-color: #f8f8f8;">
user</span><span style="background-color: #f8f8f8; color: #666666;">.</span><span style="background-color: #f8f8f8; color: #bb4444;">setUserId</span><span style="background-color: #f8f8f8; color: #666666;">(</span><span style="background-color: #f8f8f8; color: #bb4444;">"userId_"</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #666666;">+</span><span style="background-color: #f8f8f8;"> suffix</span><span style="background-color: #f8f8f8; color: #666666;">);</span><span style="background-color: #f8f8f8;">
user</span><span style="background-color: #f8f8f8; color: #666666;">.</span><span style="background-color: #f8f8f8; color: #bb4444;">setUserName</span><span style="background-color: #f8f8f8; color: #666666;">(</span><span style="background-color: #f8f8f8; color: #bb4444;">"User_"</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #666666;">+</span><span style="background-color: #f8f8f8;"> suffix</span><span style="background-color: #f8f8f8; color: #666666;">);</span><span style="background-color: #f8f8f8;">
user</span><span style="background-color: #f8f8f8; color: #666666;">.</span><span style="background-color: #f8f8f8; color: #bb4444;">setAge</span><span style="background-color: #f8f8f8; color: #666666;">(25);</span><span style="background-color: #f8f8f8;">
user</span><span style="background-color: #f8f8f8; color: #666666;">.</span><span style="background-color: #f8f8f8; color: #bb4444;">setHobbies</span><span style="background-color: #f8f8f8; color: #666666;">(</span><span style="background-color: #f8f8f8; color: #aa22ff; font-weight: bold;">new</span><span style="background-color: #f8f8f8;"> HashSet</span><span style="background-color: #f8f8f8; color: #666666;"><>(</span><span style="background-color: #f8f8f8;">hobbies</span><span style="background-color: #f8f8f8; color: #666666;">));</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: yellow;">DynamoDBMapper mapper <span style="color: #666666;">=</span> <span style="color: #aa22ff; font-weight: bold;">new</span> DynamoDBMapper<span style="color: #666666;">(</span>client<span style="color: #666666;">);</span></span><span style="background-color: #f8f8f8;">
Map</span><span style="background-color: #f8f8f8; color: #666666;"><</span><span style="background-color: #f8f8f8;">String</span><span style="background-color: #f8f8f8; color: #666666;">,</span><span style="background-color: #f8f8f8;"> ExpectedAttributeValue</span><span style="background-color: #f8f8f8; color: #666666;">></span><span style="background-color: #f8f8f8;"> expected </span><span style="background-color: #f8f8f8; color: #666666;">=</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #aa22ff; font-weight: bold;">new</span><span style="background-color: #f8f8f8;"> HashMap</span><span style="background-color: #f8f8f8; color: #666666;"><>();</span><span style="background-color: #f8f8f8;">
expected</span><span style="background-color: #f8f8f8; color: #666666;">.</span><span style="background-color: #f8f8f8; color: #bb4444;">put</span><span style="background-color: #f8f8f8; color: #666666;">(</span><span style="background-color: #f8f8f8; color: #bb4444;">"userId"</span><span style="background-color: #f8f8f8; color: #666666;">,</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #aa22ff; font-weight: bold;">new</span><span style="background-color: #f8f8f8;"> ExpectedAttributeValue</span><span style="background-color: #f8f8f8; color: #666666;">().</span><span style="background-color: #f8f8f8; color: #bb4444;">withExists</span><span style="background-color: #f8f8f8; color: #666666;">(</span><span style="background-color: #f8f8f8; color: #aa22ff; font-weight: bold;">false</span><span style="background-color: #f8f8f8; color: #666666;">));</span><span style="background-color: #f8f8f8;">
DynamoDBSaveExpression saveExpression </span><span style="background-color: #f8f8f8; color: #666666;">=</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #aa22ff; font-weight: bold;">new</span><span style="background-color: #f8f8f8;"> DynamoDBSaveExpression</span><span style="background-color: #f8f8f8; color: #666666;">().</span><span style="background-color: #f8f8f8; color: #bb4444;">withExpected</span><span style="background-color: #f8f8f8; color: #666666;">(</span><span style="background-color: #f8f8f8;">expected</span><span style="background-color: #f8f8f8; color: #666666;">);</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: yellow;">mapper<span style="color: #666666;">.</span><span style="color: #bb4444;">save</span><span style="color: #666666;">(</span>user<span style="color: #666666;">,</span> saveExpression<span style="color: #666666;">);</span></span><span style="background-color: #f8f8f8; color: #008800; font-style: italic;">// void method</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #666666;">}</span><span style="background-color: #f8f8f8;"> </span></pre>
</div>
The code uses a DynamoDBMapper instance the which is the object persistence interface in java SDK. The not exists condition is achieved using the DynamoDBSaveExpression. DynamoDBMapper is a wrapper around the AmazonDynamoDB class.<br />
The User class is as below:
<br />
<div style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="line-height: 125%; margin: 0px;"><span style="color: #aa22ff;"><span style="background-color: #f8f8f8;">@</span><span style="background-color: yellow;">DynamoDBTable</span></span><span style="background-color: #f8f8f8; color: #666666;">(</span><span style="background-color: #f8f8f8;">tableName </span><span style="background-color: #f8f8f8; color: #666666;">=</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #bb4444;">"Users"</span><span style="background-color: #f8f8f8; color: #666666;">)</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #aa22ff; font-weight: bold;">public</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #aa22ff; font-weight: bold;">class</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: blue;">User</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #666666;">{</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #aa22ff;">@</span><span style="background-color: yellow; color: #aa22ff;">DynamoDBHashKey</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #aa22ff; font-weight: bold;">private</span><span style="background-color: #f8f8f8;"> String userId</span><span style="background-color: #f8f8f8; color: #666666;">;</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #aa22ff;">@</span><span style="background-color: yellow; color: #aa22ff;">DynamoDBAttribute</span><span style="background-color: #f8f8f8; color: #666666;">(</span><span style="background-color: #f8f8f8;">attributeName </span><span style="background-color: #f8f8f8; color: #666666;">=</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #bb4444;">"name"</span><span style="background-color: #f8f8f8; color: #666666;">)</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #aa22ff; font-weight: bold;">private</span><span style="background-color: #f8f8f8;"> String userName</span><span style="background-color: #f8f8f8; color: #666666;">;</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #aa22ff; font-weight: bold;">private</span><span style="background-color: #f8f8f8;"> Set</span><span style="background-color: #f8f8f8; color: #666666;"><</span><span style="background-color: #f8f8f8;">String</span><span style="background-color: #f8f8f8; color: #666666;">></span><span style="background-color: #f8f8f8;"> hobbies</span><span style="background-color: #f8f8f8; color: #666666;">;</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #aa22ff; font-weight: bold;">private</span><span style="background-color: #f8f8f8;"> Integer age</span><span style="background-color: #f8f8f8; color: #666666;">;</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #008800; font-style: italic;">//setters and getters</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #666666;">}</span><span style="background-color: #f8f8f8;">
</span></pre>
</div>
This is similar to Hibernate annotations. The user pojo needs the DynamoDBTable and DynamoDBHashKey annotations. The DynamoDBAttribute is optional, in this case my db column name deferred from the POJO attribute name.<br />
<br />
These were the three options provided in Java SDK for Dynamo Db operations. All of these translate to POST requests executed on Dynamo endpoints.
<br />
<div style="background: rgb(248, 248, 248); border-width: 0.1em 0.1em 0.1em 0.8em; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="line-height: 125%; margin: 0px;">The Amazon DynamoDB low-level API is the protocol-level interface for DynamoDB.
At this level, every HTTP(S) request must be correctly formatted and carry a
valid digital signature.
The AWS SDKs construct low-level DynamoDB API requests on your behalf and process
the responses from DynamoDB. This lets you focus on your application logic,
instead of low-level details.
</pre>
</div>
</div>
Robin Varghesehttp://www.blogger.com/profile/00841917517097216645noreply@blogger.com0tag:blogger.com,1999:blog-6262128735818493674.post-39871651608197852612020-06-02T10:48:00.000+05:302020-06-02T10:48:31.468+05:30More on DAX <div dir="ltr" style="text-align: left;" trbidi="on">
In the previous post we setup a DAX client and tested the GetItem performance. This post will look at other DAX features<br />
<a name='more'></a><br />
BatchGetItem is also supported by DAX. DAX will fetch the items in Cache and for rest it will return the items from Dynamo. Similar is the story for BatchGet
<br />
<div style="background: #f8f8f8; border-width: 0.1em 0.1em 0.1em 0.8em; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="line-height: 125%; margin: 0;">In addition to GetItem, the DAX client also supports BatchGetItem requests. </pre>
<pre style="line-height: 125%; margin: 0;">BatchGetItem is essentially a wrapper around one or more GetItem requests, so DAX</pre>
<pre style="line-height: 125%; margin: 0;">treats each of these as an individual GetItem operation.
</pre>
</div>
<b>All Write operations are treated as 'write-through':</b><br />
<div style="background: rgb(248, 248, 248); border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="line-height: 16.25px;">BatchWriteItem, UpdateItem, DeleteItem, PutItem operations, data is first written
to the DynamoDB table, and then to the DAX cluster. The operation is successful
only if the data is successfully written to both the table and to DAX.
</pre>
</div>
These are operations against Item Cache. DAX also provides a <b>QueryCache</b><br />
<div style="background: #f8f8f8; border-width: 0.1em 0.1em 0.1em 0.8em; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="line-height: 125%; margin: 0;">DAX caches the results from Query and Scan requests in its query cache. However,
these results don't affect the item cache at all. When your application issues
a Query or Scan request with DAX, the result set is saved in the query cache — </pre>
<pre style="line-height: 125%; margin: 0;">not in the item cache. You can't "warm up" the item cache by performing a Scan
operation because the item cache and query cache are separate entities.
</pre>
</div>
<!-- HTML generated using hilite.me --><br />
<div style="background: #f8f8f8; border-width: 0.1em 0.1em 0.1em 0.8em; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="line-height: 125%; margin: 0;"> <span style="color: #aa22ff; font-weight: bold;">private</span> <span style="color: #00bb00; font-weight: bold;">void</span> <span style="color: #00a000;">scanItemsTest</span><span style="color: #666666;">(</span>LambdaLogger lambdaLogger<span style="color: #666666;">,</span> Table dbTable<span style="color: #666666;">,</span> Table daxTable<span style="color: #666666;">)</span> <span style="color: #666666;">{</span>
<span style="color: #008800; font-style: italic;">//Test 1 - Get without DAX</span>
<span style="color: #008800; font-style: italic;">//Lambda needs</span>
lambdaLogger<span style="color: #666666;">.</span><span style="color: #bb4444;">log</span><span style="color: #666666;">(</span><span style="color: #bb4444;">"Tests for GetItem"</span><span style="color: #666666;">);</span>
lambdaLogger<span style="color: #666666;">.</span><span style="color: #bb4444;">log</span><span style="color: #666666;">(</span><span style="color: #bb4444;">"Dynamo Tests"</span><span style="color: #666666;">);</span>
computeScanItemDuration<span style="color: #666666;">(</span>lambdaLogger<span style="color: #666666;">,</span> dbTable<span style="color: #666666;">);</span>
<span style="color: #008800; font-style: italic;">//Test 2 - Get with DAX - no entry in cache</span>
lambdaLogger<span style="color: #666666;">.</span><span style="color: #bb4444;">log</span><span style="color: #666666;">(</span><span style="color: #bb4444;">"DAX Tests - not in Cache"</span><span style="color: #666666;">);</span>
computeScanItemDuration<span style="color: #666666;">(</span>lambdaLogger<span style="color: #666666;">,</span> daxTable<span style="color: #666666;">);</span>
<span style="color: #008800; font-style: italic;">//Test 3 - Get with DAX - entry in cache</span>
lambdaLogger<span style="color: #666666;">.</span><span style="color: #bb4444;">log</span><span style="color: #666666;">(</span><span style="color: #bb4444;">"DAX Tests - present in Cache"</span><span style="color: #666666;">);</span>
computeScanItemDuration<span style="color: #666666;">(</span>lambdaLogger<span style="color: #666666;">,</span> daxTable<span style="color: #666666;">);</span>
<span style="color: #666666;">}</span>
<span style="color: #aa22ff; font-weight: bold;">private</span> <span style="color: #00bb00; font-weight: bold;">void</span> <span style="color: #00a000;">computeScanItemDuration</span><span style="color: #666666;">(</span>LambdaLogger lambdaLogger<span style="color: #666666;">,</span> Table table<span style="color: #666666;">)</span> <span style="color: #666666;">{</span>
ScanFilter scanFilter <span style="color: #666666;">=</span> <span style="color: #aa22ff; font-weight: bold;">new</span> ScanFilter<span style="color: #666666;">(</span><span style="color: #bb4444;">"age"</span><span style="color: #666666;">)</span>
<span style="color: #666666;">.</span><span style="color: #bb4444;">between</span><span style="color: #666666;">(20,</span> <span style="color: #666666;">25);</span>
<span style="color: #00bb00; font-weight: bold;">long</span> startTime <span style="color: #666666;">=</span> System<span style="color: #666666;">.</span><span style="color: #bb4444;">nanoTime</span><span style="color: #666666;">();</span>
ItemCollection<span style="color: #666666;"><</span>ScanOutcome<span style="color: #666666;">></span> outcomeItemCollection <span style="color: #666666;">=</span> table<span style="color: #666666;">.</span><span style="color: #bb4444;">scan</span><span style="color: #666666;">(</span>scanFilter<span style="color: #666666;">);</span>
<span style="color: #00bb00; font-weight: bold;">int</span> totalAge <span style="color: #666666;">=</span> <span style="color: #666666;">0,</span> totalRecs <span style="color: #666666;">=</span> <span style="color: #666666;">0;</span>
<span style="color: #00bb00; font-weight: bold;">int</span> pageNo <span style="color: #666666;">=</span> <span style="color: #666666;">1;</span>
<span style="color: #aa22ff; font-weight: bold;">for</span> <span style="color: #666666;">(</span>Page<span style="color: #666666;"><</span>Item<span style="color: #666666;">,</span> ScanOutcome<span style="color: #666666;">></span> page <span style="color: #666666;">:</span> outcomeItemCollection<span style="color: #666666;">.</span><span style="color: #bb4444;">pages</span><span style="color: #666666;">())</span> <span style="color: #666666;">{</span>
ScanOutcome lowLevelResult <span style="color: #666666;">=</span> page<span style="color: #666666;">.</span><span style="color: #bb4444;">getLowLevelResult</span><span style="color: #666666;">();</span>
totalAge <span style="color: #666666;">+=</span> lowLevelResult<span style="color: #666666;">.</span><span style="color: #bb4444;">getItems</span><span style="color: #666666;">().</span><span style="color: #bb4444;">stream</span><span style="color: #666666;">()</span>
<span style="color: #666666;">.</span><span style="color: #bb4444;">map</span><span style="color: #666666;">(</span>item <span style="color: #666666;">-></span> <span style="color: #666666;">(</span>BigDecimal<span style="color: #666666;">)</span> <span style="color: #666666;">(</span>item<span style="color: #666666;">.</span><span style="color: #bb4444;">get</span><span style="color: #666666;">(</span><span style="color: #bb4444;">"age"</span><span style="color: #666666;">)))</span>
<span style="color: #666666;">.</span><span style="color: #bb4444;">mapToInt</span><span style="color: #666666;">(</span><span style="color: #a0a000;">BigDecimal:</span><span style="color: #666666;">:</span>intValue<span style="color: #666666;">).</span><span style="color: #bb4444;">sum</span><span style="color: #666666;">();</span>
pageNo<span style="color: #666666;">++;</span>
totalRecs <span style="color: #666666;">+=</span> lowLevelResult<span style="color: #666666;">.</span><span style="color: #bb4444;">getItems</span><span style="color: #666666;">().</span><span style="color: #bb4444;">size</span><span style="color: #666666;">();</span>
<span style="color: #666666;">}</span>
lambdaLogger<span style="color: #666666;">.</span><span style="color: #bb4444;">log</span><span style="color: #666666;">(</span><span style="color: #bb4444;">"Tested total of "</span> <span style="color: #666666;">+</span> totalRecs <span style="color: #666666;">+</span> <span style="color: #bb4444;">" with avg age found to be "</span> <span style="color: #666666;">+</span> <span style="color: #666666;">(</span>totalAge <span style="color: #666666;">/</span> totalRecs<span style="color: #666666;">)</span> <span style="color: #666666;">+</span> <span style="color: #bb4444;">". Total Pages "</span> <span style="color: #666666;">+</span> <span style="color: #666666;">(</span>pageNo<span style="color: #666666;">-1));</span>
lambdaLogger<span style="color: #666666;">.</span><span style="color: #bb4444;">log</span><span style="color: #666666;">(</span><span style="color: #bb4444;">"Average FetchTime = "</span> <span style="color: #666666;">+</span> <span style="color: #666666;">(</span>System<span style="color: #666666;">.</span><span style="color: #bb4444;">nanoTime</span><span style="color: #666666;">()</span> <span style="color: #666666;">-</span> startTime<span style="color: #666666;">)</span> <span style="color: #666666;">+</span> <span style="color: #bb4444;">" nano seconds"</span><span style="color: #666666;">);</span>
<span style="color: #666666;">}</span>
</pre>
</div>
The results of the test are as below:
<!-- HTML generated using hilite.me --><br />
<div style="background: #f8f8f8; border-width: 0.1em 0.1em 0.1em 0.8em; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="line-height: 125%; margin: 0;">START RequestId: 7de2a34d-abd0-4d7f-9d3c-87ae3cf117ce Version: $LATEST
Tests for ScanItem
Dynamo Tests
Tested total of 10042 with avg age found to be 22. Total Pages 4
Average FetchTime = 11459072286 nano seconds
DAX Tests - not in Cache
Tested total of 10042 with avg age found to be 22. Total Pages 4</pre>
<pre style="line-height: 125%; margin: 0;">Average FetchTime = 4819390513 nano seconds
DAX Tests - present in Cache
Tested total of 10042 with avg age found to be 22. Total Pages 4
Average FetchTime = 1823347748 nano seconds</pre>
<pre style="line-height: 125%; margin: 0;">END RequestId: 7de2a34d-abd0-4d7f-9d3c-87ae3cf117ce
REPORT RequestId: 7de2a34d-abd0-4d7f-9d3c-87ae3cf117ce </pre>
<pre style="line-height: 125%; margin: 0;">Duration: 18234.71 ms Billed Duration: 18300 ms </pre>
<pre style="line-height: 125%; margin: 0;">Memory Size: 256 MB Max Memory Used: 142 MB Init Duration: 1757.49 ms
</pre>
</div>
<br />
As seen the Query cache times are lower than Dynamo. However there are some caveats to using the Query Cache.
<!-- HTML generated using hilite.me --><br />
<div style="background: #f8f8f8; border-width: 0.1em 0.1em 0.1em 0.8em; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="line-height: 125%; margin: 0;"><span style="color: red;">Updates to the item cache, or to the underlying DynamoDB table, do not invalidate
or modify the results stored in the query cache. </span>Your application should consider
the TTL value for the query cache and how long your application can tolerate
inconsistent results between the query cache and the item cache.
</pre>
</div>
If we added some records to table with age 25, we would expect the average age to change. However since DAX query cache is not aware of these changes it would not reflect the update.<br />
There is another corner case with DAX -<br />
<!-- HTML generated using hilite.me --><br />
<div style="background: #f8f8f8; border-width: 0.1em 0.1em 0.1em 0.8em; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="line-height: 125%; margin: 0;">DynamoDb read requests (Get, Scan, query) are eventually consistent by default.
DAX attempts to return results for such queries. If the requests are marked as
<b>strongly consistent reads</b>, than DAX ignores such calls forwarding to Dynamo.
It does not update it's cache data based on results of such calls.
</pre>
</div>
<br />
One more cool aspect with DAX is 'negative caching'. Consider the case where we have multiple clients trying to locate an item in DAX. If the item exists, DAX would cache it for future use. But what if the item does not exists ?
<br />
<div style="background: #f8f8f8; border-width: 0.1em 0.1em 0.1em 0.8em; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="line-height: 125%; margin: 0;">DAX supports <b><span style="color: blue;">negative cache</span></b> entries in both the item cache and the query cache.
A negative cache entry occurs when DAX can't find requested items in an
underlying DynamoDB table. Instead of generating an error, DAX caches an
empty result and returns that result to the user.
For example, suppose that an application sends a GetItem request to a DAX
cluster, and that there is no matching item in the DAX item cache. This causes
DAX to read the corresponding item from the underlying DynamoDB table. If the
item doesn't exist in DynamoDB, DAX stores an empty item in its item cache
and returns the empty item to the application. Now suppose that the
application sends another GetItem request for the same item. DAX finds the
empty item in the item cache and returns it to the application immediately.
It does not consult DynamoDB at all.
<span style="color: blue;">A negative cache entry remains in the DAX item cache until its item TTL has
expired, its LRU is invoked, or the item is modified using PutItem,
UpdateItem, or DeleteItem.</span>
The DAX query cache handles negative cache results in a similar way. If an
application performs a Query or Scan, and the DAX query cache doesn't
contain a cached result, DAX sends the request to DynamoDB. If there are
no matching items in the result set, DAX stores an empty result set in
the query cache and returns the empty result set to the application.
Subsequent Query or Scan requests yield the same (empty) result set,
until the TTL for that result set has expired.
</pre>
</div>
<br /></div>
Robin Varghesehttp://www.blogger.com/profile/00841917517097216645noreply@blogger.com1tag:blogger.com,1999:blog-6262128735818493674.post-4367805762400257152020-06-02T10:47:00.000+05:302020-06-02T10:47:01.295+05:30AWS Athena<div dir="ltr" style="text-align: left;" trbidi="on">
I have been going through use cases where some basic analytics needed to be run on structured logs generated by our system. The way I did it till now, is to spin up an EMR cluster, load my logs on it and execute hive queries.
<br />
<a name='more'></a>Then I found Athena.<br />
<div style="background: #f8f8f8; border-width: 0.1em 0.1em 0.1em 0.8em; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="line-height: 125%; margin: 0;">Amazon Athena is an interactive query service that makes it easy to analyze
data directly in Amazon Simple Storage Service (Amazon S3) using standard SQL.
</pre>
</div>
We already have most of our relevant data in S3. Which means we cold use Athena directly with it. Other Benefits:
<br />
<div style="background: #f8f8f8; border-width: 0.1em 0.1em 0.1em 0.8em; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="line-height: 125%; margin: 0;">Athena is serverless, so <span style="color: blue;">there is no infrastructure to set up or manage</span>,
and you <span style="color: blue;">pay only for the queries you run</span>.
Athena <span style="color: blue;">scales automatically -executing queries in parallel</span>—so results
are fast, even with large datasets and complex queries.
</pre>
</div>
I wrote up a dummy code that created files of user data
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiIHG_gay6gZfJtN7JfhyG2yA1a7X4Td1X7NQwoiRPpmvVkRm5CTvdTlFKfCV2GEl5GXE6mit2p6ntD2hi0Kg6_skgeopOrhUulqlqdAwzlbHitTTCqYoud5Xp8j2vxPbipGzqg-0A3cQo/s1600/SamplUserDataAthena.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="95" data-original-width="704" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiIHG_gay6gZfJtN7JfhyG2yA1a7X4Td1X7NQwoiRPpmvVkRm5CTvdTlFKfCV2GEl5GXE6mit2p6ntD2hi0Kg6_skgeopOrhUulqlqdAwzlbHitTTCqYoud5Xp8j2vxPbipGzqg-0A3cQo/s1600/SamplUserDataAthena.png" /></a></div>
I created a few files and then setup a directory structure in s3:
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjq8RjOW7KWhXhBK2Ut-BzTz5CcDNqrHOmeaOH7uKOqNQAuvFT7aXm-k9lua9XaG3R3qb2NNT4etx3nIFxw2VGwr93SL3ecsW2bEd95YzJnHkXUO6StF9KhkZMam84CRjSAMuhy6oEczk0/s1600/S3DataLocnAthena.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="455" data-original-width="480" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjq8RjOW7KWhXhBK2Ut-BzTz5CcDNqrHOmeaOH7uKOqNQAuvFT7aXm-k9lua9XaG3R3qb2NNT4etx3nIFxw2VGwr93SL3ecsW2bEd95YzJnHkXUO6StF9KhkZMam84CRjSAMuhy6oEczk0/s1600/S3DataLocnAthena.png" /></a></div>
The next step was to query this data in Athena. For this we need to make a database and table in Athena.<br />
<div style="background: #f8f8f8; border-width: 0.1em 0.1em 0.1em 0.8em; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="line-height: 125%; margin: 0;">For each dataset, a table needs to exist in Athena. The metadata in the
table tells Athena where the data is located in Amazon S3, and
specifies the structure of the data, for example, column names, data
types, and the name of the table. Databases are a logical grouping of
tables, and also hold only metadata and schema information for a dataset.
The tables creation process registers the dataset with Athena. This
registration occurs in the AWS Glue Data Catalog and enables Athena to
run queries on the data.
</pre>
</div>
<br />
Step 1: Create the database:
<!-- HTML generated using hilite.me --><br />
<div style="background: #f8f8f8; border-width: 0.1em 0.1em 0.1em 0.8em; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="line-height: 125%; margin: 0;"><span style="color: #aa22ff; font-weight: bold;">CREATE</span> <span style="color: #aa22ff; font-weight: bold;">DATABASE</span> users_db
</pre>
</div>
Step 2: Create the users table:
<!-- HTML generated using hilite.me --><br />
<div style="background: #f8f8f8; border-width: 0.1em 0.1em 0.1em 0.8em; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="line-height: 125%; margin: 0;"><span style="color: #aa22ff; font-weight: bold;">CREATE</span> <span style="color: #aa22ff; font-weight: bold;">EXTERNAL</span> <span style="color: #aa22ff; font-weight: bold;">TABLE</span> IF <span style="color: #aa22ff; font-weight: bold;">NOT</span> <span style="color: #aa22ff; font-weight: bold;">EXISTS</span> userRecs (
user_id <span style="color: #aa22ff;">int</span>,
name String,
phone_no String,
age <span style="color: #aa22ff;">int</span>,
hobbies <span style="color: #aa22ff;">array</span><span style="color: #666666;"><</span>string<span style="color: #666666;">></span>,
<span style="color: #aa22ff; font-weight: bold;">state</span> String,
country String)
<span style="color: #aa22ff; font-weight: bold;">COMMENT</span> <span style="color: #bb4444;">'User details'</span>
<span style="color: #aa22ff; font-weight: bold;">ROW</span> FORMAT DELIMITED
FIELDS TERMINATED <span style="color: #aa22ff; font-weight: bold;">BY</span> <span style="color: #bb4444;">'\t'</span>
COLLECTION ITEMS TERMINATED <span style="color: #aa22ff; font-weight: bold;">BY</span> <span style="color: #bb4444;">','</span>
LINES TERMINATED <span style="color: #aa22ff; font-weight: bold;">BY</span> <span style="color: #bb4444;">'\n'</span>
STORED <span style="color: #aa22ff; font-weight: bold;">AS</span> TEXTFILE
<span style="color: #aa22ff; font-weight: bold;">LOCATION</span> <span style="color: #bb4444;">'s3://athene-test-dump/'</span>;
</pre>
</div>
The table here is created over the base S3 bucket. I did not apply any partioning. Simply provided the base folder and let Athena detect the files under the folder hierarchy.
<br />
<div style="background: #f8f8f8; border-width: 0.1em 0.1em 0.1em 0.8em; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="line-height: 125%; margin: 0;">When you query an existing table, under the hood, Amazon Athena uses Presto,
a distributed SQL engine.
</pre>
</div>
I executed a simple query:<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiD6SoQ4QoceohzqDJqH8ceQQ240HmzWkYeb6YGCcwo3oVNcUAV8_yh51VnWw25DmaVVOfA7TpyRdK8yulFtKuEcHD7gtIz2kp3ShMbPddArLCYg4RgFepYp1bNzEtAQMAc5_YGHEnqc5s/s1600/SampleQueryAthena.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="419" data-original-width="1021" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiD6SoQ4QoceohzqDJqH8ceQQ240HmzWkYeb6YGCcwo3oVNcUAV8_yh51VnWw25DmaVVOfA7TpyRdK8yulFtKuEcHD7gtIz2kp3ShMbPddArLCYg4RgFepYp1bNzEtAQMAc5_YGHEnqc5s/s1600/SampleQueryAthena.png" /></a></div>
The query execution history is as below:<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgXM5XA9XXV8K7NtYadajo78K0LzluRVDM6pY3ZcRLAQymghyphenhyphenV1SGRU7N5OKIjqpPcr7he_qMqBdG-M0LBsLz1Q40s3gkh1UL59T_RBnSfP19qvTJf0chURDHBsweL9W-DReBKnlfXKOzk/s1600/AthenaExecHistory.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="266" data-original-width="954" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgXM5XA9XXV8K7NtYadajo78K0LzluRVDM6pY3ZcRLAQymghyphenhyphenV1SGRU7N5OKIjqpPcr7he_qMqBdG-M0LBsLz1Q40s3gkh1UL59T_RBnSfP19qvTJf0chURDHBsweL9W-DReBKnlfXKOzk/s1600/AthenaExecHistory.png" /></a></div>
I also decided to setup a partitioned table for the same data:<br />
<!-- HTML generated using hilite.me --><br />
<div style="background: #f8f8f8; border-width: 0.1em 0.1em 0.1em 0.8em; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="line-height: 125%; margin: 0;"><span style="color: #aa22ff; font-weight: bold;">CREATE</span> <span style="color: #aa22ff; font-weight: bold;">EXTERNAL</span> <span style="color: #aa22ff; font-weight: bold;">TABLE</span> IF <span style="color: #aa22ff; font-weight: bold;">NOT</span> <span style="color: #aa22ff; font-weight: bold;">EXISTS</span> users_partitioned (
user_id <span style="color: #aa22ff;">int</span>,
name String,
phone_no String,
age <span style="color: #aa22ff;">int</span>,
hobbies <span style="color: #aa22ff;">array</span><span style="color: #666666;"><</span>string<span style="color: #666666;">></span>,
<span style="color: #aa22ff; font-weight: bold;">state</span> String,
country String
) <span style="color: #aa22ff; font-weight: bold;">COMMENT</span> <span style="color: #bb4444;">'User details'</span> PARTITIONED <span style="color: #aa22ff; font-weight: bold;">BY</span> (dataset_date String) <span style="color: #aa22ff; font-weight: bold;">ROW</span> FORMAT DELIMITED FIELDS TERMINATED <span style="color: #aa22ff; font-weight: bold;">BY</span> <span style="color: #bb4444;">'\t'</span> COLLECTION ITEMS TERMINATED <span style="color: #aa22ff; font-weight: bold;">BY</span> <span style="color: #bb4444;">','</span> LINES TERMINATED <span style="color: #aa22ff; font-weight: bold;">BY</span> <span style="color: #bb4444;">'\n'</span> STORED <span style="color: #aa22ff; font-weight: bold;">AS</span> TEXTFILE <span style="color: #aa22ff; font-weight: bold;">LOCATION</span> <span style="color: #bb4444;">'s3://athene-test-dump/'</span>;
</pre>
</div>
The query executed successfully and Athena UI gave the below comment:
<br />
<div style="background: #f8f8f8; border-width: 0.1em 0.1em 0.1em 0.8em; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="line-height: 125%; margin: 0;">Query successful. If your table has partitions, you need to load these partitions
to be able to query data. You can either load all partitions or load them
individually. If you use the load all partitions (MSCK REPAIR TABLE) command,
partitions must be in a format understood by Hive. Learn more.
</pre>
</div>
My data is not partitioned in the Hive format. So I will have to manually load the partitions.
<br />
<div style="background: #f8f8f8; border-width: 0.1em 0.1em 0.1em 0.8em; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="line-height: 125%; margin: 0;"><span style="color: #aa22ff; font-weight: bold;">ALTER</span> <span style="color: #aa22ff; font-weight: bold;">TABLE</span> users_partitioned <span style="color: #aa22ff; font-weight: bold;">ADD</span> PARTITION (dataset_date<span style="color: #666666;">=</span><span style="color: #bb4444;">'2020-05-28'</span>) </pre>
<pre style="line-height: 125%; margin: 0;"><span style="color: #aa22ff; font-weight: bold;">location</span> <span style="color: #bb4444;">'s3://athene-test-dump/2020/05/28'</span>;
</pre>
</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi4rzM2Oe-GDiR-jJLrw4iG-jrtYBBUMq58RgRwUpBT4VBZLQCfeP-avFT6RZXwWn_9y1lEMKkzkIUlojsPM-G3GHMbce0ze8Q_1IHoj47-P_7uPnyeRKthChczPxmOF50dD2JJlB7m-pw/s1600/AthenaPartionedQuery.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="506" data-original-width="1277" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi4rzM2Oe-GDiR-jJLrw4iG-jrtYBBUMq58RgRwUpBT4VBZLQCfeP-avFT6RZXwWn_9y1lEMKkzkIUlojsPM-G3GHMbce0ze8Q_1IHoj47-P_7uPnyeRKthChczPxmOF50dD2JJlB7m-pw/s1600/AthenaPartionedQuery.png" /></a></div>
<br />
<br />
If instead of date being parts of different folder levels (i.e. s3://athene-test-dump/2020/05/28), I had it as 's3://athene-test-dump/dataset=2020-05-28', than I could have loaded it using the MSCK REPAIR TABLE command.</div>
Robin Varghesehttp://www.blogger.com/profile/00841917517097216645noreply@blogger.com1tag:blogger.com,1999:blog-6262128735818493674.post-19611282081420670542020-05-31T04:44:00.001+05:302020-05-31T04:44:51.951+05:30DAX - speeding/cheapening up Dynamo Db<div dir="ltr" style="text-align: left;" trbidi="on">
Amazon DynamoDb comes with its own cache layer DAX or DynamoDB Accelerator. In this post I am going to play around with this feature.<br />
<a name='more'></a>DAX as the AWS docs say provide 3 benefits:<br />
<br />
<ol style="text-align: left;">
<li>It's a cache - so the response from caches will be faster than hitting the actual table.</li>
<li>It's a cache - so the number of calls to database are reduced, allowing us to save up on costs.</li>
<li>It's provided by DynamoDb - The api is like a wrapper that can be dropped in code. So minimal changes and easier than setting up our own cache layer.</li>
</ol>
DAX like ElastiCache is setup within a VPC. The cache is essentially a wrapper:
<!-- HTML generated using hilite.me --><br />
<div style="background: #f8f8f8; border-width: 0.1em 0.1em 0.1em 0.8em; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="line-height: 125%; margin: 0;">At runtime, the DAX client directs all of your application's DynamoDB API requests
to the DAX cluster. If DAX can process one of these API requests directly, it
does so. Otherwise, it passes the request through to DynamoDB. Finally, the
DAX cluster returns the results to your application.
</pre>
</div>
<br />
To test DAX, I need to setup a Dynamo table first:<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiRZ44nGo5Lfs7bWsWyNkGcm5wuoss3ct0mmr7oXEkOQu4nDZ6cly2SDaRXcP1GR2gtDd63SW0jD7qTTxWfwC2gvA_TnfXQEUzJWVACBXKoSHAMu-ZHDZr0ZGFrzlOZ-GkJpw9250YCFMk/s1600/UserDynTbl.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="596" data-original-width="454" height="640" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiRZ44nGo5Lfs7bWsWyNkGcm5wuoss3ct0mmr7oXEkOQu4nDZ6cly2SDaRXcP1GR2gtDd63SW0jD7qTTxWfwC2gvA_TnfXQEUzJWVACBXKoSHAMu-ZHDZr0ZGFrzlOZ-GkJpw9250YCFMk/s640/UserDynTbl.png" width="486" /></a></div>
<br />
Below code creates 100K records in Users table:<br />
<div style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="background-color: #f8f8f8; line-height: 125%; margin: 0px;"> <span style="color: #aa22ff; font-weight: bold;">public</span> <span style="color: #aa22ff; font-weight: bold;">static</span> <span style="color: #00bb00; font-weight: bold;">void</span> <span style="color: #00a000;">main</span><span style="color: #666666;">(</span>String<span style="color: #666666;">[]</span> args<span style="color: #666666;">)</span> <span style="color: #666666;">{</span>
AWSCredentials awsCredentials <span style="color: #666666;">=</span> <span style="color: #aa22ff; font-weight: bold;">new</span> BasicAWSCredentials<span style="color: #666666;">(</span></pre>
<pre style="background-color: #f8f8f8; line-height: 125%; margin: 0px;"><span style="color: #bb4444;"> "AccessKey"</span><span style="color: #666666;">,</span><span style="color: #666666;"> </span><span style="color: #bb4444;">"SecretKey"</span><span style="color: #666666;">);</span>
AWSCredentialsProvider awsCredentialsProvider <span style="color: #666666;">=</span> <span style="color: #aa22ff; font-weight: bold;">new</span> </pre>
<pre style="line-height: 125%; margin: 0px;"><span style="background-color: #f8f8f8;"> AWSStaticCredentialsProvider</span><span style="background-color: #f8f8f8; color: #666666;">(</span><span style="background-color: #f8f8f8;">awsCredentials</span><span style="background-color: #f8f8f8; color: #666666;">);</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: yellow;">AmazonDynamoDB client <span style="color: #666666;">=</span> AmazonDynamoDBClientBuilder<span style="color: #666666;">.</span><span style="color: #bb4444;">standard</span><span style="color: #666666;">()</span></span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #666666;">.</span><span style="background-color: #f8f8f8; color: #bb4444;">withRegion</span><span style="background-color: #f8f8f8; color: #666666;">(</span><span style="background-color: #f8f8f8; color: #bb4444;">"us-east-1"</span><span style="background-color: #f8f8f8; color: #666666;">)</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #666666;">.</span><span style="background-color: #f8f8f8; color: #bb4444;">withCredentials</span><span style="background-color: #f8f8f8; color: #666666;">(</span><span style="background-color: #f8f8f8;">awsCredentialsProvider</span><span style="background-color: #f8f8f8; color: #666666;">).</span><span style="background-color: #f8f8f8; color: #bb4444;">build</span><span style="background-color: #f8f8f8; color: #666666;">();</span><span style="background-color: #f8f8f8;">
DynamoDB dynamoDB </span><span style="background-color: #f8f8f8; color: #666666;">=</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: yellow;"><span style="color: #aa22ff; font-weight: bold;">new</span> DynamoDB<span style="color: #666666;">(</span>client<span style="color: #666666;">);</span></span><span style="background-color: #f8f8f8;">
List</span><span style="background-color: #f8f8f8; color: #666666;"><</span><span style="background-color: #f8f8f8;">Item</span><span style="background-color: #f8f8f8; color: #666666;">></span><span style="background-color: #f8f8f8;"> userItems </span><span style="background-color: #f8f8f8; color: #666666;">=</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #aa22ff; font-weight: bold;">new</span><span style="background-color: #f8f8f8;"> ArrayList</span><span style="background-color: #f8f8f8; color: #666666;"><>();</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #aa22ff; font-weight: bold;">for</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #666666;">(</span><span style="background-color: #f8f8f8; color: #00bb00; font-weight: bold;">int</span><span style="background-color: #f8f8f8;"> i </span><span style="background-color: #f8f8f8; color: #666666;">=</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #666666;">0;</span><span style="background-color: #f8f8f8;"> i </span><span style="background-color: #f8f8f8; color: #666666;"><</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #666666;">100000;</span><span style="background-color: #f8f8f8;"> i</span><span style="background-color: #f8f8f8; color: #666666;">++)</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #666666;">{</span><span style="background-color: #f8f8f8;">
Item item </span><span style="background-color: #f8f8f8; color: #666666;">=</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #aa22ff; font-weight: bold;">new</span><span style="background-color: #f8f8f8;"> Item</span><span style="background-color: #f8f8f8; color: #666666;">().</span><span style="background-color: #f8f8f8; color: #bb4444;">withPrimaryKey</span><span style="background-color: #f8f8f8; color: #666666;">(</span><span style="background-color: #f8f8f8; color: #bb4444;">"userId"</span><span style="background-color: #f8f8f8; color: #666666;">,</span><span style="background-color: #f8f8f8;"> Integer</span><span style="background-color: #f8f8f8; color: #666666;">.</span><span style="background-color: #f8f8f8; color: #bb4444;">valueOf</span><span style="background-color: #f8f8f8; color: #666666;">(</span><span style="background-color: #f8f8f8;">i</span><span style="background-color: #f8f8f8; color: #666666;">))</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #666666;">.</span><span style="background-color: #f8f8f8; color: #bb4444;">withString</span><span style="background-color: #f8f8f8; color: #666666;">(</span><span style="background-color: #f8f8f8; color: #bb4444;">"name"</span><span style="background-color: #f8f8f8; color: #666666;">,</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #bb4444;">"User_"</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #666666;">+</span><span style="background-color: #f8f8f8;"> i</span><span style="background-color: #f8f8f8; color: #666666;">)</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #666666;">.</span><span style="background-color: #f8f8f8; color: #bb4444;">withNumber</span><span style="background-color: #f8f8f8; color: #666666;">(</span><span style="background-color: #f8f8f8; color: #bb4444;">"age"</span><span style="background-color: #f8f8f8; color: #666666;">,</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #666666;">(</span><span style="background-color: #f8f8f8; color: #00bb00; font-weight: bold;">int</span><span style="background-color: #f8f8f8; color: #666666;">)</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #666666;">(</span><span style="background-color: #f8f8f8;">Math</span><span style="background-color: #f8f8f8; color: #666666;">.</span><span style="background-color: #f8f8f8; color: #bb4444;">random</span><span style="background-color: #f8f8f8; color: #666666;">()</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #666666;">*</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #666666;">60));</span><span style="background-color: #f8f8f8;">
userItems</span><span style="background-color: #f8f8f8; color: #666666;">.</span><span style="background-color: #f8f8f8; color: #bb4444;">add</span><span style="background-color: #f8f8f8; color: #666666;">(</span><span style="background-color: #f8f8f8;">item</span><span style="background-color: #f8f8f8; color: #666666;">);</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #aa22ff; font-weight: bold;">if</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #666666;">(</span><span style="background-color: #f8f8f8;">userItems</span><span style="background-color: #f8f8f8; color: #666666;">.</span><span style="background-color: #f8f8f8; color: #bb4444;">size</span><span style="background-color: #f8f8f8; color: #666666;">()</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #666666;">==</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #666666;">25)</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #666666;">{</span><span style="background-color: #f8f8f8; color: #008800; font-style: italic;">//max entries allowed is 25</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: yellow;">TableWriteItems</span><span style="background-color: #f8f8f8;"> userTableWriteItems </span><span style="background-color: #f8f8f8; color: #666666;">=</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #aa22ff; font-weight: bold;">new</span><span style="background-color: #f8f8f8;"> TableWriteItems</span><span style="background-color: #f8f8f8; color: #666666;">(</span><span style="background-color: #f8f8f8; color: #bb4444;">"Users"</span><span style="background-color: #f8f8f8; color: #666666;">)</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #666666;">.</span><span style="background-color: #f8f8f8; color: #bb4444;">withItemsToPut</span><span style="background-color: #f8f8f8; color: #666666;">(</span><span style="background-color: #f8f8f8;">userItems</span><span style="background-color: #f8f8f8; color: #666666;">);</span><span style="background-color: #f8f8f8;">
System</span><span style="background-color: #f8f8f8; color: #666666;">.</span><span style="background-color: #f8f8f8; color: #bb4444;">out</span><span style="background-color: #f8f8f8; color: #666666;">.</span><span style="background-color: #f8f8f8; color: #bb4444;">println</span><span style="background-color: #f8f8f8; color: #666666;">(</span><span style="background-color: #f8f8f8; color: #bb4444;">"Making the request... "</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #666666;">+</span><span style="background-color: #f8f8f8;"> i</span><span style="background-color: #f8f8f8; color: #666666;">);</span><span style="background-color: #f8f8f8;">
batchWriteItems</span><span style="background-color: #f8f8f8; color: #666666;">(</span><span style="background-color: #f8f8f8;">dynamoDB</span><span style="background-color: #f8f8f8; color: #666666;">,</span><span style="background-color: #f8f8f8;"> userTableWriteItems</span><span style="background-color: #f8f8f8; color: #666666;">);</span><span style="background-color: #f8f8f8;">
userItems</span><span style="background-color: #f8f8f8; color: #666666;">.</span><span style="background-color: #f8f8f8; color: #bb4444;">clear</span><span style="background-color: #f8f8f8; color: #666666;">();</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #666666;">}</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #666666;">}</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #666666;">}</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #aa22ff; font-weight: bold;">private</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #aa22ff; font-weight: bold;">static</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #00bb00; font-weight: bold;">void</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #00a000;">batchWriteItems</span><span style="background-color: #f8f8f8; color: #666666;">(</span><span style="background-color: #f8f8f8;">DynamoDB dynamoDB</span><span style="background-color: #f8f8f8; color: #666666;">,</span><span style="background-color: #f8f8f8;"> </span></pre>
<pre style="line-height: 125%; margin: 0px;"><span style="background-color: #f8f8f8;"> TableWriteItems userTableWriteItems<span style="color: #666666;">)</span> <span style="color: #666666;">{</span>
BatchWriteItemOutcome outcome <span style="color: #666666;">=</span> </span><span style="background-color: yellow;">dynamoDB<span style="color: #666666;">.</span><span style="color: #bb4444;">batchWriteItem</span><span style="color: #666666;">(</span>userTableWriteItems<span style="color: #666666;">);</span></span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #008800; font-style: italic;">// Check for unprocessed keys which could happen if you exceed</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #008800; font-style: italic;">// provisioned throughput</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #00bb00; font-weight: bold;">int</span><span style="background-color: #f8f8f8;"> retryAttempt </span><span style="background-color: #f8f8f8; color: #666666;">=</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #666666;">0;</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #aa22ff; font-weight: bold;">do</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #666666;">{</span><span style="background-color: #f8f8f8;">
Map</span><span style="background-color: #f8f8f8; color: #666666;"><</span><span style="background-color: #f8f8f8;">String</span><span style="background-color: #f8f8f8; color: #666666;">,</span><span style="background-color: #f8f8f8;"> List</span><span style="background-color: #f8f8f8; color: #666666;"><</span><span style="background-color: #f8f8f8;">WriteRequest</span><span style="background-color: #f8f8f8; color: #666666;">>></span><span style="background-color: #f8f8f8;"> unprocessedItems </span><span style="background-color: #f8f8f8; color: #666666;">=</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: yellow;">outcome<span style="color: #666666;">.</span><span style="color: #bb4444;">getUnprocessedItems</span><span style="color: #666666;">();</span></span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #aa22ff; font-weight: bold;">if</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #666666;">(</span><span style="background-color: #f8f8f8;">outcome</span><span style="background-color: #f8f8f8; color: #666666;">.</span><span style="background-color: #f8f8f8; color: #bb4444;">getUnprocessedItems</span><span style="background-color: #f8f8f8; color: #666666;">().</span><span style="background-color: #f8f8f8; color: #bb4444;">size</span><span style="background-color: #f8f8f8; color: #666666;">()</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #666666;">==</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #666666;">0)</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #666666;">{</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #008800; font-style: italic;">//all is well, move on</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #666666;">}</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #aa22ff; font-weight: bold;">else</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #666666;">{</span><span style="background-color: #f8f8f8;">
System</span><span style="background-color: #f8f8f8; color: #666666;">.</span><span style="background-color: #f8f8f8; color: #bb4444;">out</span><span style="background-color: #f8f8f8; color: #666666;">.</span><span style="background-color: #f8f8f8; color: #bb4444;">println</span><span style="background-color: #f8f8f8; color: #666666;">(</span><span style="background-color: #f8f8f8; color: #bb4444;">"Retrieving "</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #666666;">+</span><span style="background-color: #f8f8f8;"> outcome</span><span style="background-color: #f8f8f8; color: #666666;">.</span><span style="background-color: #f8f8f8; color: #bb4444;">getUnprocessedItems</span><span style="background-color: #f8f8f8; color: #666666;">().</span><span style="background-color: #f8f8f8; color: #bb4444;">size</span><span style="background-color: #f8f8f8; color: #666666;">()</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #666666;">+</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #bb4444;">" unprocessed items"</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #666666;">+</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #bb4444;">" Attempt No "</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #666666;">+</span><span style="background-color: #f8f8f8;"> retryAttempt</span><span style="background-color: #f8f8f8; color: #666666;">);</span><span style="background-color: #f8f8f8;">
outcome </span><span style="background-color: #f8f8f8; color: #666666;">=</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: yellow;">dynamoDB<span style="color: #666666;">.</span><span style="color: #bb4444;">batchWriteItemUnprocessed</span><span style="color: #666666;">(</span>unprocessedItems<span style="color: #666666;">);</span></span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #666666;">}</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #666666;">}</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #aa22ff; font-weight: bold;">while</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #666666;">(</span><span style="background-color: #f8f8f8;">outcome</span><span style="background-color: #f8f8f8; color: #666666;">.</span><span style="background-color: #f8f8f8; color: #bb4444;">getUnprocessedItems</span><span style="background-color: #f8f8f8; color: #666666;">().</span><span style="background-color: #f8f8f8; color: #bb4444;">size</span><span style="background-color: #f8f8f8; color: #666666;">()</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #666666;">></span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #666666;">0);</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #666666;">}</span><span style="background-color: #f8f8f8;">
</span></pre>
</div>
<br />
The code does not use DAX, it directly uses a DynamoDb client to write to the Users table. The next step would be to setup a DAX cluster for use with our table.<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgdULu9pe_3aCHuRpfj3_ezHhTRxFkeuFz0eQi9q43DL1VmLsokCM7LpgJ4mRycaN4x_54KXq7x2XNDRUZ7vRAv_eFN4FbIjCyVoNFUWXxgL5TjYpKYV2foGZSKETCur0R3bfqKrgwgaBc/s1600/DaxSetup.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="694" data-original-width="656" height="400" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgdULu9pe_3aCHuRpfj3_ezHhTRxFkeuFz0eQi9q43DL1VmLsokCM7LpgJ4mRycaN4x_54KXq7x2XNDRUZ7vRAv_eFN4FbIjCyVoNFUWXxgL5TjYpKYV2foGZSKETCur0R3bfqKrgwgaBc/s400/DaxSetup.png" width="377" /></a></div>
<b><br /></b>
<b>Step 1: Allow DAX to interact with DynamoDb</b><br />
As we read earlier, DAX intercepts all DynamoDb calls, and executing the necessary ones on Dynamo. To do this execution, DAX needs the requisite permissions.<br />
<div style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="line-height: 125%; margin: 0px;"><span style="background-color: #f8f8f8;">Before you can create an Amazon DynamoDB Accelerator (DAX) cluster, you must
create a </span><span style="background-color: yellow;">service role</span><span style="background-color: #f8f8f8;"> for it. A service role is an AWS Identity and Access
Management (IAM) role that authorizes an AWS service to act on your behalf.
The service role allows DAX to access your DynamoDB tables as if you were
accessing those tables yourself.</span></pre>
</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgWEgezQHp-kDwIMUyOJwjOvD3bcStlR7fClPPn_TsmoT-iap2jVZL-7CRNpJLUl5uCkHu6uVUoAtH1yio9blfUAJ6xg87FxXbxATPAn61eZEEzUDhhMsiXkopYaHq20mTZM3d1qZ3XQMM/s1600/DaxServiceRole.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="536" data-original-width="481" height="400" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgWEgezQHp-kDwIMUyOJwjOvD3bcStlR7fClPPn_TsmoT-iap2jVZL-7CRNpJLUl5uCkHu6uVUoAtH1yio9blfUAJ6xg87FxXbxATPAn61eZEEzUDhhMsiXkopYaHq20mTZM3d1qZ3XQMM/s400/DaxServiceRole.png" width="358" /></a></div>
<b>Step 2: Configure subnet to allow DAX (which exists within VPC) to connect with Dynamo Db.</b><br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjx9nJ7h8i3iqcIWqRHOnyfTqNI9eZ0SUWBpr91rLw8qzIAEgEAoVJnqUfRDoiTfs5ZaycyY4Gm9WhCeduz7SPoo13zmndQ69UygRN4ikNfob8oXI9BntJCz5kdJlRowbXHRFuKxYOvJ-s/s1600/DaxSubnet.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="697" data-original-width="636" height="640" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjx9nJ7h8i3iqcIWqRHOnyfTqNI9eZ0SUWBpr91rLw8qzIAEgEAoVJnqUfRDoiTfs5ZaycyY4Gm9WhCeduz7SPoo13zmndQ69UygRN4ikNfob8oXI9BntJCz5kdJlRowbXHRFuKxYOvJ-s/s640/DaxSubnet.png" width="580" /></a></div>
<br />
<br />
The next step was to test the DAX cluster.<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEglKiaONSy_qCyxbkHcJqtjiNEtsxy2kJWF5M_J3coZoXIBQ9psKTmTrc_cCc32v9eKGTvLWnGXaRzMAazkMBYVdkgQZbJVREPcUC02NisXijdkdn6ozh3ExgvmP2sdgMCfAVAHK8XY1HA/s1600/DaxCluster.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="547" data-original-width="563" height="619" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEglKiaONSy_qCyxbkHcJqtjiNEtsxy2kJWF5M_J3coZoXIBQ9psKTmTrc_cCc32v9eKGTvLWnGXaRzMAazkMBYVdkgQZbJVREPcUC02NisXijdkdn6ozh3ExgvmP2sdgMCfAVAHK8XY1HA/s640/DaxCluster.png" width="640" /></a></div>
<br />
To perform my tests, I decided to setup a simple Lambda that uses a DAX client to access the table.<br />
<!-- HTML generated using hilite.me --><br />
<div style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="line-height: 125%; margin: 0px;"><span style="background-color: #f8f8f8; color: #aa22ff; font-weight: bold;">public</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #aa22ff; font-weight: bold;">class</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: blue;">DAXTester</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #aa22ff; font-weight: bold;">implements</span><span style="background-color: #f8f8f8;"> RequestHandler</span><span style="background-color: #f8f8f8; color: #666666;"><</span><span style="background-color: #f8f8f8;">Void</span><span style="background-color: #f8f8f8; color: #666666;">,</span><span style="background-color: #f8f8f8;"> Void</span><span style="background-color: #f8f8f8; color: #666666;">></span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #666666;">{</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #aa22ff; font-weight: bold;">private</span><span style="background-color: #f8f8f8;"> DynamoDB daxClient</span><span style="background-color: #f8f8f8; color: #666666;">;</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #aa22ff; font-weight: bold;">private</span><span style="background-color: #f8f8f8;"> DynamoDB dbClient</span><span style="background-color: #f8f8f8; color: #666666;">;</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #aa22ff; font-weight: bold;">public</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #00a000;">DAXTester</span><span style="background-color: #f8f8f8; color: #666666;">()</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #666666;">{</span><span style="background-color: #f8f8f8;">
AWSCredentials awsCredentials </span><span style="background-color: #f8f8f8; color: #666666;">=</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #aa22ff; font-weight: bold;">new</span><span style="background-color: #f8f8f8;"> BasicAWSCredentials</span><span style="background-color: #f8f8f8; color: #666666;">(</span><span style="background-color: #f8f8f8; color: #bb4444;">"AccessKey"</span><span style="background-color: #f8f8f8; color: #666666;">,</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #bb4444;">"SecretKey"</span><span style="background-color: #f8f8f8; color: #666666;">);</span><span style="background-color: #f8f8f8;">
AWSCredentialsProvider awsCredentialsProvider </span><span style="background-color: #f8f8f8; color: #666666;">=</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #aa22ff; font-weight: bold;">new</span><span style="background-color: #f8f8f8;"> AWSStaticCredentialsProvider</span><span style="background-color: #f8f8f8; color: #666666;">(</span><span style="background-color: #f8f8f8;">awsCredentials</span><span style="background-color: #f8f8f8; color: #666666;">);</span><span style="background-color: #f8f8f8;">
String daxEndpointUrl </span><span style="background-color: #f8f8f8; color: #666666;">=</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #bb4444;">"users-dax-cluster.xr5tql.clustercfg.dax.use1.cache.amazonaws.com:8111"</span><span style="background-color: #f8f8f8; color: #666666;">;</span><span style="background-color: #f8f8f8;">
AmazonDaxClientBuilder daxClientBuilder </span><span style="background-color: #f8f8f8; color: #666666;">=</span><span style="background-color: #f8f8f8;"> AmazonDaxClientBuilder</span><span style="background-color: #f8f8f8; color: #666666;">.</span><span style="background-color: #f8f8f8; color: #bb4444;">standard</span><span style="background-color: #f8f8f8; color: #666666;">();</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: yellow;"> daxClientBuilder<span style="color: #666666;">.</span><span style="color: #bb4444;">withRegion</span><span style="color: #666666;">(</span><span style="color: #bb4444;">"us-east-1"</span><span style="color: #666666;">)</span>
<span style="color: #666666;">.</span><span style="color: #bb4444;">withCredentials</span><span style="color: #666666;">(</span>awsCredentialsProvider<span style="color: #666666;">)</span>
<span style="color: #666666;">.</span><span style="color: #bb4444;">withEndpointConfiguration</span><span style="color: #666666;">(</span>daxEndpointUrl<span style="color: #666666;">);</span>
daxClient <span style="color: #666666;">=</span> <span style="color: #aa22ff; font-weight: bold;">new</span> DynamoDB<span style="color: #666666;">(</span>daxClientBuilder<span style="color: #666666;">.</span><span style="color: #bb4444;">build</span><span style="color: #666666;">());</span></span><span style="background-color: #f8f8f8;">
dbClient </span><span style="background-color: #f8f8f8; color: #666666;">=</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #aa22ff; font-weight: bold;">new</span><span style="background-color: #f8f8f8;"> DynamoDB</span><span style="background-color: #f8f8f8; color: #666666;">(</span><span style="background-color: #f8f8f8;">AmazonDynamoDBClientBuilder</span><span style="background-color: #f8f8f8; color: #666666;">.</span><span style="background-color: #f8f8f8; color: #bb4444;">standard</span><span style="background-color: #f8f8f8; color: #666666;">()</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #666666;">.</span><span style="background-color: #f8f8f8; color: #bb4444;">withRegion</span><span style="background-color: #f8f8f8; color: #666666;">(</span><span style="background-color: #f8f8f8; color: #bb4444;">"us-east-1"</span><span style="background-color: #f8f8f8; color: #666666;">)</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #666666;">.</span><span style="background-color: #f8f8f8; color: #bb4444;">withCredentials</span><span style="background-color: #f8f8f8; color: #666666;">(</span><span style="background-color: #f8f8f8;">awsCredentialsProvider</span><span style="background-color: #f8f8f8; color: #666666;">).</span><span style="background-color: #f8f8f8; color: #bb4444;">build</span><span style="background-color: #f8f8f8; color: #666666;">());</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #666666;">}</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #aa22ff;">@Override</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #aa22ff; font-weight: bold;">public</span><span style="background-color: #f8f8f8;"> Void </span><span style="background-color: #f8f8f8; color: #00a000;">handleRequest</span><span style="background-color: #f8f8f8; color: #666666;">(</span><span style="background-color: #f8f8f8;">Void input</span><span style="background-color: #f8f8f8; color: #666666;">,</span><span style="background-color: #f8f8f8;"> Context context</span><span style="background-color: #f8f8f8; color: #666666;">)</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #666666;">{</span><span style="background-color: #f8f8f8;">
LambdaLogger lambdaLogger </span><span style="background-color: #f8f8f8; color: #666666;">=</span><span style="background-color: #f8f8f8;"> context</span><span style="background-color: #f8f8f8; color: #666666;">.</span><span style="background-color: #f8f8f8; color: #bb4444;">getLogger</span><span style="background-color: #f8f8f8; color: #666666;">();</span><span style="background-color: #f8f8f8;">
String usersTable </span><span style="background-color: #f8f8f8; color: #666666;">=</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #bb4444;">"Users"</span><span style="background-color: #f8f8f8; color: #666666;">;</span><span style="background-color: #f8f8f8;">
Table dbTable </span><span style="background-color: #f8f8f8; color: #666666;">=</span><span style="background-color: #f8f8f8;"> dbClient</span><span style="background-color: #f8f8f8; color: #666666;">.</span><span style="background-color: #f8f8f8; color: #bb4444;">getTable</span><span style="background-color: #f8f8f8; color: #666666;">(</span><span style="background-color: #f8f8f8;">usersTable</span><span style="background-color: #f8f8f8; color: #666666;">);</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #008800; font-style: italic;">//Test 1 - Get without DAX</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #008800; font-style: italic;">//Lambda needs</span><span style="background-color: #f8f8f8;">
lambdaLogger</span><span style="background-color: #f8f8f8; color: #666666;">.</span><span style="background-color: #f8f8f8; color: #bb4444;">log</span><span style="background-color: #f8f8f8; color: #666666;">(</span><span style="background-color: #f8f8f8; color: #bb4444;">"Tests for GetItem"</span><span style="background-color: #f8f8f8; color: #666666;">);</span><span style="background-color: #f8f8f8;">
lambdaLogger</span><span style="background-color: #f8f8f8; color: #666666;">.</span><span style="background-color: #f8f8f8; color: #bb4444;">log</span><span style="background-color: #f8f8f8; color: #666666;">(</span><span style="background-color: #f8f8f8; color: #bb4444;">"Dynamo Tests"</span><span style="background-color: #f8f8f8; color: #666666;">);</span><span style="background-color: #f8f8f8;">
computeGetItemDuration</span><span style="background-color: #f8f8f8; color: #666666;">(</span><span style="background-color: #f8f8f8;">lambdaLogger</span><span style="background-color: #f8f8f8; color: #666666;">,</span><span style="background-color: #f8f8f8;"> dbTable</span><span style="background-color: #f8f8f8; color: #666666;">);</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #008800; font-style: italic;">//Test 2 - Get with DAX - no entry in cache</span><span style="background-color: #f8f8f8;">
Table daxTable </span><span style="background-color: #f8f8f8; color: #666666;">=</span><span style="background-color: #f8f8f8;"> daxClient</span><span style="background-color: #f8f8f8; color: #666666;">.</span><span style="background-color: #f8f8f8; color: #bb4444;">getTable</span><span style="background-color: #f8f8f8; color: #666666;">(</span><span style="background-color: #f8f8f8;">usersTable</span><span style="background-color: #f8f8f8; color: #666666;">);</span><span style="background-color: #f8f8f8;">
lambdaLogger</span><span style="background-color: #f8f8f8; color: #666666;">.</span><span style="background-color: #f8f8f8; color: #bb4444;">log</span><span style="background-color: #f8f8f8; color: #666666;">(</span><span style="background-color: #f8f8f8; color: #bb4444;">"DAX Tests - not in Cache"</span><span style="background-color: #f8f8f8; color: #666666;">);</span><span style="background-color: #f8f8f8;">
computeGetItemDuration</span><span style="background-color: #f8f8f8; color: #666666;">(</span><span style="background-color: #f8f8f8;">lambdaLogger</span><span style="background-color: #f8f8f8; color: #666666;">,</span><span style="background-color: #f8f8f8;"> daxTable</span><span style="background-color: #f8f8f8; color: #666666;">);</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #008800; font-style: italic;">//Test 3 - Get with DAX - entry in cache</span><span style="background-color: #f8f8f8;">
lambdaLogger</span><span style="background-color: #f8f8f8; color: #666666;">.</span><span style="background-color: #f8f8f8; color: #bb4444;">log</span><span style="background-color: #f8f8f8; color: #666666;">(</span><span style="background-color: #f8f8f8; color: #bb4444;">"DAX Tests - present in Cache"</span><span style="background-color: #f8f8f8; color: #666666;">);</span><span style="background-color: #f8f8f8;">
computeGetItemDuration</span><span style="background-color: #f8f8f8; color: #666666;">(</span><span style="background-color: #f8f8f8;">lambdaLogger</span><span style="background-color: #f8f8f8; color: #666666;">,</span><span style="background-color: #f8f8f8;"> daxTable</span><span style="background-color: #f8f8f8; color: #666666;">);</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #aa22ff; font-weight: bold;">return</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #aa22ff; font-weight: bold;">null</span><span style="background-color: #f8f8f8; color: #666666;">;</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #666666;">}</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #aa22ff; font-weight: bold;">private</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #00bb00; font-weight: bold;">void</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #00a000;">computeGetItemDuration</span><span style="background-color: #f8f8f8; color: #666666;">(</span><span style="background-color: #f8f8f8;">LambdaLogger lambdaLogger</span><span style="background-color: #f8f8f8; color: #666666;">,</span><span style="background-color: #f8f8f8;"> Table table</span><span style="background-color: #f8f8f8; color: #666666;">)</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #666666;">{</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #00bb00; font-weight: bold;">double</span><span style="background-color: #f8f8f8;"> duration </span><span style="background-color: #f8f8f8; color: #666666;">=</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #666666;">0;</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #00bb00; font-weight: bold;">int</span><span style="background-color: #f8f8f8;"> noOfTests </span><span style="background-color: #f8f8f8; color: #666666;">=</span><span style="background-color: #f8f8f8;"> 5</span><span style="background-color: #f8f8f8; color: #666666;">00;</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #aa22ff; font-weight: bold;">for</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #666666;">(</span><span style="background-color: #f8f8f8; color: #00bb00; font-weight: bold;">int</span><span style="background-color: #f8f8f8;"> i </span><span style="background-color: #f8f8f8; color: #666666;">=</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #666666;">0;</span><span style="background-color: #f8f8f8;"> i </span><span style="background-color: #f8f8f8; color: #666666;"><</span><span style="background-color: #f8f8f8;"> noOfTests</span><span style="background-color: #f8f8f8; color: #666666;">;</span><span style="background-color: #f8f8f8;"> i</span><span style="background-color: #f8f8f8; color: #666666;">++)</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #666666;">{</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #00bb00; font-weight: bold;">long</span><span style="background-color: #f8f8f8;"> startTime </span><span style="background-color: #f8f8f8; color: #666666;">=</span><span style="background-color: #f8f8f8;"> System</span><span style="background-color: #f8f8f8; color: #666666;">.</span><span style="background-color: #f8f8f8; color: #bb4444;">nanoTime</span><span style="background-color: #f8f8f8; color: #666666;">();</span><span style="background-color: #f8f8f8;">
GetItemOutcome outcome </span><span style="background-color: #f8f8f8; color: #666666;">=</span><span style="background-color: #f8f8f8;"> table</span><span style="background-color: #f8f8f8; color: #666666;">.</span><span style="background-color: #f8f8f8; color: #bb4444;">getItemOutcome</span><span style="background-color: #f8f8f8; color: #666666;">(</span><span style="background-color: #f8f8f8; color: #bb4444;">"userId"</span><span style="background-color: #f8f8f8; color: #666666;">,</span><span style="background-color: #f8f8f8;"> Integer</span><span style="background-color: #f8f8f8; color: #666666;">.</span><span style="background-color: #f8f8f8; color: #bb4444;">valueOf</span><span style="background-color: #f8f8f8; color: #666666;">(</span><span style="background-color: #f8f8f8;">i</span><span style="background-color: #f8f8f8; color: #666666;">));</span><span style="background-color: #f8f8f8;">
outcome</span><span style="background-color: #f8f8f8; color: #666666;">.</span><span style="background-color: #f8f8f8; color: #bb4444;">getItem</span><span style="background-color: #f8f8f8; color: #666666;">().</span><span style="background-color: #f8f8f8; color: #bb4444;">get</span><span style="background-color: #f8f8f8; color: #666666;">(</span><span style="background-color: #f8f8f8; color: #bb4444;">"name"</span><span style="background-color: #f8f8f8; color: #666666;">);</span><span style="background-color: #f8f8f8;">
duration </span><span style="background-color: #f8f8f8; color: #666666;">+=</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #666666;">(</span><span style="background-color: #f8f8f8;">System</span><span style="background-color: #f8f8f8; color: #666666;">.</span><span style="background-color: #f8f8f8; color: #bb4444;">nanoTime</span><span style="background-color: #f8f8f8; color: #666666;">()</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #666666;">-</span><span style="background-color: #f8f8f8;"> startTime</span><span style="background-color: #f8f8f8; color: #666666;">);</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #666666;">}</span><span style="background-color: #f8f8f8;">
DecimalFormat df </span><span style="background-color: #f8f8f8; color: #666666;">=</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #aa22ff; font-weight: bold;">new</span><span style="background-color: #f8f8f8;"> DecimalFormat</span><span style="background-color: #f8f8f8; color: #666666;">(</span><span style="background-color: #f8f8f8; color: #bb4444;">"#.##"</span><span style="background-color: #f8f8f8; color: #666666;">);</span><span style="background-color: #f8f8f8;">
String formattedNumber </span><span style="background-color: #f8f8f8; color: #666666;">=</span><span style="background-color: #f8f8f8;"> df</span><span style="background-color: #f8f8f8; color: #666666;">.</span><span style="background-color: #f8f8f8; color: #bb4444;">format</span><span style="background-color: #f8f8f8; color: #666666;">((</span><span style="background-color: #f8f8f8;">duration </span><span style="background-color: #f8f8f8; color: #666666;">/</span><span style="background-color: #f8f8f8;"> noOfTests</span><span style="background-color: #f8f8f8; color: #666666;">));</span><span style="background-color: #f8f8f8;">
lambdaLogger</span><span style="background-color: #f8f8f8; color: #666666;">.</span><span style="background-color: #f8f8f8; color: #bb4444;">log</span><span style="background-color: #f8f8f8; color: #666666;">(</span><span style="background-color: #f8f8f8; color: #bb4444;">"Average FetchTime = "</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #666666;">+</span><span style="background-color: #f8f8f8;"> formattedNumber </span><span style="background-color: #f8f8f8; color: #666666;">+</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #bb4444;">" nano seconds"</span><span style="background-color: #f8f8f8; color: #666666;">);</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #666666;">}</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #666666;">}</span><span style="background-color: #f8f8f8;">
</span></pre>
</div>
The code is pretty straight forward. It creates a DAX client and a DynamoDb client. Code measures the latency for a GetItem call across three scenarios:<br />
<ol style="text-align: left;">
<li>No Cache - directly fetch from Dynamo</li>
<li>Cache Miss - use DAX, but entry not found resulting in DAX having to call Dynamo</li>
<li>Cache Hit - entry found in DAX, so no Dynamo hit</li>
</ol>
I had some problems building the DAX Maven dependency for the project (dependency issues related to AWS and Dynamo packages). So I downloaded the amazon-dax-client jar and added that as a direct dependency. (I used the addjars-maven-plugin for this.)<br />
<br />
Once Lambda was setup, I ran into errors in executing my function:<br />
<div style="background: #f8f8f8; border-width: 0.1em 0.1em 0.1em 0.8em; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="line-height: 125%; margin: 0;">May <span style="color: #666666;">30,</span> <span style="color: #666666;">2020</span> <span style="color: #666666;">8:07:04</span> PM com<span style="color: #666666;">.</span><span style="color: #bb4444;">amazon</span><span style="color: #666666;">.</span><span style="color: #bb4444;">dax</span><span style="color: #666666;">.</span><span style="color: #bb4444;">client</span><span style="color: #666666;">.</span><span style="color: #bb4444;">SocketTubePool</span> setLastError
<span style="color: #a0a000;">WARNING:</span> error connecting to <span style="color: #666666;">/172.31.82.151:8111</span> java<span style="color: #666666;">.</span><span style="color: #bb4444;">net</span><span style="color: #666666;">.</span><span style="color: #bb4444;">SocketTimeoutException</span><span style="color: #666666;">:</span></pre>
<pre style="line-height: 125%; margin: 0;"> connect timed out java<span style="color: #666666;">.</span><span style="color: #bb4444;">net</span><span style="color: #666666;">.</span><span style="color: #bb4444;">SocketTimeoutException</span><span style="color: #666666;">:</span> connect timed out
</pre>
</div>
<br />
I faced the same problem that I faced with Elasticache - my lambda is unable to connect to DAX.<br />
As DAX is setup in VPC, Lambda which runs outside this VPC cannot access it. To fix this we need to make Lambda VPC-aware. (Similar to what we did <a href="https://learningviacode.blogspot.com/2020/05/playing-with-elasticache.html" target="_blank">here</a>)<br />
<br />
The VPC info in Lambda allows it to communicate with DAX. But now we have another problem. DynamoDb is outside this VPC. Our Lambda calls cannot be made to Dynamo Db anymore.<br />
We faced a similar problem where Lambda within VPC could not communicate with CloudWatch (Read <a href="https://learningviacode.blogspot.com/2020/05/playing-with-elasticache-2.html" target="_blank">here</a>). To fix this, we setup an Interface Endpoint that routed Lambda calls to CloudWatch.<br />
With Dynamo Db (and S3) we need to use what is known as a Gateway Endpoint.<br />
<!-- HTML generated using hilite.me --><br />
<div style="background: #f8f8f8; border-width: 0.1em 0.1em 0.1em 0.8em; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="line-height: 125%; margin: 0;">A gateway endpoint is a gateway that you specify as a target for a route in your route table for traffic destined to a supported AWS service. The following AWS services are supported:
* Amazon S3
* DynamoDB
</pre>
</div>
Accordingly I setup an endpoint that will route Dynamo Db calls coming out of my <a href="https://docs.aws.amazon.com/vpc/latest/userguide/vpc-endpoints-access.html#vpc-endpoint-policies" target="_blank">VPC to Dynamo DB Gateway</a>.<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgIW577Y-bPWB7448A70AkR49w6h0wBHHqvHfddFSqY90j9LSU4azXmrI5FFrL3_zlFMMwYMIN3cZU1wrMkphwXaNBYuLIe_PtuA9D1QJRYKRUFaO_qusVJVqT7q1B6nsJXIt-Jtn-IRCU/s1600/GatewayEndpoint.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="692" data-original-width="825" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgIW577Y-bPWB7448A70AkR49w6h0wBHHqvHfddFSqY90j9LSU4azXmrI5FFrL3_zlFMMwYMIN3cZU1wrMkphwXaNBYuLIe_PtuA9D1QJRYKRUFaO_qusVJVqT7q1B6nsJXIt-Jtn-IRCU/s1600/GatewayEndpoint.png" /></a></div>
Now the code is ready for execution.<br />
<!-- HTML generated using hilite.me --><br />
<div style="background: #f8f8f8; border-width: 0.1em 0.1em 0.1em 0.8em; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="line-height: 125%; margin: 0;">START RequestId: 86a6c562-72a4-46a4-9c87-8899c65f1fc9 Version: $LATEST
Tests for GetItem
Dynamo Tests
Average FetchTime = 35719684.39 nano seconds</pre>
<pre style="line-height: 125%; margin: 0;">DAX Tests - not in Cache
Average FetchTime = 7340349.86 nano seconds
DAX Tests - present in Cache
Average FetchTime = 1400325.51 nano seconds
END RequestId: 86a6c562-72a4-46a4-9c87-8899c65f1fc9
REPORT RequestId: 86a6c562-72a4-46a4-9c87-8899c65f1fc9 Duration: 22365.35 ms Billed Duration: 22400 ms Memory Size: 256 MB Max Memory Used: 125 MB Init Duration: 1830.58 ms
</pre>
</div>
This is weird. Dynamo calls were slower when directly accessed than when accessed by DAX through Dynamo. (The VPC outgoing calls appear to add time)<br />
<br />
The DAX Cache lookup however is faster that Dynamo Read.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi9GSXBbaBZH5jqPGq9kd-SE06U24UwLVX7mdB8bKsl753f_4TbxCAnlulLTf_N3dohdvCoWPg6joEkS6yEFZcrXgl8DNa8UIHluRLklguJh5JFrvBHrWSEhNmDm6D00O4PuI-8q54PEuM/s1600/DAXGetItemStats.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="282" data-original-width="856" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi9GSXBbaBZH5jqPGq9kd-SE06U24UwLVX7mdB8bKsl753f_4TbxCAnlulLTf_N3dohdvCoWPg6joEkS6yEFZcrXgl8DNa8UIHluRLklguJh5JFrvBHrWSEhNmDm6D00O4PuI-8q54PEuM/s1600/DAXGetItemStats.png" /></a></div>
<br /></div>
Robin Varghesehttp://www.blogger.com/profile/00841917517097216645noreply@blogger.com0tag:blogger.com,1999:blog-6262128735818493674.post-89001386954074712322020-05-26T00:29:00.003+05:302020-05-26T00:29:58.351+05:30Deploying API Gateway<div dir="ltr" style="text-align: left;" trbidi="on">
In the last 2 posts, I have been working on setting up API access to my Dynamo Table through API Gateway. The next step would be to deploy the APIs:<br />
<a name='more'></a><br />
I decided to start with the GET method:<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj03qRehDfzEFkN6eKjQbAZBsCh_l4LMBPP1mogxr0ecXKKzoOSxRIT-qTWPkqgvoDdRpbBUKSNQ9GAAF-Ahn3ekfDr-OIhbUXxAnx-RHjEXO4Qdw2xYtduLwOOEi5LZYWCdGMq9WO4LJQ/s1600/BetaDeployStage.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1246" data-original-width="873" height="640" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj03qRehDfzEFkN6eKjQbAZBsCh_l4LMBPP1mogxr0ecXKKzoOSxRIT-qTWPkqgvoDdRpbBUKSNQ9GAAF-Ahn3ekfDr-OIhbUXxAnx-RHjEXO4Qdw2xYtduLwOOEi5LZYWCdGMq9WO4LJQ/s640/BetaDeployStage.png" width="448" /></a></div>
API Gateway provides some real cool features out of the box:
<br />
<br />
<ol style="text-align: left;">
<li><b>Cache Settings: </b>APIG includes an api level cache with configurable cache size, cache ttl.</li>
<li><b>Default Method Throttling:</b> APIG allows us to apply throttling to the API, defining the supported TPS and allowable bursts rate. APIG supports API level throttling (across all clients) as well as client specific throttling.</li>
<li><b>Web Application Firewall (WAF)</b>: Firewall settings can be setup for the endpoint allowing us to block SQL Injection, XSS attacks, catch bad clients etc. Refer this <a href="https://docs.aws.amazon.com/apigateway/latest/developerguide/apigateway-control-access-aws-waf.html" target="_blank">link</a>.</li>
<li><b>CloudWatch logs:</b> Cloud Watch logs can be enabled for APIG as well as cloud watch metrics - API call latency, integration latency, errors etc.</li>
<li><b>Client Development:</b> API can be exported using Swagger, clients can be generated in various languages</li>
<li><b>Testing</b>: Canary support is available for safe deployments</li>
</ol>
</div>
Robin Varghesehttp://www.blogger.com/profile/00841917517097216645noreply@blogger.com1tag:blogger.com,1999:blog-6262128735818493674.post-8219430353481888212020-05-25T23:51:00.003+05:302020-05-25T23:51:51.724+05:30Dynamo Db APIs through API Gateway<div dir="ltr" style="text-align: left;" trbidi="on">
In the last post, we setup a <a href="https://learningviacode.blogspot.com/2020/05/rest-dynamo-api-gateway.html" target="_blank">GET method for my table</a> through API Gateway. I wanted to go ahead and setup the other methods - DELETE, PUT, POST.<br />
<a name='more'></a>Ill start with the PUT method. PUT call on "/persons" endpoint is used to add a record in my table<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgB4116AeNy3NM62aYizrTekqPuWseK1sxW1vTZRpzt25NKSt2l5CfTQU6YOE3Cpo-zomaL4nRZSHy7JaG3wWmZKpm-wdtdggSccAOgYT5CbeO1tNRTgM0b4RoYwBvIYEydTFVhdELG99A/s1600/DbPersonsRec.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="554" data-original-width="920" height="240" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgB4116AeNy3NM62aYizrTekqPuWseK1sxW1vTZRpzt25NKSt2l5CfTQU6YOE3Cpo-zomaL4nRZSHy7JaG3wWmZKpm-wdtdggSccAOgYT5CbeO1tNRTgM0b4RoYwBvIYEydTFVhdELG99A/s400/DbPersonsRec.png" width="400" /></a></div>
The put Method is to be transformed into a PutItem API call on DynamoDb.<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg9dkJ4yun2NLANbgsNkew4I6Nk6mHzF60JtBOnXbqKvd-MCUwx5KJHPu8WuRaL5HmC9MfFa1Br9Y5WIIBMYZfNe0A2hNTLhq-3Pcy5S-mBEfpN_4qV0QLhm90M1rVgfNHWVbEs5QOvPmw/s1600/DeleteMethod.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="954" data-original-width="948" height="640" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg9dkJ4yun2NLANbgsNkew4I6Nk6mHzF60JtBOnXbqKvd-MCUwx5KJHPu8WuRaL5HmC9MfFa1Br9Y5WIIBMYZfNe0A2hNTLhq-3Pcy5S-mBEfpN_4qV0QLhm90M1rVgfNHWVbEs5QOvPmw/s640/DeleteMethod.png" width="635" /></a></div>
<br />
Accordingly I setup my mapping template in the IntegrationRequest Layer:
<br />
<div style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="line-height: 125%; margin: 0px;"><span style="background-color: #f8f8f8;">{
</span><span style="background-color: #f8f8f8; color: green; font-weight: bold;">"TableName"</span><span style="background-color: #f8f8f8;">: </span><span style="background-color: #f8f8f8; color: #bb4444;">"Person"</span><span style="background-color: #f8f8f8;">,
</span><span style="background-color: #f8f8f8; color: green; font-weight: bold;">"Item"</span><span style="background-color: #f8f8f8;">: {
</span><span style="background-color: #f8f8f8; color: green; font-weight: bold;">"pId"</span><span style="background-color: #f8f8f8;">: {
</span><span style="background-color: #f8f8f8; color: green; font-weight: bold;">"S"</span><span style="background-color: #f8f8f8;">: </span><span style="color: #bb4444;"><span style="background-color: #f8f8f8;">"</span><span style="background-color: yellow;">$context.requestId"</span></span><span style="background-color: #f8f8f8;">
},
</span><span style="background-color: #f8f8f8; color: green; font-weight: bold;">"pName"</span><span style="background-color: #f8f8f8;">: {
</span><span style="background-color: #f8f8f8; color: green; font-weight: bold;">"S"</span><span style="background-color: #f8f8f8;">: </span><span style="background-color: #f8f8f8; color: #bb4444;">"$util.escapeJavaScript($input.params('name'))"</span><span style="background-color: #f8f8f8;">
},
</span><span style="background-color: #f8f8f8; color: green; font-weight: bold;">"age"</span><span style="background-color: #f8f8f8;">: {
</span><span style="background-color: #f8f8f8; color: green; font-weight: bold;">"N"</span><span style="background-color: #f8f8f8;">: </span><span style="background-color: #f8f8f8; color: #bb4444;">"$input.params('age')"</span><span style="background-color: #f8f8f8;">
},
</span><span style="background-color: #f8f8f8; color: green; font-weight: bold;">"creationTime"</span><span style="background-color: #f8f8f8;"> : {
</span><span style="background-color: #f8f8f8; color: green; font-weight: bold;">"S"</span><span style="background-color: #f8f8f8;">:</span><span style="background-color: #f8f8f8; color: #bb4444;">"$context.requestTime"</span><span style="background-color: #f8f8f8;">
}
}
}
</span></pre>
</div>
This will create an entry using the name and age fields provided by user in request url. I needed the 'id' to be unique. I could either take the id from user and depend on the conditional check or I could use a system created unique identifier. Dynamo Db does not provide table sequences similar to Oracle. So I need to pass the unique identifier here. For this I leveraged the unique request Id generated by API Gateway. This requestId is available in the <b>context</b> object.
<br />
On testing my method the APIG logs are as below:<br />
<!-- HTML generated using hilite.me --><br />
<div style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="background-color: #f8f8f8; line-height: 125%; margin: 0px;">Execution log for request <span style="color: blue;">852a5371-b157-4a72-91f5-8e7a0317e0fe</span>
Mon May 25 17:24:05 UTC 2020 : Starting execution for request: </pre>
<pre style="background-color: #f8f8f8; line-height: 125%; margin: 0px;">852a5371-b157-4a72-91f5-8e7a0317e0fe
Mon May 25 17:24:05 UTC 2020 : HTTP Method: PUT, Resource Path: /persons
Mon May 25 17:24:05 UTC 2020 : Method request path: {}
<span style="color: blue;">Mon May 25 17:24:05 UTC 2020 : Method request query string: {name=Foo Bar, age=28}
</span>Mon May 25 17:24:05 UTC 2020 : Method request headers: {}
Mon May 25 17:24:05 UTC 2020 : Method request body before transformations:
<span style="color: blue;">Mon May 25 17:24:05 UTC 2020 : Endpoint request URI: </span></pre>
<pre style="background-color: #f8f8f8; line-height: 125%; margin: 0px;"><span style="color: blue;">https://dynamodb.us-east-1.amazonaws.com/?Action=PutItem
</span>Mon May 25 17:24:05 UTC 2020 : Endpoint request headers: {</pre>
<pre style="background-color: #f8f8f8; line-height: 125%; margin: 0px;">Authorization=******a4cf74,
X-Amz-Date=20200525T172405Z, x-amzn-apigateway-api-id=qbotcgknke,
Accept=application/json, User-Agent=AmazonAPIGateway_qbotcgknke,
X-Amz-Security-Token=IQoJb3JpZ [TRUNCATED]
Mon May 25 17:24:05 UTC 2020 : Endpoint request body after transformations: {
"TableName": "Person",
"Item": {
"pId": {
"S": "<span style="color: blue;">852a5371-b157-4a72-91f5-8e7a0317e0fe</span>"
},
"pName": {
"S": "Foo Bar"
},
"age": {
"N": "28"
},
"creationTime" : {
<span style="color: blue;">"S":"25/May/2020:17:24:05 +0000"</span>
}
}
}
Mon May 25 17:24:05 UTC 2020 : </pre>
<pre style="line-height: 125%; margin: 0px;"><span style="background-color: #f8f8f8; color: blue;">Sending request to https://dynamodb.us-east-1.amazonaws.com/?Action=PutItem</span><span style="background-color: #f8f8f8;">
Mon May 25 17:24:05 UTC 2020 : Received response. Status: 200, Integration latency: 23 ms
Mon May 25 17:24:05 UTC 2020 : Endpoint response headers: {Server=Server,
Date=Mon, 25 May 2020 17:24:05 GMT, Content-Type=application/x-amz-json-1.0,
Content-Length=2, Connection=keep-alive,
x-amzn-RequestId=S9HLQJMK9I85EPHO821DHSKLOJVV4KQNSO5AEMVJF66Q9ASUAAJG,
x-amz-crc32=2745614147}
Mon May 25 17:24:05 UTC 2020 : </span><span style="background-color: yellow;">Endpoint response body before transformations: {}</span><span style="background-color: #f8f8f8;">
Mon May 25 17:24:05 UTC 2020 : Method response body after transformations: {}
Mon May 25 17:24:05 UTC 2020 : </span><span style="background-color: #f8f8f8; color: blue;">Method response headers: {
X-Amzn-Trace-Id=Root=1-5ecbff35-1b8bb0370fcd46fe20912834,
Content-Type=application/json}</span><span style="background-color: #f8f8f8;">
Mon May 25 17:24:05 UTC 2020 : Successfully completed execution
Mon May 25 17:24:05 UTC 2020 : Method completed with status: 200
</span></pre>
</div>
As seen from logs, PutItem does not return a response. So I have the problem of communicating my unique identifier back in response. This does make user id with validations look to be a better method.<br />
<br />
My remaining 3 methods - GET/Update/Delete operate at id resource. We saw GET in the previous post.<br />
<!-- HTML generated using hilite.me --><br />
<div style="background: #f8f8f8; border-width: 0.1em 0.1em 0.1em 0.8em; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="line-height: 125%; margin: 0;">{
<span style="color: green; font-weight: bold;">"TableName"</span>: <span style="color: #bb4444;">"Person"</span>,
<span style="color: green; font-weight: bold;">"PrimaryKey"</span>: <span style="color: #bb4444;">"pId"</span>,
<span style="color: green; font-weight: bold;">"KeyConditionExpression"</span>: <span style="color: #bb4444;">"pId = :v1"</span>,
<span style="color: green; font-weight: bold;">"ExpressionAttributeValues"</span>: {
<span style="color: green; font-weight: bold;">":v1"</span>: {
<span style="color: green; font-weight: bold;">"S"</span>: <span style="color: #bb4444;">"$input.params('id')"</span>
}
}
}
</pre>
</div>
The mapping template will read the id from request url and use it for a Query. Since we did not want to return the dynamodb JSON, but a simpler version, I setup a (Velocity) mapping template in my IntegrationResponse layer:<br />
<div style="background: #f8f8f8; border-width: 0.1em 0.1em 0.1em 0.8em; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="line-height: 125%; margin: 0;"><span style="color: #008800;">#</span><span style="color: #00a000;">set</span>($<span style="color: darkgoldenrod;">inputRoot</span> <span style="color: #666666;">=</span> $<span style="color: darkgoldenrod;">input</span>.<span style="color: darkgoldenrod;">path</span>(<span style="color: #bb4444;">'$'</span>))
<span style="border: 1px solid #FF0000;">{</span>
<span style="color: #008800;">#</span><span style="color: #00a000;">if</span>($<span style="color: darkgoldenrod;">inputRoot</span>.<span style="color: darkgoldenrod;">Count</span> <span style="color: #666666;">>=</span> <span style="color: #666666;">1</span>)
"id": "$<span style="color: darkgoldenrod;">inputRoot</span>.<span style="color: darkgoldenrod;">Items</span>[0].pId.S",
"name": "$<span style="color: darkgoldenrod;">inputRoot</span>.<span style="color: darkgoldenrod;">Items</span>[0].pName.S",
"age": "$<span style="color: darkgoldenrod;">inputRoot</span>.<span style="color: darkgoldenrod;">Items</span>[0].age.N"
<span style="color: #008800;">#</span><span style="color: #00a000;">end</span>
}
</pre>
</div>
Next is the Delete method. This accepts only the Person identifier which the IntegrationRequest layer converts into a DynamoDb DeleteItem API call.
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg1gju58H4C5ysd1FYxoQZkd50kdrO-FzfIVVHmlNXAFI7WYfOvZ3iAtaTIMUPUNxU53gYqIhADKUiC6arrkZPWJLb9gelLTzFMXjvb37tcRlTGKMfQ96G2r8SNSSY672EVE_Vwn-9BNcU/s1600/Delete_Method.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="964" data-original-width="920" height="640" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg1gju58H4C5ysd1FYxoQZkd50kdrO-FzfIVVHmlNXAFI7WYfOvZ3iAtaTIMUPUNxU53gYqIhADKUiC6arrkZPWJLb9gelLTzFMXjvb37tcRlTGKMfQ96G2r8SNSSY672EVE_Vwn-9BNcU/s640/Delete_Method.png" width="609" /></a></div>
<br />
The MappingTemplate to convert to a DynamoDb DeleteItem call is as below:
<br />
<div style="background: #f8f8f8; border-width: 0.1em 0.1em 0.1em 0.8em; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="line-height: 125%; margin: 0;">{
<span style="color: green; font-weight: bold;">"TableName"</span>: <span style="color: #bb4444;">"Person"</span>,
<span style="color: green; font-weight: bold;">"Key"</span>: {
<span style="color: green; font-weight: bold;">"pId"</span>: {
<span style="color: green; font-weight: bold;">"S"</span>: <span style="color: #bb4444;">"$input.params('id')"</span>
}
},
<span style="color: green; font-weight: bold;">"ReturnValues"</span>: <span style="color: #bb4444;">"ALL_OLD"</span>
}
</pre>
</div>
This will return the Dynamo Db record if it exists. If we do not want dynamo db JSON to be passed through we need to setup a IntegrationResponse mapping template (similar to what we did in GET method)
I tested the method and the logs are:
<!-- HTML generated using hilite.me --><br />
<div style="background: #f8f8f8; border-width: 0.1em 0.1em 0.1em 0.8em; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="line-height: 125%; margin: 0;"><span style="color: blue;">Execution log for request 3f7e59b3-05f7-41ac-9479-a59a79ab7397</span>
Mon May 25 17:44:19 UTC 2020 : Starting execution for request: 3f7e59b3-05f7-41ac-9479-a59a79ab7397
<span style="color: blue;">Mon May 25 17:44:19 UTC 2020 : HTTP Method: DELETE, Resource Path:</span></pre>
<pre style="line-height: 125%; margin: 0;"><span style="color: blue;"> /persons/f85fc54c-0119-4dc8-98f0-4a644b074557
</span>Mon May 25 17:44:19 UTC 2020 : Method request path: {id=f85fc54c-0119-4dc8-98f0-4a644b074557}
Mon May 25 17:44:19 UTC 2020 : Method request query string: {}
Mon May 25 17:44:19 UTC 2020 : Method request headers: {}
Mon May 25 17:44:19 UTC 2020 : Method request body before transformations:
Mon May 25 17:44:19 UTC 2020 : Endpoint request URI: https://dynamodb.us-east-1.amazonaws.com/?Action=DeleteItem
Mon May 25 17:44:19 UTC 2020 : Endpoint request headers: {Authorization=****3cb1e4, X-Amz-Date=20200525T174419Z,
x-amzn-apigateway-api-id=qbotcgknke, Accept=application/json, User-Agent=AmazonAPIGateway_qbotcgknke,
X-Amz-Security-Token=IQoJb3JpZ2luX2VjED [TRUNCATED]
<span style="color: blue;">Mon May 25 17:44:19 UTC 2020 : Endpoint request body after transformations: {
"TableName": "Person",
"Key": {
"pId": {
"S": "f85fc54c-0119-4dc8-98f0-4a644b074557"
}
},
"ReturnValues": "ALL_OLD"
}</span>
<span style="color: blue;">Mon May 25 17:44:19 UTC 2020 : Sending request to </span></pre>
<pre style="line-height: 125%; margin: 0;"><span style="color: blue;">https://dynamodb.us-east-1.amazonaws.com/?Action=DeleteItem</span>
Mon May 25 17:44:19 UTC 2020 : Received response. Status: 200, Integration latency: 26 ms
Mon May 25 17:44:19 UTC 2020 : Endpoint response headers: {Server=Server, Date=Mon, 25 May 2020 17:44:18 GMT,
Content-Type=application/x-amz-json-1.0, Content-Length=158, Connection=keep-alive, x-amzn-RequestId=MSO723GT56G3QF2QDKHN9PQ8Q3VV4KQNSO5AEMVJF66Q9ASUAAJG, x-amz-crc32=4050537793}
Mon May 25 17:44:19 UTC 2020 : Endpoint response body before transformations:</pre>
<pre style="line-height: 125%; margin: 0;"> {"Attributes":{"pId":{"S":"f85fc54c-0119-4dc8-98f0-4a644b074557"},</pre>
<pre style="line-height: 125%; margin: 0;">"pName":{"S":"Foo Bar"},"creationTime":{"S":"25/May/2020:17:09:27 +0000"},</pre>
<pre style="line-height: 125%; margin: 0;">"age":{"N":"35"}}}
<span style="color: blue;">Mon May 25 17:44:19 UTC 2020 : Method response body after transformations: </span></pre>
<pre style="line-height: 125%; margin: 0;"><span style="color: blue;">{"Attributes":{"pId":{"S":"f85fc54c-0119-4dc8-98f0-4a644b074557"},</span></pre>
<pre style="line-height: 125%; margin: 0;"><span style="color: blue;">"pName":{"S":"Foo Bar"},"creationTime":{"S":"25/May/2020:17:09:27 +0000"},</span></pre>
<pre style="line-height: 125%; margin: 0;"><span style="color: blue;">"age":{"N":"35"}}}</span>
Mon May 25 17:44:19 UTC 2020 : Method response headers: {X-Amzn-Trace-Id=Root=1-5ecc03f3-cb333de7541ff8c3f8aebeb6,
Content-Type=application/json}
Mon May 25 17:44:19 UTC 2020 : Successfully completed execution
Mon May 25 17:44:19 UTC 2020 : Method completed with status: 200
</pre>
</div>
The last method is the POST method which was used to update an Item. The method uses the Id to identify the item, allowing user to provide name and age values.<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhRNtsMsOS0BXxyYbA2izXR92_pYH2WVcgTQ-NcZay1tvIBABhOtxZKKuyFS6r9m45CDv7MntzxMjfrjssH8IGjOQhDoYfJDnjyVok7ABAZJOA6vz6uPP4PdlWgoylUkmC0Bh7J2Vcvp4k/s1600/PostMethod.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="972" data-original-width="924" height="640" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhRNtsMsOS0BXxyYbA2izXR92_pYH2WVcgTQ-NcZay1tvIBABhOtxZKKuyFS6r9m45CDv7MntzxMjfrjssH8IGjOQhDoYfJDnjyVok7ABAZJOA6vz6uPP4PdlWgoylUkmC0Bh7J2Vcvp4k/s640/PostMethod.png" width="608" /></a></div>
<br />
The mapping template for Integration Request is:
<!-- HTML generated using hilite.me --><br />
<div style="background: #f8f8f8; border-width: 0.1em 0.1em 0.1em 0.8em; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="line-height: 125%; margin: 0;">{
<span style="color: green; font-weight: bold;">"TableName"</span>: <span style="color: #bb4444;">"Person"</span>,
<span style="color: green; font-weight: bold;">"Key"</span>: {
<span style="color: green; font-weight: bold;">"pId"</span>: {
<span style="color: green; font-weight: bold;">"S"</span>: <span style="color: #bb4444;">"$input.params('id')"</span>
}
},
<span style="color: green; font-weight: bold;">"UpdateExpression"</span> : <span style="color: #bb4444;">"SET pName = :pName, age = :age, updationTime = :utime"</span>,
<span style="color: green; font-weight: bold;">"ExpressionAttributeValues"</span>: {
<span style="color: green; font-weight: bold;">":pName"</span> : { <span style="color: green; font-weight: bold;">"S"</span>:<span style="color: #bb4444;">"$util.escapeJavaScript($input.params('name'))"</span> },
<span style="color: green; font-weight: bold;">":age"</span>: { <span style="color: green; font-weight: bold;">"N"</span>: <span style="color: #bb4444;">"$input.params('age')"</span>},
<span style="color: green; font-weight: bold;">":utime"</span>:{ <span style="color: green; font-weight: bold;">"S"</span> : <span style="color: #bb4444;">"$context.requestTime"</span>}
},
<span style="color: green; font-weight: bold;">"ReturnValues"</span>: <span style="color: #bb4444;">"ALL_NEW"</span>
}
</pre>
</div>
On hitting the url:
<!-- HTML generated using hilite.me --><br />
<div style="background: #f8f8f8; border-width: 0.1em 0.1em 0.1em 0.8em; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="line-height: 125%; margin: 0;">Execution log for request 55e87c6e-3998-43a5-8523-ef936c9acb44
Mon May 25 18:17:24 UTC 2020 : Starting execution for request: </pre>
<pre style="line-height: 125%; margin: 0;">55e87c6e-3998-43a5-8523-ef936c9acb44
<span style="color: blue;">Mon May 25 18:17:24 UTC 2020 : HTTP Method: POST, </span></pre>
<pre style="line-height: 125%; margin: 0;"><span style="color: blue;">Resource Path: /persons/52a5371-b157-4a72-91f5-8e7a0317e0fe
Mon May 25 18:17:24 UTC 2020 : Method request path: </span></pre>
<pre style="line-height: 125%; margin: 0;"><span style="color: blue;">{id=52a5371-b157-4a72-91f5-8e7a0317e0fe}
Mon May 25 18:17:24 UTC 2020 : Method request query string: </span></pre>
<pre style="line-height: 125%; margin: 0;"><span style="color: blue;">{name=Bar Bar, age=31}</span>
Mon May 25 18:17:24 UTC 2020 : Method request headers: {}
Mon May 25 18:17:24 UTC 2020 : Method request body before transformations:
Mon May 25 18:17:24 UTC 2020 : Endpoint request URI:
https://dynamodb.us-east-1.amazonaws.com/?Action=UpdateItem
Mon May 25 18:17:24 UTC 2020 : Endpoint request headers: {</pre>
<pre style="line-height: 125%; margin: 0;">Authorization=***d7c400, X-Amz-Date=20200525T181724Z,
x-amzn-apigateway-api-id=qbotcgknke, Accept=application/json, User-Agent=</pre>
<pre style="line-height: 125%; margin: 0;">AmazonAPIGateway_qbotcgknke, X-Amz-Security-Token=IQoJb3JpZ2lu [TRUNCATED]</pre>
<pre style="line-height: 125%; margin: 0;">Mon May 25 18:17:24 UTC 2020 : Endpoint request body after transformations: <span style="color: blue;">{
"TableName": "Person",
"Key": {
"pId": {
"S": "52a5371-b157-4a72-91f5-8e7a0317e0fe"
}
},
"UpdateExpression" : "SET pName = :pName, age = :age, updationTime = :utime",
"ExpressionAttributeValues": {
":pName" : { "S":"Bar Bar" },
":age": { "N": "31"},
":utime":{ "S" : "25/May/2020:18:17:24 +0000"}
},
"ReturnValues": "ALL_NEW"
}</span>
<span style="color: blue;">Mon May 25 18:17:24 UTC 2020 : Sending request to </span></pre>
<pre style="line-height: 125%; margin: 0;"><span style="color: blue;">https://dynamodb.us-east-1.amazonaws.com/?Action=UpdateItem
</span>Mon May 25 18:17:24 UTC 2020 : Received response. Status: 200, Integration latency: 24 ms
Mon May 25 18:17:24 UTC 2020 : Endpoint response headers: {Server=Server,
Date=Mon, 25 May 2020 18:17:24 GMT, Content-Type=application/x-amz-json-1.0,
Content-Length=157, Connection=keep-alive,
x-amzn-RequestId=H3CT55IRA94BPAQ973EAREL46VVV4KQNSO5AEMVJF66Q9ASUAAJG,</pre>
<pre style="line-height: 125%; margin: 0;"> x-amz-crc32=1694387638}
Mon May 25 18:17:24 UTC 2020 : Endpoint response body before transformations:</pre>
<pre style="line-height: 125%; margin: 0;"><span style="color: blue;"> {"Attributes":{"pId":{"S":"52a5371-b157-4a72-91f5-8e7a0317e0fe"},</span></pre>
<pre style="line-height: 125%; margin: 0;"><span style="color: blue;">"updationTime":{"S":"25/May/2020:18:17:24 +0000"},"pName":{"S":"Bar Bar"},</span></pre>
<pre style="line-height: 125%; margin: 0;"><span style="color: blue;">"age":{"N":"31"}}}</span>
Mon May 25 18:17:24 UTC 2020 : Method response body after transformations:</pre>
<pre style="line-height: 125%; margin: 0;"> {"Attributes":{"pId":{"S":"52a5371-b157-4a72-91f5-8e7a0317e0fe"},</pre>
<pre style="line-height: 125%; margin: 0;">"updationTime":{"S":"25/May/2020:18:17:24 +0000"},"pName":{"S":"Bar Bar"},</pre>
<pre style="line-height: 125%; margin: 0;">"age":{"N":"31"}}}
Mon May 25 18:17:24 UTC 2020 : Method response headers: {X-Amzn-Trace-Id=Root</pre>
<pre style="line-height: 125%; margin: 0;">=1-5ecc0bb4-66d6e13e89875088eb2ff333, Content-Type=application/json}
Mon May 25 18:17:24 UTC 2020 : Successfully completed execution
Mon May 25 18:17:24 UTC 2020 : Method completed with status: 200
</pre>
</div>
</div>
Robin Varghesehttp://www.blogger.com/profile/00841917517097216645noreply@blogger.com1tag:blogger.com,1999:blog-6262128735818493674.post-41526658528219903502020-05-24T05:02:00.000+05:302020-05-24T05:02:02.716+05:30REST + Dynamo = API Gateway<div dir="ltr" style="text-align: left;" trbidi="on">
My use case is very straight forward - I have a table and I want to expose CRUD operations on the table - GET, ADD, DELETE, UPDATE.<br />
<a name='more'></a>My solution would be to expose a REST interface to this table. The GET translates to a table read, PUT to an insert, DELETE to a table delete and so on. AWS provides a feature called API Gateway which fits this use case. From <a href="https://docs.aws.amazon.com/apigateway/latest/developerguide/welcome.html" target="_blank">AWS docs</a>:
<br />
<div style="background: #f8f8f8; border-width: 0.1em 0.1em 0.1em 0.8em; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="line-height: 125%; margin: 0;">API Gateway creates RESTful APIs that:
* Are HTTP-based.
* Enable stateless client-server communication.
* Implement standard HTTP methods such as GET, POST, PUT, PATCH, and DELETE.
</pre>
</div>
In this use case API Gateway(APIG) becomes the web interface for our table:
<br />
<div style="background: #f8f8f8; border-width: 0.1em 0.1em 0.1em 0.8em; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="line-height: 125%; margin: 0;">API Gateway handles all the tasks involved in accepting and processing up to </pre>
<pre style="line-height: 125%; margin: 0;">hundreds of thousands of concurrent API calls. These tasks include traffic </pre>
<pre style="line-height: 125%; margin: 0;">management, authorization and access control, monitoring, and API version </pre>
<pre style="line-height: 125%; margin: 0;">management.
</pre>
</div>
I am going to setup a simple Person table here with hash key as PersonId.<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhOhatrz_oEX9n5UuNPXaAjg2X3GhgqNppqdnz6yyArdMR9Fy85EY8xLM2Xdse8dEssHHm-9Tzb04B2BD5t91iR0WxZzI5Y5eLgs9INBDpbPCL3qCv8bC3wGnTRG9SLLVDGXTNYSkEDdk8/s1600/Dynamo.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="703" data-original-width="590" height="640" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhOhatrz_oEX9n5UuNPXaAjg2X3GhgqNppqdnz6yyArdMR9Fy85EY8xLM2Xdse8dEssHHm-9Tzb04B2BD5t91iR0WxZzI5Y5eLgs9INBDpbPCL3qCv8bC3wGnTRG9SLLVDGXTNYSkEDdk8/s640/Dynamo.png" width="537" /></a></div>
Now to setup a mechanism to access this record from HTTP - or a GET method for Person resource.<br />
API Gateway link showed support for HTTP, REST and WEB SOCKET API. From the docs it appears that HTTP API is the new recommendation for APIG but it does not support all features supported by REST API yet. Also only REST API has support for non-Lambda integrations (e.g. Dynamo).<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgCd_TF9KhZPxMNV1jLesdjAaKzRMWMakQ8-qXP1f6UqyqBzXc-LBBsp9NwD3FJInDOhGiyr7kB6zoydXrhlkH2n0hTfzRdYmZTqcJYVJJsqIUy_n_t6VpdsWVQAvfsqKKL7m-RxacHogU/s1600/GetMethod.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="841" data-original-width="966" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgCd_TF9KhZPxMNV1jLesdjAaKzRMWMakQ8-qXP1f6UqyqBzXc-LBBsp9NwD3FJInDOhGiyr7kB6zoydXrhlkH2n0hTfzRdYmZTqcJYVJJsqIUy_n_t6VpdsWVQAvfsqKKL7m-RxacHogU/s1600/GetMethod.png" /></a></div>
This sets up the API Gateway with GET method. As seen I had to add an IAM policy to allow API Gateway to talk with Dynamo Db:<br />
<br />
<b>Step 1: Create a Policy to give APIG access to Dynamo:</b><br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgxNUSwgkgY1fO-5AE5XHUBUVsSvfZBWyHRMcRdYi0VocoVH0KWrTe8lvb8S2LODHWEzP0U0IP0X9syUS36hvyG3_QU6PUKRqkWIV49E-ndeQRd_jNLCnu4Rh70VAQt4jkRpll-TCajIkI/s1600/APIGPolicy.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="623" data-original-width="626" height="636" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgxNUSwgkgY1fO-5AE5XHUBUVsSvfZBWyHRMcRdYi0VocoVH0KWrTe8lvb8S2LODHWEzP0U0IP0X9syUS36hvyG3_QU6PUKRqkWIV49E-ndeQRd_jNLCnu4Rh70VAQt4jkRpll-TCajIkI/s640/APIGPolicy.png" width="640" /></a></div>
<br />
<b> Step2: Create a role and attach the policy to it:</b><br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEizBn8NvSjAzc06NLhqUB2gxWTejM0lEbg-U6_IdK7ZDBDpG30kqDCcWZqpotK1pJlk23uAevxswlGyn9tXTbJp9zKVZ0NGJImT8Fh_wOji1N8Tvf6weZCjUAlFP32xgaqKuWkoXQVWXAQ/s1600/ApigIamRole.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="498" data-original-width="618" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEizBn8NvSjAzc06NLhqUB2gxWTejM0lEbg-U6_IdK7ZDBDpG30kqDCcWZqpotK1pJlk23uAevxswlGyn9tXTbJp9zKVZ0NGJImT8Fh_wOji1N8Tvf6weZCjUAlFP32xgaqKuWkoXQVWXAQ/s1600/ApigIamRole.png" /></a></div>
The GET method looks as below:<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhOatbc9_c9mCXjIIO4luEv1g-TYmACO9b4s7Y7nqeT1BNrf628biKNLwX2D9gSI5qzABBZIIVSNrIGlBpo9PtEocF2m-nuXL-OU_lmnWR7LhB-tn_3x_mj4w58n4hDdsXxjg1o90JesRk/s1600/APIGFlow.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="539" data-original-width="867" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhOatbc9_c9mCXjIIO4luEv1g-TYmACO9b4s7Y7nqeT1BNrf628biKNLwX2D9gSI5qzABBZIIVSNrIGlBpo9PtEocF2m-nuXL-OU_lmnWR7LhB-tn_3x_mj4w58n4hDdsXxjg1o90JesRk/s1600/APIGFlow.png" /></a></div>
<br />
<br />
This also shows how the Gateway works. A HTTP GET call is converted into a Dynamo Db POST (the Query API). Dynamo DB response is then converted to the Gateway HTTP Response.<br />
Now to complete the setup of the GET method.<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjNYXoID07dIJMBYZn5hJBVj0ogJcCaLJhvHCRyPodlT6pFjuqp9hwjSrurlWvw3WcmmjCmwg18eoRPkGgLLOKdW4GOIg2tytJ_ySGe91zvimw3SFa2k4ronDCYEGUQN1fYvKsr5kS9j4Q/s1600/IntegRequestGet.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="447" data-original-width="561" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjNYXoID07dIJMBYZn5hJBVj0ogJcCaLJhvHCRyPodlT6pFjuqp9hwjSrurlWvw3WcmmjCmwg18eoRPkGgLLOKdW4GOIg2tytJ_ySGe91zvimw3SFa2k4ronDCYEGUQN1fYvKsr5kS9j4Q/s1600/IntegRequestGet.png" /></a></div>
What is the mapping template ?
<br />
<div style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="line-height: 125%; margin: 0px;"><span style="background-color: yellow;">API Gateway uses mapping templates to transform incoming requests before they are
sent to the integration back end.</span><span style="background-color: #f8f8f8;"> With API Gateway, you can define one mapping
template for each possible content type. The content type selection is based on
the Content-Type header of the incoming request. If no content type is specified
in the request, API Gateway uses an application/json mapping template. By
default, mapping templates are configured to simply pass through the request input.
Mapping templates use Apache Velocity to generate a request for your back end. API
Gateway's reference documentation lists all of the properties and functions that
API Gateway makes available in the templates.
</span></pre>
</div>
<br />
The API Gateway is now ready. I tested my GET method<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgrTpXSVCvIrheE2-ZNp5I_ZTQTAXXgGNL8Zux-Pjtmfv-iGTNBeuvzKJDf8U-7GuIOUfafjbWArl9HFzgZuQoLE_tb2t-Hf_lXnw0oV9TKzBteZX9O7gvx5EXXf8B2S11nQAUPUZ3KOLQ/s1600/ApigTestResp.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="685" data-original-width="983" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgrTpXSVCvIrheE2-ZNp5I_ZTQTAXXgGNL8Zux-Pjtmfv-iGTNBeuvzKJDf8U-7GuIOUfafjbWArl9HFzgZuQoLE_tb2t-Hf_lXnw0oV9TKzBteZX9O7gvx5EXXf8B2S11nQAUPUZ3KOLQ/s1600/ApigTestResp.png" /></a></div>
As seen I added the id to the url path and the resultant GET call retuned the Dynamo Record.
<!-- HTML generated using hilite.me --><br />
<div style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="line-height: 125%; margin: 0px;"><span style="background-color: #f8f8f8;">Execution log for request 1d3be14f-edbb-4460-9851-bd8fca8c77a4
Sat May 23 22:18:55 UTC 2020 : Starting
execution for request: 1d3be14f-edbb-4460-9851-bd8fca8c77a4
</span><span style="background-color: yellow;">Sat May 23 22:18:55 UTC 2020 : HTTP Method: GET, Resource Path: /persons/1
Sat May 23 22:18:55 UTC 2020 : Method request path: {id=1}
Sat May 23 22:18:55 UTC 2020 : Method request query string: {}</span><span style="background-color: #f8f8f8;">
Sat May 23 22:18:55 UTC 2020 : Method request headers: {}
Sat May 23 22:18:55 UTC 2020 : Method request body before transformations:
Sat May 23 22:18:55 UTC 2020 : </span><span style="background-color: yellow;">Endpoint request URI: </span></pre>
<pre style="line-height: 125%; margin: 0px;"><span style="background-color: yellow;">https://dynamodb.us-east-1.amazonaws.com/?Action=Query</span><span style="background-color: #f8f8f8;">
Sat May 23 22:18:55 UTC 2020 : Endpoint request headers: {</span></pre>
<pre style="line-height: 125%; margin: 0px;"><span style="background-color: #f8f8f8;">Authorization=************142f25, X-Amz-Date=20200523T221855Z, </span></pre>
<pre style="line-height: 125%; margin: 0px;"><span style="background-color: #f8f8f8;">x-amzn-apigateway-api-id=qbotcgknke, Accept=application/json, </span></pre>
<pre style="line-height: 125%; margin: 0px;"><span style="background-color: #f8f8f8;">User-Agent=AmazonAPIGateway_qbotcgknke, </span></pre>
<pre style="line-height: 125%; margin: 0px;"><span style="background-color: #f8f8f8;">X-Amz-Security-Token=cqJAK7rT/grZmyP[TRUNCATED]
Sat May 23 22:18:55 UTC 2020 : Endpoint request body after transformations: {
"TableName": "Person",
"PrimaryKey": "pId",
"KeyConditionExpression": "pId = :v1",
"ExpressionAttributeValues": {
":v1": {
"S": "1"
}
}
}
Sat May 23 22:18:55 UTC 2020 : Sending request to </span></pre>
<pre style="line-height: 125%; margin: 0px;"><span style="background-color: #f8f8f8;">https://dynamodb.us-east-1.amazonaws.com/?Action=Query
Sat May 23 22:18:55 UTC 2020 : Received response. Status: 200, Integration </span></pre>
<pre style="line-height: 125%; margin: 0px;"><span style="background-color: #f8f8f8;">latency: 22 ms
Sat May 23 22:18:55 UTC 2020 : Endpoint response headers: {Server=Server, </span></pre>
<pre style="line-height: 125%; margin: 0px;"><span style="background-color: #f8f8f8;">Date=Sat, 23 May 2020 22:18:55 GMT, Content-Type=application/x-amz-json-1.0, </span></pre>
<pre style="line-height: 125%; margin: 0px;"><span style="background-color: #f8f8f8;">Content-Length=94, Connection=keep-alive, </span></pre>
<pre style="line-height: 125%; margin: 0px;"><span style="background-color: #f8f8f8;">x-amzn-RequestId=A1OCMDLIK8DID4E08UFSKIJOJJVV4KQNSO5AEMVJF66Q9ASUAAJG, </span></pre>
<pre style="line-height: 125%; margin: 0px;"><span style="background-color: #f8f8f8;">x-amz-crc32=492219340}
Sat May 23 22:18:55 UTC 2020 : Endpoint response body before transformations: </span></pre>
<pre style="line-height: 125%; margin: 0px;"><span style="background-color: #f8f8f8;">{"Count":1,"Items":[{"pId":{"S":"1"},"name":{"S":"Robin"},"age":{"N":"20"}}],</span></pre>
<pre style="line-height: 125%; margin: 0px;"><span style="background-color: #f8f8f8;">"ScannedCount":1}
Sat May 23 22:18:55 UTC 2020 : Method response body after transformations: </span></pre>
<pre style="line-height: 125%; margin: 0px;"><span style="background-color: #f8f8f8;">{"Count":1,"Items":[{"pId":{"S":"1"},"name":{"S":"Robin"},"age":{"N":"20"}}]</span></pre>
<pre style="line-height: 125%; margin: 0px;"><span style="background-color: #f8f8f8;">,"ScannedCount":1}
Sat May 23 22:18:55 UTC 2020 : Method response headers: {</span></pre>
<pre style="line-height: 125%; margin: 0px;"><span style="background-color: #f8f8f8;">X-Amzn-Trace-Id=Root=1-5ec9a14f-06c516d4e8c4c6627fdcbf16, </span></pre>
<pre style="line-height: 125%; margin: 0px;"><span style="background-color: #f8f8f8;">Content-Type=application/json}
Sat May 23 22:18:55 UTC 2020 : Successfully completed execution
Sat May 23 22:18:55 UTC 2020 : Method completed with status: 200
</span></pre>
</div>
We might not want to expose a Dynamo Db response to the client and therefore a suitable approach would be to implement a Integration Response that cleans up the dynamo db json to a clean Json
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg-bRkpcn1CSlZF0KWzMJR-PDggBtV35LLNPEDnaZgIZE1th9apMR1QMh20b1rUlQQkxZF5G-OfWu6Y4Q50YGXOCKso31mDrpRN71fcNk7gpKZbo_A-PgzqxZdijxulahDUedaG92kmyMA/s1600/ResponseTemplate.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="237" data-original-width="415" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg-bRkpcn1CSlZF0KWzMJR-PDggBtV35LLNPEDnaZgIZE1th9apMR1QMh20b1rUlQQkxZF5G-OfWu6Y4Q50YGXOCKso31mDrpRN71fcNk7gpKZbo_A-PgzqxZdijxulahDUedaG92kmyMA/s1600/ResponseTemplate.png" /></a></div>
<br /></div>
Robin Varghesehttp://www.blogger.com/profile/00841917517097216645noreply@blogger.com0tag:blogger.com,1999:blog-6262128735818493674.post-81145061090262318332020-05-21T23:16:00.006+05:302020-05-21T23:16:53.884+05:30Playing with ElastiCache - 2<div dir="ltr" style="text-align: left;" trbidi="on">
In the previous post we setup a basic Lambda function that wrote to Redis. I updated the function to log some Cloud Watch metrics.<br />
<a name='more'></a><!-- HTML generated using hilite.me --><br />
<div style="background: #f8f8f8; border-width: 0.1em 0.1em 0.1em 0.8em; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="line-height: 125%; margin: 0;"><span style="color: #aa22ff; font-weight: bold;">public</span> <span style="color: #aa22ff; font-weight: bold;">class</span> <span style="color: blue;">TokenBucketRefresherLambdaFn</span> <span style="color: #aa22ff; font-weight: bold;">implements</span> RequestHandler<span style="color: #666666;"><</span>String<span style="color: #666666;">,</span> Void<span style="color: #666666;">></span> <span style="color: #666666;">{</span>
<span style="color: #aa22ff; font-weight: bold;">private</span> <span style="color: #aa22ff; font-weight: bold;">final</span> Jedis jedis<span style="color: #666666;">;</span>
<span style="color: #aa22ff; font-weight: bold;">private</span> <span style="color: #aa22ff; font-weight: bold;">final</span> AmazonCloudWatchAsync cloudWatchClient<span style="color: #666666;">;</span>
<span style="color: #aa22ff; font-weight: bold;">public</span> <span style="color: #00a000;">TokenBucketRefresherLambdaFn</span><span style="color: #666666;">()</span> <span style="color: #666666;">{</span>
jedis <span style="color: #666666;">=</span> <span style="color: #aa22ff; font-weight: bold;">new</span> Jedis<span style="color: #666666;">(</span><span style="color: #aa22ff; font-weight: bold;">new</span> HostAndPort<span style="color: #666666;">(</span></pre>
<pre style="line-height: 125%; margin: 0;"><span style="color: #bb4444;"> "redis-test-cache-001.vmaptv.0001.use1.cache.amazonaws.com"</span><span style="color: #666666;">,</span><span style="color: #666666;"> </span><span style="color: #666666;">6379));</span>
AWSCredentials awsCredentials <span style="color: #666666;">=</span> <span style="color: #aa22ff; font-weight: bold;">new</span> BasicAWSCredentials<span style="color: #666666;">(</span><span style="color: #bb4444;">"AccessKey"</span><span style="color: #666666;">,</span>
<span style="color: #bb4444;">"SecretKey"</span><span style="color: #666666;">);</span>
AWSCredentialsProvider awsCredentialsProvider <span style="color: #666666;">=</span> <span style="color: #aa22ff; font-weight: bold;">new</span> </pre>
<pre style="line-height: 125%; margin: 0;"> AWSStaticCredentialsProvider<span style="color: #666666;">(</span>awsCredentials<span style="color: #666666;">);</span>
cloudWatchClient <span style="color: #666666;">=</span> AmazonCloudWatchAsyncClientBuilder<span style="color: #666666;">.</span><span style="color: #bb4444;">standard</span><span style="color: #666666;">()</span></pre>
<pre style="line-height: 125%; margin: 0;"><span style="color: #666666;"> .</span><span style="color: #bb4444;">withCredentials</span><span style="color: #666666;">(</span>awsCredentialsProvider<span style="color: #666666;">)</span>
<span style="color: #666666;">.</span><span style="color: #bb4444;">withClientConfiguration</span><span style="color: #666666;">(</span><span style="color: #aa22ff; font-weight: bold;">new</span> ClientConfiguration<span style="color: #666666;">()).</span><span style="color: #bb4444;">build</span><span style="color: #666666;">();</span>
<span style="color: #666666;">}</span>
<span style="color: #aa22ff;">@Override</span>
<span style="color: #aa22ff; font-weight: bold;">public</span> Void <span style="color: #00a000;">handleRequest</span><span style="color: #666666;">(</span>String input<span style="color: #666666;">,</span> Context context<span style="color: #666666;">)</span> <span style="color: #666666;">{</span>
LambdaLogger logger <span style="color: #666666;">=</span> context<span style="color: #666666;">.</span><span style="color: #bb4444;">getLogger</span><span style="color: #666666;">();</span>
logger<span style="color: #666666;">.</span><span style="color: #bb4444;">log</span><span style="color: #666666;">(</span><span style="color: #bb4444;">"In Handler: Executing "</span> <span style="color: #666666;">+</span> context<span style="color: #666666;">.</span><span style="color: #bb4444;">getFunctionName</span><span style="color: #666666;">()</span> <span style="color: #666666;">+</span> <span style="color: #bb4444;">", "</span> </pre>
<pre style="line-height: 125%; margin: 0;"><span style="color: #666666;"> +</span> context<span style="color: #666666;">.</span><span style="color: #bb4444;">getFunctionVersion</span><span style="color: #666666;">());</span>
logger<span style="color: #666666;">.</span><span style="color: #bb4444;">log</span><span style="color: #666666;">(</span><span style="color: #bb4444;">"Input received "</span> <span style="color: #666666;">+</span> input<span style="color: #666666;">);</span> <span style="color: #008800; font-style: italic;">//This input is not really used now</span>
<span style="color: #00bb00; font-weight: bold;">int</span> totalRuns <span style="color: #666666;">=</span> <span style="color: #666666;">0;</span>
<span style="color: #00bb00; font-weight: bold;">long</span> startTime <span style="color: #666666;">=</span> System<span style="color: #666666;">.</span><span style="color: #bb4444;">currentTimeMillis</span><span style="color: #666666;">();</span>
List<span style="color: #666666;"><</span>Integer<span style="color: #666666;">></span> availCapacities <span style="color: #666666;">=</span> <span style="color: #aa22ff; font-weight: bold;">new</span> ArrayList<span style="color: #666666;"><>(60);</span>
<span style="color: #aa22ff; font-weight: bold;">while</span> <span style="color: #666666;">(</span>totalRuns <span style="color: #666666;"><</span> <span style="color: #666666;">60</span> <span style="color: #666666;">&&</span> <span style="color: #666666;">(</span>System<span style="color: #666666;">.</span><span style="color: #bb4444;">currentTimeMillis</span><span style="color: #666666;">()</span> <span style="color: #666666;">-</span> startTime <span style="color: #666666;"><</span> <span style="color: #666666;">1000</span> <span style="color: #666666;">*</span> <span style="color: #666666;">60))</span> <span style="color: #666666;">{</span>
<span style="color: #00bb00; font-weight: bold;">long</span> timestamp <span style="color: #666666;">=</span> System<span style="color: #666666;">.</span><span style="color: #bb4444;">currentTimeMillis</span><span style="color: #666666;">();</span>
<span style="color: #00bb00; font-weight: bold;">int</span> availCapacity <span style="color: #666666;">=</span> updateTokenCapacity<span style="color: #666666;">(</span>logger<span style="color: #666666;">);</span>
availCapacities<span style="color: #666666;">.</span><span style="color: #bb4444;">add</span><span style="color: #666666;">(</span>availCapacity<span style="color: #666666;">);</span>
asyncLogMetricsToCloudWatch<span style="color: #666666;">(</span>availCapacity<span style="color: #666666;">);</span>
<span style="color: #00bb00; font-weight: bold;">long</span> duration <span style="color: #666666;">=</span> System<span style="color: #666666;">.</span><span style="color: #bb4444;">currentTimeMillis</span><span style="color: #666666;">()</span> <span style="color: #666666;">-</span> timestamp<span style="color: #666666;">;</span>
<span style="color: #aa22ff; font-weight: bold;">if</span> <span style="color: #666666;">(</span>duration <span style="color: #666666;"><</span> <span style="color: #666666;">1000)</span> <span style="color: #666666;">{</span>
<span style="color: #008800; font-style: italic;">//Operation took less than a second, let Thread sleep before running again</span>
<span style="color: #aa22ff; font-weight: bold;">try</span> <span style="color: #666666;">{</span>
Thread<span style="color: #666666;">.</span><span style="color: #bb4444;">sleep</span><span style="color: #666666;">(1000</span> <span style="color: #666666;">-</span> duration<span style="color: #666666;">);</span>
<span style="color: #666666;">}</span> <span style="color: #aa22ff; font-weight: bold;">catch</span> <span style="color: #666666;">(</span>InterruptedException e<span style="color: #666666;">)</span> <span style="color: #666666;">{</span>
logger<span style="color: #666666;">.</span><span style="color: #bb4444;">log</span><span style="color: #666666;">(</span><span style="color: #bb4444;">"Sleep failed "</span> <span style="color: #666666;">+</span> e<span style="color: #666666;">.</span><span style="color: #bb4444;">getMessage</span><span style="color: #666666;">());</span>
<span style="color: #666666;">}</span>
<span style="color: #666666;">}</span>
totalRuns<span style="color: #666666;">++;</span>
<span style="color: #666666;">}</span>
logger<span style="color: #666666;">.</span><span style="color: #bb4444;">log</span><span style="color: #666666;">(</span><span style="color: #bb4444;">"Total Runs "</span> <span style="color: #666666;">+</span> totalRuns <span style="color: #666666;">+</span> <span style="color: #bb4444;">" and available capacities detected "</span> <span style="color: #666666;">+</span> availCapacities<span style="color: #666666;">);</span>
<span style="color: #aa22ff; font-weight: bold;">return</span> <span style="color: #aa22ff; font-weight: bold;">null</span><span style="color: #666666;">;</span>
<span style="color: #666666;">}</span>
<span style="color: #aa22ff; font-weight: bold;">private</span> <span style="color: #00bb00; font-weight: bold;">int</span> <span style="color: #00a000;">updateTokenCapacity</span><span style="color: #666666;">(</span>LambdaLogger logger<span style="color: #666666;">)</span> <span style="color: #666666;">{</span>
String token <span style="color: #666666;">=</span> <span style="color: #bb4444;">"TokenCount"</span><span style="color: #666666;">;</span>
<span style="color: #00bb00; font-weight: bold;">int</span> capacity <span style="color: #666666;">=</span> <span style="color: #666666;">60;</span> <span style="color: #008800; font-style: italic;">//adds 60 tokens at every run. Bucket capacity is 100</span>
String value <span style="color: #666666;">=</span> jedis<span style="color: #666666;">.</span><span style="color: #bb4444;">get</span><span style="color: #666666;">(</span>token<span style="color: #666666;">);</span>
<span style="color: #00bb00; font-weight: bold;">int</span> crtBucketValue <span style="color: #666666;">=</span> <span style="color: #666666;">0;</span>
<span style="color: #aa22ff; font-weight: bold;">if</span> <span style="color: #666666;">(</span>value <span style="color: #666666;">==</span> <span style="color: #aa22ff; font-weight: bold;">null</span><span style="color: #666666;">)</span> <span style="color: #666666;">{</span>
logger<span style="color: #666666;">.</span><span style="color: #bb4444;">log</span><span style="color: #666666;">(</span><span style="color: #bb4444;">"No value detected for Key, adding"</span><span style="color: #666666;">);</span>
<span style="color: #666666;">}</span> <span style="color: #aa22ff; font-weight: bold;">else</span> <span style="color: #666666;">{</span>
crtBucketValue <span style="color: #666666;">=</span> Integer<span style="color: #666666;">.</span><span style="color: #bb4444;">parseInt</span><span style="color: #666666;">(</span>value<span style="color: #666666;">);</span>
<span style="color: #666666;">}</span>
<span style="color: #00bb00; font-weight: bold;">int</span> newBucketCapacity <span style="color: #666666;">=</span> Math<span style="color: #666666;">.</span><span style="color: #bb4444;">min</span><span style="color: #666666;">(100,</span> crtBucketValue <span style="color: #666666;">+</span> <span style="color: #666666;">60);</span>
jedis<span style="color: #666666;">.</span><span style="color: #bb4444;">set</span><span style="color: #666666;">(</span>token<span style="color: #666666;">,</span> String<span style="color: #666666;">.</span><span style="color: #bb4444;">valueOf</span><span style="color: #666666;">(</span>newBucketCapacity<span style="color: #666666;">));</span>
<span style="color: #aa22ff; font-weight: bold;">return</span> value <span style="color: #666666;">==</span> <span style="color: #aa22ff; font-weight: bold;">null</span> <span style="color: #666666;">?</span> <span style="color: #666666;">0</span> <span style="color: #666666;">:</span> Integer<span style="color: #666666;">.</span><span style="color: #bb4444;">valueOf</span><span style="color: #666666;">(</span>value<span style="color: #666666;">); </span></pre>
<pre style="line-height: 125%; margin: 0;"><span style="color: #008800; font-style: italic;"> //capacity available for use</span>
<span style="color: #666666;">}</span>
<span style="color: #aa22ff; font-weight: bold;">private</span> <span style="color: #00bb00; font-weight: bold;">void</span> <span style="color: #00a000;">asyncLogMetricsToCloudWatch</span><span style="color: #666666;">(</span><span style="color: #00bb00; font-weight: bold;">int</span> unusedCapacity<span style="color: #666666;">)</span> <span style="color: #666666;">{</span>
Collection<span style="color: #666666;"><</span>MetricDatum<span style="color: #666666;">></span> metricDatums <span style="color: #666666;">=</span> <span style="color: #aa22ff; font-weight: bold;">new</span> ArrayList<span style="color: #666666;"><>(1);</span>
metricDatums<span style="color: #666666;">.</span><span style="color: #bb4444;">add</span><span style="color: #666666;">(</span><span style="color: #aa22ff; font-weight: bold;">new</span> MetricDatum<span style="color: #666666;">()</span>
<span style="color: #666666;">.</span><span style="color: #bb4444;">withMetricName</span><span style="color: #666666;">(</span><span style="color: #bb4444;">"capacityAvailable"</span><span style="color: #666666;">)</span>
<span style="color: #666666;">.</span><span style="color: #bb4444;">withValue</span><span style="color: #666666;">(</span>Double<span style="color: #666666;">.</span><span style="color: #bb4444;">valueOf</span><span style="color: #666666;">(</span>unusedCapacity<span style="color: #666666;">))</span>
<span style="color: #666666;">.</span><span style="color: #bb4444;">withUnit</span><span style="color: #666666;">(</span>StandardUnit<span style="color: #666666;">.</span><span style="color: #bb4444;">None</span><span style="color: #666666;">));</span>
PutMetricDataRequest putMetricDataRequest <span style="color: #666666;">=</span> <span style="color: #aa22ff; font-weight: bold;">new</span> PutMetricDataRequest<span style="color: #666666;">();</span>
putMetricDataRequest<span style="color: #666666;">.</span><span style="color: #bb4444;">setMetricData</span><span style="color: #666666;">(</span>metricDatums<span style="color: #666666;">);</span>
putMetricDataRequest<span style="color: #666666;">.</span><span style="color: #bb4444;">setNamespace</span><span style="color: #666666;">(</span><span style="color: #bb4444;">"TokenBucket"</span><span style="color: #666666;">);</span>
<span style="color: #008800; font-style: italic;">//Asynchronous push - returns a future with result</span>
cloudWatchClient<span style="color: #666666;">.</span><span style="color: #bb4444;">putMetricDataAsync</span><span style="color: #666666;">(</span>putMetricDataRequest<span style="color: #666666;">);</span>
<span style="color: #666666;">}</span>
<span style="color: #666666;">}</span>
</pre>
</div>
This code is trying to do a very simplified implementation of the Token Bucket Algorithm.<br />
The Lambda runs every minute. Within the code, it attempts to update the bucket every second by adding 60 tokens in every cache update. The bucket size is capped at 100 tokens. The Lambda also logs the number of tokens it detected as available within the bucket.<br />
I am using Cloud Watch events to invoke my Lambda. As Cloud Watch events can be scheduled at maximum frequency of once per minute, I decided to run a 1 minute loop within my function. Effectively achieving a 1 second update for my Token Bucket<br />
<br />
On running this code kept failing with a timeout exception.
<!-- HTML generated using hilite.me --><br />
<div style="background: #f8f8f8; border-width: 0.1em 0.1em 0.1em 0.8em; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="line-height: 125%; margin: 0;">Logging Cloud Watch metric 100
<span style="color: red;">Unable to execute HTTP request: Connect to monitoring.us-east-1.amazonaws.com:443</span></pre>
<pre style="line-height: 125%; margin: 0;"><span style="color: red;"> [monitoring.us-east-1.amazonaws.com/52.94.238.171] failed: connect timed out: </span></pre>
<pre style="line-height: 125%; margin: 0;"><span style="color: red;">com.amazonaws.SdkClientException</span>
com.amazonaws.SdkClientException: Unable to execute HTTP request: Connect to </pre>
<pre style="line-height: 125%; margin: 0;">monitoring.us-east-1.amazonaws.com:443 [monitoring.us-east-1.amazonaws.com/</pre>
<pre style="line-height: 125%; margin: 0;">52.94.238.171] failed: connect timed out
</pre>
</div>
<br />
A little read-up revealed that my Lambda was unable to contact to AWS Cloud Watch endpoints.
<!-- HTML generated using hilite.me --><br />
<div style="background: #f8f8f8; border-width: 0.1em 0.1em 0.1em 0.8em; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="line-height: 125%; margin: 0;">By default, Lambda runs your functions in a secure VPC with access to AWS </pre>
<pre style="line-height: 125%; margin: 0;">services and the internet. The VPC is owned by Lambda and does not connect to </pre>
<pre style="line-height: 125%; margin: 0;">your account's default VPC. When you connect a function to a VPC in your account,</pre>
<pre style="line-height: 125%; margin: 0;">it does not have access to the internet unless your VPC provides access.
</pre>
</div>
<br />
The Elasti Cache instance exists within a VPC. In order for Lambda function to be able to access the Cache, we provided Lambda with VPC, subnets and security group details. However that means Lambda function can no longer connect to Cloud Watch API. To resolve this we need to use VPC Endpoints.<br />
<div style="background: #f8f8f8; border-width: 0.1em 0.1em 0.1em 0.8em; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="line-height: 125%; margin: 0;">A VPC endpoint enables you to privately connect your VPC to supported AWS </pre>
<pre style="line-height: 125%; margin: 0;">services and VPC endpoint services powered by AWS PrivateLink without requiring </pre>
<pre style="line-height: 125%; margin: 0;">an internet gateway, NAT device, VPN connection, or AWS Direct Connect </pre>
<pre style="line-height: 125%; margin: 0;">connection.Instances in your VPC do not require public IP addresses to </pre>
<pre style="line-height: 125%; margin: 0;">communicate with resources in the service. Traffic between your VPC and the other</pre>
<pre style="line-height: 125%; margin: 0;">service does not leave the Amazon network.</pre>
</div>
<br />
CloudWatch falls in this category.
<br />
<div style="background: #f8f8f8; border-width: 0.1em 0.1em 0.1em 0.8em; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="line-height: 125%; margin: 0;">If you use Amazon Virtual Private Cloud (Amazon VPC) to host your AWS resources,</pre>
<pre style="line-height: 125%; margin: 0;"> you can establish a private connection between your VPC, CloudWatch, and </pre>
<pre style="line-height: 125%; margin: 0;">CloudWatch Synthetics. You can use these connections to enable CloudWatch and </pre>
<pre style="line-height: 125%; margin: 0;">CloudWatch Synthetics to communicate with your resources on your VPC without </pre>
<pre style="line-height: 125%; margin: 0;">going through the public internet.
</pre>
</div>
<br />
I created Interface VPC Endpoint to talk with AWS Cloudwatch as suggested <a href="https://docs.aws.amazon.com/vpc/latest/userguide/vpce-interface.html#create-interface-endpoint.html" target="_blank">here</a>.<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjQUV6XPU7mSJON13Xp9nNPvpYbTei51MG1EIbo1_cCle6Yjelf2zDKuuvp7bUh_MVHlD6vcQHG83AoQJPSUmBn0oNAtx8_DySAVP_oUtUWDtEyVYhoem5M6uPfOo2F3bzzx0F326uQoXs/s1600/EndpintCloudWatch.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="716" data-original-width="1166" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjQUV6XPU7mSJON13Xp9nNPvpYbTei51MG1EIbo1_cCle6Yjelf2zDKuuvp7bUh_MVHlD6vcQHG83AoQJPSUmBn0oNAtx8_DySAVP_oUtUWDtEyVYhoem5M6uPfOo2F3bzzx0F326uQoXs/s1600/EndpintCloudWatch.png" /></a></div>
<br />
The subnets and security group are same as my VPC. The policy is edited to only support CloudWatch.<br />
<br />
Rerunning the Lambda worked.<br />
<!-- HTML generated using hilite.me --><br />
<div style="background: #f8f8f8; border-width: 0.1em 0.1em 0.1em 0.8em; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="line-height: 125%; margin: 0;">In Handler: Executing TokenBucketRefresher, $LATEST
Input received 2020-05-21T16:40:54Z
Total Runs 60 and available capacities detected [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,</pre>
<pre style="line-height: 125%; margin: 0;"> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 60, 100, 0, 0, 0, 0, 0, 0, 0, 0,</pre>
<pre style="line-height: 125%; margin: 0;"> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
END RequestId: fecd5efb-5d00-4b84-ba40-da144007cc9a
REPORT RequestId: fecd5efb-5d00-4b84-ba40-da144007cc9a Duration: 60165.64 ms </pre>
<pre style="line-height: 125%; margin: 0;">Billed Duration: 60200 ms Memory Size: 512 MB Max Memory Used: 143 MB </pre>
<pre style="line-height: 125%; margin: 0;">Init Duration: 1993.44 ms
</pre>
</div>
I set up a cloud watch event to invoke my Lambda every minute.<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh99esqrJIzBnQNJrRHwD0jAKgid56WFzKPSqUpUQFT0PhMsyE7X0Cv0Ec-uAakN92M4XP4cLTjpNAz81gciXtPkNIDY1-XlP3V5fITQH2XFL01Fhm01NQFSXMXtKHs98a4Hv6XIqP45A4/s1600/RefresherLambdaRule.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="306" data-original-width="873" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh99esqrJIzBnQNJrRHwD0jAKgid56WFzKPSqUpUQFT0PhMsyE7X0Cv0Ec-uAakN92M4XP4cLTjpNAz81gciXtPkNIDY1-XlP3V5fITQH2XFL01Fhm01NQFSXMXtKHs98a4Hv6XIqP45A4/s1600/RefresherLambdaRule.png" /></a></div>
Similarly I created a Lambda function that consumes tokens from the bucket<br />
<!-- HTML generated using hilite.me --><br />
<div style="background: #f8f8f8; border-width: 0.1em 0.1em 0.1em 0.8em; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="line-height: 125%; margin: 0;"><span style="color: #aa22ff; font-weight: bold;">public</span> <span style="color: #aa22ff; font-weight: bold;">class</span> <span style="color: blue;">TokenBucketUserLambdaFn</span> <span style="color: #aa22ff; font-weight: bold;">implements</span> RequestHandler<span style="color: #666666;"><</span>String<span style="color: #666666;">,</span> Void<span style="color: #666666;">></span> <span style="color: #666666;">{</span>
<span style="color: #aa22ff; font-weight: bold;">private</span> <span style="color: #aa22ff; font-weight: bold;">final</span> Jedis jedis<span style="color: #666666;">;</span>
<span style="color: #aa22ff; font-weight: bold;">private</span> <span style="color: #aa22ff; font-weight: bold;">final</span> AmazonCloudWatchAsync cloudWatchClient<span style="color: #666666;">;</span>
<span style="color: #aa22ff; font-weight: bold;">private</span> Map<span style="color: #666666;"><</span>String<span style="color: #666666;">,</span> Integer<span style="color: #666666;">></span> NEEDED_CAPACITY_MAPPING <span style="color: #666666;">=</span> <span style="color: #aa22ff; font-weight: bold;">new</span> HashMap<span style="color: #666666;"><>();</span>
<span style="color: #666666;">{</span>
NEEDED_CAPACITY_MAPPING<span style="color: #666666;">.</span><span style="color: #bb4444;">put</span><span style="color: #666666;">(</span><span style="color: #bb4444;">"BLUE"</span><span style="color: #666666;">,</span> <span style="color: #666666;">60);</span>
NEEDED_CAPACITY_MAPPING<span style="color: #666666;">.</span><span style="color: #bb4444;">put</span><span style="color: #666666;">(</span><span style="color: #bb4444;">"RED"</span><span style="color: #666666;">,</span> <span style="color: #666666;">90);</span>
NEEDED_CAPACITY_MAPPING<span style="color: #666666;">.</span><span style="color: #bb4444;">put</span><span style="color: #666666;">(</span><span style="color: #bb4444;">"YELLOW"</span><span style="color: #666666;">,</span> <span style="color: #666666;">10);</span>
<span style="color: #666666;">}</span>
<span style="color: #aa22ff; font-weight: bold;">public</span> <span style="color: #00a000;">TokenBucketUserLambdaFn</span><span style="color: #666666;">()</span> <span style="color: #666666;">{</span>
jedis <span style="color: #666666;">=</span> <span style="color: #aa22ff; font-weight: bold;">new</span> Jedis<span style="color: #666666;">(</span><span style="color: #aa22ff; font-weight: bold;">new</span> HostAndPort<span style="color: #666666;">(</span><span style="color: #bb4444;">"redis-test-cache-001.vmaptv.0001.use1.cache.amazonaws.com"</span><span style="color: #666666;">,</span>
<span style="color: #666666;">6379));</span>
AWSCredentials awsCredentials <span style="color: #666666;">=</span> <span style="color: #aa22ff; font-weight: bold;">new</span> BasicAWSCredentials<span style="color: #666666;">(</span><span style="color: #bb4444;">"AccessKey"</span><span style="color: #666666;">,</span>
<span style="color: #666666;"> <span style="color: #bb4444;">"SecretKey"</span>);</span>
AWSCredentialsProvider awsCredentialsProvider <span style="color: #666666;">=</span> <span style="color: #aa22ff; font-weight: bold;">new</span> AWSStaticCredentialsProvider<span style="color: #666666;">(</span>awsCredentials<span style="color: #666666;">);</span>
cloudWatchClient <span style="color: #666666;">=</span> AmazonCloudWatchAsyncClientBuilder<span style="color: #666666;">.</span><span style="color: #bb4444;">standard</span><span style="color: #666666;">()</span></pre>
<pre style="line-height: 125%; margin: 0;"><span style="color: #666666;"> .</span><span style="color: #bb4444;">withCredentials</span><span style="color: #666666;">(</span>awsCredentialsProvider<span style="color: #666666;">)</span>
<span style="color: #666666;">.</span><span style="color: #bb4444;">withClientConfiguration</span><span style="color: #666666;">(</span><span style="color: #aa22ff; font-weight: bold;">new</span> ClientConfiguration<span style="color: #666666;">()).</span><span style="color: #bb4444;">build</span><span style="color: #666666;">();</span>
<span style="color: #666666;">}</span>
<span style="color: #aa22ff;">@Override</span>
<span style="color: #aa22ff; font-weight: bold;">public</span> Void <span style="color: #00a000;">handleRequest</span><span style="color: #666666;">(</span>String input<span style="color: #666666;">,</span> Context context<span style="color: #666666;">)</span> <span style="color: #666666;">{</span>
LambdaLogger logger <span style="color: #666666;">=</span> context<span style="color: #666666;">.</span><span style="color: #bb4444;">getLogger</span><span style="color: #666666;">();</span>
logger<span style="color: #666666;">.</span><span style="color: #bb4444;">log</span><span style="color: #666666;">(</span><span style="color: #bb4444;">"In Handler: Executing "</span> <span style="color: #666666;">+</span> context<span style="color: #666666;">.</span><span style="color: #bb4444;">getFunctionName</span><span style="color: #666666;">()</span> <span style="color: #666666;">+</span> <span style="color: #bb4444;">", "</span> <span style="color: #666666;">+</span> context<span style="color: #666666;">.</span><span style="color: #bb4444;">getFunctionVersion</span><span style="color: #666666;">());</span>
logger<span style="color: #666666;">.</span><span style="color: #bb4444;">log</span><span style="color: #666666;">(</span><span style="color: #bb4444;">"Input received "</span> <span style="color: #666666;">+</span> input<span style="color: #666666;">);</span>
<span style="color: #aa22ff; font-weight: bold;">if</span> <span style="color: #666666;">(!</span>NEEDED_CAPACITY_MAPPING<span style="color: #666666;">.</span><span style="color: #bb4444;">containsKey</span><span style="color: #666666;">(</span>input<span style="color: #666666;">))</span> <span style="color: #666666;">{</span>
<span style="color: #aa22ff; font-weight: bold;">throw</span> <span style="color: #aa22ff; font-weight: bold;">new</span> <span style="color: #00a000;">IllegalArgumentException</span><span style="color: #666666;">(</span><span style="color: #bb4444;">"Input is invalid - "</span> <span style="color: #666666;">+</span> input<span style="color: #666666;">);</span>
<span style="color: #666666;">}</span>
<span style="color: #00bb00; font-weight: bold;">int</span> totalRuns <span style="color: #666666;">=</span> <span style="color: #666666;">0;</span>
<span style="color: #00bb00; font-weight: bold;">long</span> startTime <span style="color: #666666;">=</span> System<span style="color: #666666;">.</span><span style="color: #bb4444;">currentTimeMillis</span><span style="color: #666666;">();</span>
List<span style="color: #666666;"><</span>Integer<span style="color: #666666;">></span> availCapacities <span style="color: #666666;">=</span> <span style="color: #aa22ff; font-weight: bold;">new</span> ArrayList<span style="color: #666666;"><>(60);</span>
<span style="color: #aa22ff; font-weight: bold;">while</span> <span style="color: #666666;">(</span>totalRuns <span style="color: #666666;"><</span> <span style="color: #666666;">60</span> <span style="color: #666666;">&&</span> <span style="color: #666666;">(</span>System<span style="color: #666666;">.</span><span style="color: #bb4444;">currentTimeMillis</span><span style="color: #666666;">()</span> <span style="color: #666666;">-</span> startTime <span style="color: #666666;"><</span> <span style="color: #666666;">1000</span> <span style="color: #666666;">*</span> <span style="color: #666666;">60))</span> <span style="color: #666666;">{</span>
<span style="color: #00bb00; font-weight: bold;">long</span> timestamp <span style="color: #666666;">=</span> System<span style="color: #666666;">.</span><span style="color: #bb4444;">currentTimeMillis</span><span style="color: #666666;">();</span>
<span style="color: #00bb00; font-weight: bold;">int</span> availCapacity <span style="color: #666666;">=</span> getTokens<span style="color: #666666;">(</span>logger<span style="color: #666666;">,</span> NEEDED_CAPACITY_MAPPING<span style="color: #666666;">.</span><span style="color: #bb4444;">get</span><span style="color: #666666;">(</span>input<span style="color: #666666;">));</span>
availCapacities<span style="color: #666666;">.</span><span style="color: #bb4444;">add</span><span style="color: #666666;">(</span>availCapacity<span style="color: #666666;">);</span>
asyncLogMetricsToCloudWatch<span style="color: #666666;">(</span>availCapacity<span style="color: #666666;">,</span> input<span style="color: #666666;">);</span>
<span style="color: #00bb00; font-weight: bold;">long</span> duration <span style="color: #666666;">=</span> System<span style="color: #666666;">.</span><span style="color: #bb4444;">currentTimeMillis</span><span style="color: #666666;">()</span> <span style="color: #666666;">-</span> timestamp<span style="color: #666666;">;</span>
<span style="color: #aa22ff; font-weight: bold;">if</span> <span style="color: #666666;">(</span>duration <span style="color: #666666;"><</span> <span style="color: #666666;">1000)</span> <span style="color: #666666;">{</span>
<span style="color: #008800; font-style: italic;">//Operation took less than a second, let Thread sleep before running again</span>
<span style="color: #aa22ff; font-weight: bold;">try</span> <span style="color: #666666;">{</span>
Thread<span style="color: #666666;">.</span><span style="color: #bb4444;">sleep</span><span style="color: #666666;">(1000</span> <span style="color: #666666;">-</span> duration<span style="color: #666666;">);</span>
<span style="color: #666666;">}</span> <span style="color: #aa22ff; font-weight: bold;">catch</span> <span style="color: #666666;">(</span>InterruptedException e<span style="color: #666666;">)</span> <span style="color: #666666;">{</span>
logger<span style="color: #666666;">.</span><span style="color: #bb4444;">log</span><span style="color: #666666;">(</span><span style="color: #bb4444;">"Sleep failed "</span> <span style="color: #666666;">+</span> e<span style="color: #666666;">.</span><span style="color: #bb4444;">getMessage</span><span style="color: #666666;">());</span>
<span style="color: #666666;">}</span>
<span style="color: #666666;">}</span>
totalRuns<span style="color: #666666;">++;</span>
<span style="color: #666666;">}</span>
logger<span style="color: #666666;">.</span><span style="color: #bb4444;">log</span><span style="color: #666666;">(</span><span style="color: #bb4444;">"Total Runs "</span> <span style="color: #666666;">+</span> totalRuns <span style="color: #666666;">+</span> <span style="color: #bb4444;">" for "</span> <span style="color: #666666;">+</span> input <span style="color: #666666;">+</span> <span style="color: #bb4444;">" and available capacities detected "</span> <span style="color: #666666;">+</span> availCapacities<span style="color: #666666;">);</span>
<span style="color: #aa22ff; font-weight: bold;">return</span> <span style="color: #aa22ff; font-weight: bold;">null</span><span style="color: #666666;">;</span>
<span style="color: #666666;">}</span>
<span style="color: #aa22ff; font-weight: bold;">private</span> <span style="color: #00bb00; font-weight: bold;">int</span> <span style="color: #00a000;">getTokens</span><span style="color: #666666;">(</span>LambdaLogger logger<span style="color: #666666;">,</span> <span style="color: #00bb00; font-weight: bold;">int</span> requestedCapacity<span style="color: #666666;">)</span> <span style="color: #666666;">{</span>
String token <span style="color: #666666;">=</span> <span style="color: #bb4444;">"TokenCount"</span><span style="color: #666666;">;</span>
String value <span style="color: #666666;">=</span> jedis<span style="color: #666666;">.</span><span style="color: #bb4444;">get</span><span style="color: #666666;">(</span>token<span style="color: #666666;">);</span><span style="color: #008800; font-style: italic;">//Get available tokens</span>
<span style="color: #aa22ff; font-weight: bold;">if</span> <span style="color: #666666;">(</span>value <span style="color: #666666;">==</span> <span style="color: #aa22ff; font-weight: bold;">null</span><span style="color: #666666;">)</span> <span style="color: #666666;">{</span>
logger<span style="color: #666666;">.</span><span style="color: #bb4444;">log</span><span style="color: #666666;">(</span><span style="color: #bb4444;">"No value detected for Key"</span><span style="color: #666666;">);</span>
<span style="color: #aa22ff; font-weight: bold;">return</span> <span style="color: #666666;">0;</span>
<span style="color: #666666;">}</span>
<span style="color: #00bb00; font-weight: bold;">int</span> availCapacity <span style="color: #666666;">=</span> Integer<span style="color: #666666;">.</span><span style="color: #bb4444;">valueOf</span><span style="color: #666666;">(</span>value<span style="color: #666666;">);</span>
<span style="color: #00bb00; font-weight: bold;">int</span> tokensUsed <span style="color: #666666;">=</span> availCapacity <span style="color: #666666;"><</span> requestedCapacity <span style="color: #666666;">?</span> availCapacity <span style="color: #666666;">:</span> requestedCapacity<span style="color: #666666;">;</span>
jedis<span style="color: #666666;">.</span><span style="color: #bb4444;">set</span><span style="color: #666666;">(</span>token<span style="color: #666666;">,</span> String<span style="color: #666666;">.</span><span style="color: #bb4444;">valueOf</span><span style="color: #666666;">(</span>availCapacity <span style="color: #666666;">-</span> tokensUsed<span style="color: #666666;">));</span>
<span style="color: #aa22ff; font-weight: bold;">return</span> tokensUsed<span style="color: #666666;">;</span><span style="color: #008800; font-style: italic;">//used capacity</span>
<span style="color: #666666;">}</span>
<span style="color: #aa22ff; font-weight: bold;">private</span> <span style="color: #00bb00; font-weight: bold;">void</span> <span style="color: #00a000;">asyncLogMetricsToCloudWatch</span><span style="color: #666666;">(</span><span style="color: #00bb00; font-weight: bold;">int</span> unusedCapacity<span style="color: #666666;">,</span> String input<span style="color: #666666;">)</span> <span style="color: #666666;">{</span>
Collection<span style="color: #666666;"><</span>MetricDatum<span style="color: #666666;">></span> metricDatums <span style="color: #666666;">=</span> <span style="color: #aa22ff; font-weight: bold;">new</span> ArrayList<span style="color: #666666;"><>(1);</span>
metricDatums<span style="color: #666666;">.</span><span style="color: #bb4444;">add</span><span style="color: #666666;">(</span><span style="color: #aa22ff; font-weight: bold;">new</span> MetricDatum<span style="color: #666666;">()</span>
<span style="color: #666666;">.</span><span style="color: #bb4444;">withMetricName</span><span style="color: #666666;">(</span>input <span style="color: #666666;">+</span> <span style="color: #bb4444;">"CapacityUsed"</span><span style="color: #666666;">)</span>
<span style="color: #666666;">.</span><span style="color: #bb4444;">withValue</span><span style="color: #666666;">(</span>Double<span style="color: #666666;">.</span><span style="color: #bb4444;">valueOf</span><span style="color: #666666;">(</span>unusedCapacity<span style="color: #666666;">))</span>
<span style="color: #666666;">.</span><span style="color: #bb4444;">withUnit</span><span style="color: #666666;">(</span>StandardUnit<span style="color: #666666;">.</span><span style="color: #bb4444;">None</span><span style="color: #666666;">));</span>
PutMetricDataRequest putMetricDataRequest <span style="color: #666666;">=</span> <span style="color: #aa22ff; font-weight: bold;">new</span> PutMetricDataRequest<span style="color: #666666;">();</span>
putMetricDataRequest<span style="color: #666666;">.</span><span style="color: #bb4444;">setMetricData</span><span style="color: #666666;">(</span>metricDatums<span style="color: #666666;">);</span>
putMetricDataRequest<span style="color: #666666;">.</span><span style="color: #bb4444;">setNamespace</span><span style="color: #666666;">(</span><span style="color: #bb4444;">"TokenBucket"</span><span style="color: #666666;">);</span>
<span style="color: #008800; font-style: italic;">//Asynchronous push - returns a future with result</span>
cloudWatchClient<span style="color: #666666;">.</span><span style="color: #bb4444;">putMetricDataAsync</span><span style="color: #666666;">(</span>putMetricDataRequest<span style="color: #666666;">);</span>
<span style="color: #666666;">}</span>
<span style="color: #666666;">}</span>
</pre>
</div>
The Lambda can receive three different inputs (BLUE/RED/YELLOW) based on which it tries to acquire certain tokens every second.<br />
The Lambda again executes in a loop 60 times. To run the Lambda for three different inputs I setup another Cloud Watch Rule<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhXm9gK464wQFSpmDPJv3Nh7dCeFiS7tAnAlzYBYzKsF6U_Dagc_GDg1tf19bwEueZVaCyE_XeDxvTeUelyZKEFvAPI35EU7k2BXeGQU8wYIVRCBLD1fZNgUMgNVfl0EJ1eNXvjbMav1aI/s1600/TokenUserRule.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="396" data-original-width="737" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhXm9gK464wQFSpmDPJv3Nh7dCeFiS7tAnAlzYBYzKsF6U_Dagc_GDg1tf19bwEueZVaCyE_XeDxvTeUelyZKEFvAPI35EU7k2BXeGQU8wYIVRCBLD1fZNgUMgNVfl0EJ1eNXvjbMav1aI/s1600/TokenUserRule.png" /></a></div>
<!-- HTML generated using hilite.me --><br />
<div style="background: #f8f8f8; border-width: 0.1em 0.1em 0.1em 0.8em; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="line-height: 125%; margin: 0;">START RequestId: 960f7fbe-cd6d-4301-97d5-057913abea22 Version: $LATEST
In Handler: Executing TokenBucketUser, $LATEST
Input received RED
Total Runs 60 for RED and available capacities detected [50, 50, 50, 50, 50, 50,</pre>
<pre style="line-height: 125%; margin: 0;"> 50, 50, 50, 50, 50, 50, 50, 50, 50, 50, 50, 50, 50, 50, 50, 50, 50, 50, 50, 50,</pre>
<pre style="line-height: 125%; margin: 0;"> 50, 50, 50, 50, 50, 50, 90, 50, 50, 50, 50, 50, 50, 50, 50, 50, 50, 50, 50, 50,</pre>
<pre style="line-height: 125%; margin: 0;"> 50, 50, 50, 50, 50, 50, 50, 50, 50, 50, 50, 50, 50, 50]
END RequestId: 960f7fbe-cd6d-4301-97d5-057913abea22
REPORT RequestId: 960f7fbe-cd6d-4301-97d5-057913abea22 Duration: 60009.49 ms Billed Duration: 60100 ms Memory Size: 512 MB Max Memory Used: 170 MB
START RequestId: b801600b-724d-4909-999a-029350e49388 Version: $LATEST
</pre>
</div>
Now to watch metrics as the functions execute every minute.<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgufJz2TKKuUKesDtq3zru96JEQ63QtAhCTksnUoSiP-nBOGiejvhVkA-HUH2k5aIFms_oFh0AvpUxloMndZvYgzYoiFRpDyUmZPh3UYBk1OptAg6I1DNzDGzUUm1ngVlJN-bkdDKbylYY/s1600/TBS-CloudWatch.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="289" data-original-width="1142" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgufJz2TKKuUKesDtq3zru96JEQ63QtAhCTksnUoSiP-nBOGiejvhVkA-HUH2k5aIFms_oFh0AvpUxloMndZvYgzYoiFRpDyUmZPh3UYBk1OptAg6I1DNzDGzUUm1ngVlJN-bkdDKbylYY/s1600/TBS-CloudWatch.png" /></a></div>
As seen here, all though the consumers would like to execute at higher rates. But they can only work with TPS available. </div>
Robin Varghesehttp://www.blogger.com/profile/00841917517097216645noreply@blogger.com0tag:blogger.com,1999:blog-6262128735818493674.post-1100970293947254272020-05-17T10:06:00.001+05:302020-05-17T10:06:42.963+05:30Playing with ElastiCache<div dir="ltr" style="text-align: left;" trbidi="on">
I have never used Elastic Cache yet. My caching use cases were solved either via an in memory cache or a hosted cache. I finally got the chance to venture beyond - to cloud caches.<br />
<a name='more'></a><br />
AWS offers two cache solutions (<a href="https://docs.aws.amazon.com/AmazonElastiCache/latest/red-ug/SelectEngine.html" target="_blank">or cache engines</a>) - <b>Redis</b> and <b>Memcached</b>. I have used Memcached before and it has performed well for my use cases.<br />
<div style="background: #f8f8f8; border-width: 0.1em 0.1em 0.1em 0.8em; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="line-height: 125%; margin: 0;">memcached is a high-performance, distributed memory object caching system, </pre>
<pre style="line-height: 125%; margin: 0;">generic in nature, but originally intended for use in speeding up dynamic web </pre>
<pre style="line-height: 125%; margin: 0;">applications by alleviating database load.memcached allows you to take memory </pre>
<pre style="line-height: 125%; margin: 0;">from parts of your system where you have more than you need and make it </pre>
<pre style="line-height: 125%; margin: 0;">accessible to areas where you have less than you need
</pre>
</div>
Redis is far more powerful and versatile. From Redis website
<!-- HTML generated using hilite.me --><br />
<div style="background: #f8f8f8; border-width: 0.1em 0.1em 0.1em 0.8em; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="line-height: 125%; margin: 0;">Redis is an open source (BSD licensed), in-memory data structure store, used as
a database, cache and message broker. It supports data structures such as </pre>
<pre style="line-height: 125%; margin: 0;">strings, hashes, lists, sets, sorted sets with range queries, bitmaps, </pre>
<pre style="line-height: 125%; margin: 0;">hyperloglogs, geospatial indexes with radius queries and streams. Redis has </pre>
<pre style="line-height: 125%; margin: 0;">built-in replication, Lua scripting, LRU eviction, transactions and different </pre>
<pre style="line-height: 125%; margin: 0;">levels of on-disk persistence, and provides high availability via Redis Sentinel </pre>
<pre style="line-height: 125%; margin: 0;">and automatic partitioning with Redis Cluster</pre>
</div>
Wanting to play with the new shiny toy, I decided to experiment with Redis first. I setup a simple cluster that has 1 replica nodes.<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjj4UnR8eMdwAThGtO7GKu6nWdhY9z7DirUoZ_XpCJyn4fhgF6h7VJxJ5c9Ip92XZpYSLDyRBTSybUwZeEks6fSx7rW8vLGQ8QbcplKt9_Pe04hgXEhP13O2Z4V03d-sjU18NtmxQB7lWw/s1600/ElastiCacheRedis.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="662" data-original-width="768" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjj4UnR8eMdwAThGtO7GKu6nWdhY9z7DirUoZ_XpCJyn4fhgF6h7VJxJ5c9Ip92XZpYSLDyRBTSybUwZeEks6fSx7rW8vLGQ8QbcplKt9_Pe04hgXEhP13O2Z4V03d-sjU18NtmxQB7lWw/s1600/ElastiCacheRedis.png" /></a></div>
<br />
As seen my cache is up and running with 1 primary node and 1 replica node. Similar to Kinesis, the data key range can be distributed between shards, In this case I have chosen to setup a single shard for my elasticache (i.e. cluster mode disabled). I have also added a single replica node. So my Cluster is composed of 2 nodes - a primary and a replica or backup node
<!-- HTML generated using hilite.me --><br />
<div style="background: #f8f8f8; border-width: 0.1em 0.1em 0.1em 0.8em; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="line-height: 125%; margin: 0;">If the cluster with replicas has Multi-AZ with Automatic Failover enabled and </pre>
<pre style="line-height: 125%; margin: 0;">the primary node fails, the primary fails over to a read replica. Because the </pre>
<pre style="line-height: 125%; margin: 0;">data is updated on the replica nodes asynchronously, there may be some data loss </pre>
<pre style="line-height: 125%; margin: 0;">due to latency in updating the replica nodes
</pre>
</div>
Our cache is using 'Multi-AZ with Automatic Failover enabled' (this is default option). AW will also take daily backups of my cluster during a scheduled maintenance window. It creates a single .rdb file per shard (One in this case). This <a href="https://docs.aws.amazon.com/AmazonElastiCache/latest/red-ug/Replication.Redis-RedisCluster.html" target="_blank">AWS Link</a> provides good guidelines on when to use which option.<br />
The backup node can also serve read requests ( providing a read scaling behavior).<br />
All the ElastiCache instances are launched within an Amazon VPC. This means you need to provide a subnet group ''
<!-- HTML generated using hilite.me --><br />
<div style="background: #f8f8f8; border-width: 0.1em 0.1em 0.1em 0.8em; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="line-height: 125%; margin: 0;">A subnet group is a collection of subnets (typically private) that you can </pre>
<pre style="line-height: 125%; margin: 0;">designate for your clusters running in an Amazon Virtual Private Cloud (VPC) </pre>
<pre style="line-height: 125%; margin: 0;">environment. When you create a cluster in an Amazon VPC, you must specify a </pre>
<pre style="line-height: 125%; margin: 0;">subnet group. ElastiCache uses that subnet group to choose a subnet and IP </pre>
<pre style="line-height: 125%; margin: 0;">addresses within that subnet to associate with your nodes.
(A subnet is a range of IP addresses in your VPC.)
</pre>
</div>
The endpoints are for<a href="https://docs.aws.amazon.com/AmazonElastiCache/latest/red-ug/Endpoints.html" target="_blank"> connecting to our ElastiCache</a><br />
<div style="background: #f8f8f8; border-width: 0.1em 0.1em 0.1em 0.8em; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="line-height: 125%; margin: 0;">Redis (cluster mode disabled) clusters, use the Primary Endpoint for all write </pre>
<pre style="line-height: 125%; margin: 0;">operations. Use the Reader Endpoint to evenly split incoming connections to the </pre>
<pre style="line-height: 125%; margin: 0;">endpoint between all read replicas. Use the individual Node Endpoints for read </pre>
<pre style="line-height: 125%; margin: 0;">operations</pre>
</div>
Now I want to start writing to the cache. Being a serverless fan, I wanted to try out the integration with a Lambda function.
<br />
My function is pretty straight forward. It runs every 1 second, adding resetting the tokens in a bucket.<br />
<!-- HTML generated using hilite.me --><br />
<div style="background: #f8f8f8; border-width: 0.1em 0.1em 0.1em 0.8em; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<table><tbody>
<tr><td><pre style="line-height: 125%; margin: 0;"> 1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26</pre>
</td><td><pre style="line-height: 125%; margin: 0;"><span style="color: #aa22ff; font-weight: bold;">public</span> <span style="color: #aa22ff; font-weight: bold;">class</span> <span style="color: blue;">TokenBucketRefresherLambdaFn</span> <span style="color: #aa22ff; font-weight: bold;">implements</span> RequestHandler<span style="color: #666666;"><</span>String<span style="color: #666666;">,</span> Void<span style="color: #666666;">></span> <span style="color: #666666;">{</span>
<span style="color: #aa22ff; font-weight: bold;">private</span> <span style="color: #aa22ff; font-weight: bold;">final</span> Jedis jedis<span style="color: #666666;">;</span>
<span style="color: #aa22ff; font-weight: bold;">public</span> <span style="color: #00a000;">TokenBucketRefresherLambdaFn</span><span style="color: #666666;">()</span> <span style="color: #666666;">{</span>
<span style="background-color: yellow;">jedis <span style="color: #666666;">=</span> <span style="color: #aa22ff; font-weight: bold;">new</span> Jedis<span style="color: #666666;">(</span><span style="color: #aa22ff; font-weight: bold;">new</span> HostAndPort<span style="color: #666666;">(</span></span></pre>
<pre style="line-height: 125%; margin: 0;"><span style="color: #bb4444;"> <span style="background-color: yellow;">"redis-test-cache.ql6jf2.ng.0001.use1.cache.amazonaws.com"</span></span><span style="background-color: yellow;"><span style="color: #666666;">,</span><span style="color: #666666;"> </span><span style="color: #666666;">6379));</span></span>
<span style="color: #666666;">}</span>
<span style="color: #aa22ff;">@Override</span>
<span style="color: #aa22ff; font-weight: bold;">public</span> Void <span style="color: #00a000;">handleRequest</span><span style="color: #666666;">(</span>String input<span style="color: #666666;">,</span> Context context<span style="color: #666666;">)</span> <span style="color: #666666;">{</span>
LambdaLogger logger <span style="color: #666666;">=</span> context<span style="color: #666666;">.</span><span style="color: #bb4444;">getLogger</span><span style="color: #666666;">();</span>
logger<span style="color: #666666;">.</span><span style="color: #bb4444;">log</span><span style="color: #666666;">(</span><span style="color: #bb4444;">"In Handler: Executing "</span> <span style="color: #666666;">+</span> context<span style="color: #666666;">.</span><span style="color: #bb4444;">getFunctionName</span><span style="color: #666666;">()</span> </pre>
<pre style="line-height: 125%; margin: 0;"><span style="color: #666666;"> +</span> <span style="color: #bb4444;">", "</span> <span style="color: #666666;">+</span> context<span style="color: #666666;">.</span><span style="color: #bb4444;">getFunctionVersion</span><span style="color: #666666;">());</span>
logger<span style="color: #666666;">.</span><span style="color: #bb4444;">log</span><span style="color: #666666;">(</span>input<span style="color: #666666;">);</span> <span style="color: #008800; font-style: italic;">//This input is not really used now</span>
String token <span style="color: #666666;">=</span> <span style="color: #bb4444;">"TokenCount"</span><span style="color: #666666;">;</span>
<span style="color: #00bb00; font-weight: bold;">int</span> capacity <span style="color: #666666;">=</span> <span style="color: #666666;">100;</span>
String value <span style="color: #666666;">=</span> jedis<span style="color: #666666;">.</span><span style="color: #bb4444;">get</span><span style="color: #666666;">(</span>token<span style="color: #666666;">);</span>
<span style="color: #aa22ff; font-weight: bold;">if</span> <span style="color: #666666;">(</span>value <span style="color: #666666;">==</span> <span style="color: #aa22ff; font-weight: bold;">null</span><span style="color: #666666;">)</span> <span style="color: #666666;">{</span>
logger<span style="color: #666666;">.</span><span style="color: #bb4444;">log</span><span style="color: #666666;">(</span><span style="color: #bb4444;">"No value detected for Key, adding"</span><span style="color: #666666;">);</span>
<span style="color: #666666;">}</span> <span style="color: #aa22ff; font-weight: bold;">else</span> <span style="color: #666666;">{</span>
logger<span style="color: #666666;">.</span><span style="color: #bb4444;">log</span><span style="color: #666666;">(</span><span style="color: #bb4444;">"Value detected for "</span> <span style="color: #666666;">+</span> token <span style="color: #666666;">+</span> <span style="color: #bb4444;">" is "</span> <span style="color: #666666;">+</span> capacity<span style="color: #666666;">);</span>
<span style="color: #666666;">}</span>
<span style="background-color: yellow;">jedis<span style="color: #666666;">.</span><span style="color: #bb4444;">set</span><span style="color: #666666;">(</span>token<span style="color: #666666;">,</span> String<span style="color: #666666;">.</span><span style="color: #bb4444;">valueOf</span><span style="color: #666666;">(</span>capacity<span style="color: #666666;">));</span></span>
<span style="color: #aa22ff; font-weight: bold;">return</span> <span style="color: #aa22ff; font-weight: bold;">null</span><span style="color: #666666;">;</span>
<span style="color: #666666;">}</span>
<span style="color: #666666;">}</span>
</pre>
</td></tr>
</tbody></table>
</div>
Amazon does not provide any AWS library to connect with Redis. I used the Jedis Redis Client instead.
<!-- HTML generated using hilite.me --><br />
<div style="background: #f8f8f8; border-width: 0.1em 0.1em 0.1em 0.8em; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="line-height: 125%; margin: 0;"> <span style="color: green; font-weight: bold;"><dependency></span>
<span style="color: green; font-weight: bold;"><groupId></span>redis.clients<span style="color: green; font-weight: bold;"></groupId></span>
<span style="color: green; font-weight: bold;"><artifactId></span>jedis<span style="color: green; font-weight: bold;"></artifactId></span>
<span style="color: green; font-weight: bold;"><version></span>3.2.0<span style="color: green; font-weight: bold;"></version></span>
<span style="color: green; font-weight: bold;"><type></span>jar<span style="color: green; font-weight: bold;"></type></span>
<span style="color: green; font-weight: bold;"><scope></span>compile<span style="color: green; font-weight: bold;"></scope></span>
<span style="color: green; font-weight: bold;"></dependency></span>
</pre>
</div>
The code is pretty straight forward. It adds tokens to the Bucket. However to run this code, we need the Lambda to be VPC aware. Since my Elasti Cache is setup in the VPC.<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhsQJ3rW7NIAfANeMvEGeyPCw9nYPues3AVkqMBRZwCRzueahp3fXqxldEklxdg4F-2hbqqCk8Vf1l7IPu82uZa6U342fR3lGQLbFhG-J1u3YjACEYmo4BTXLyhOHFGKewr8e9_mheKNkE/s1600/LambdaVpc.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="601" data-original-width="1600" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhsQJ3rW7NIAfANeMvEGeyPCw9nYPues3AVkqMBRZwCRzueahp3fXqxldEklxdg4F-2hbqqCk8Vf1l7IPu82uZa6U342fR3lGQLbFhG-J1u3YjACEYmo4BTXLyhOHFGKewr8e9_mheKNkE/s1600/LambdaVpc.png" /></a></div>
I tested this code through the Lambda console. The Cloud Watch logs were as below:<br />
Run 1:<br />
<div style="background: #f8f8f8; overflow:auto;width:auto;border:solid gray;border-width:.1em .1em .1em .8em;padding:.2em .6em;"><pre style="margin: 0; line-height: 125%">In Handler: Executing TokenBucketRefresherLambda, $LATEST
EXAMPLE
No value detected for Key, adding
END RequestId: 00d372e1-acfa-4d5d-9099-7131e24e7533
REPORT RequestId: 00d372e1-acfa-4d5d-9099-7131e24e7533 Duration: 116.45 ms Billed Duration: 200 ms Memory Size: 512 MB Max Memory Used: 90 MB Init Duration: 371.09 ms
</pre></div>
Run 2:
<!-- HTML generated using hilite.me --><div style="background: #f8f8f8; overflow:auto;width:auto;border:solid gray;border-width:.1em .1em .1em .8em;padding:.2em .6em;"><pre style="margin: 0; line-height: 125%">START RequestId: 0fa896fd-0931-4390-b4db-63eaf442bb4c Version: $LATEST
In Handler: Executing TokenBucketRefresherLambda, $LATEST
EXAMPLE
Value detected for TokenCount is 100
END RequestId: 0fa896fd-0931-4390-b4db-63eaf442bb4c
REPORT RequestId: 0fa896fd-0931-4390-b4db-63eaf442bb4c Duration: 3.86 ms Billed Duration: 100 ms Memory Size: 512 MB Max Memory Used: 91 MB
</pre></div>
Now for Token Bucket, I would schedule this Lambda to run every second refershing the tokens or providing a capacity of 100 Tokens/second
<br /></div>
Robin Varghesehttp://www.blogger.com/profile/00841917517097216645noreply@blogger.com3tag:blogger.com,1999:blog-6262128735818493674.post-60372355428145320962020-05-09T23:58:00.004+05:302020-05-09T23:59:08.228+05:30Playing with the Paths of Step functions<div dir="ltr" style="text-align: left;" trbidi="on">
AWS Step Functions work by moving from one state to another state. States need a way to communicate with each other - DATA needs to be passed between them<br />
<a name='more'></a>All such data needs to be JSON. From the docs
<br />
<pre style="background: #ffffff; color: black;">When a state machine is started, the caller can provide an initial JSON text as input,
which is passed to the machine's start state as input. If no input is provided, the
default is an empty JSON object, {}. As each state is executed, it receives a JSON text
as input and can produce arbitrary output, which MUST be a JSON text. When two states
are linked by a transition, the output from the first state is passed as input to the
second state. The output from the machine's terminal state is treated as its output.
</pre>
For a state to refer to Json data it uses a syntax called <a href="https://github.com/json-path/JsonPath" target="_blank">JsonPath</a><br />
However there is an additional constrain on JsonPath used in Step Functions:<br />
<pre style="background: #ffffff; color: black;">When a state machine is started, the caller can provide an initial JSON text as input,
which is passed to the machine's start state as input. If no input is provided, the
default is an empty JSON object, {}. As each state is executed, it receives a JSON
text as input and can produce arbitrary output, which MUST be a JSON text. When two
states are linked by a transition, the output from the first state is passed as input
to the second state. The output from the machine's terminal state is treated as its output.
</pre>
I decided to play around with the data flow in this post.<br />
I wrote a Car assembly Step function which takes as input customer selection on color and car series and generates a spec chart based on input:<br />
Sample Input :<br />
<!-- HTML generated using hilite.me --><br />
<div style="background: #f8f8f8; border-width: 0.1em 0.1em 0.1em 0.8em; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="line-height: 125%; margin: 0;">{
<span style="color: green; font-weight: bold;">"color"</span>: <span style="color: #bb4444;">"silver"</span>,
<span style="color: green; font-weight: bold;">"class"</span> :<span style="color: #bb4444;">"LX"</span>
}
</pre>
</div>
My Step Function:<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgnermAe4i03kNE0YFXBUUXRb0aFOJcKcAaIiYR9WpD-pF-PrN0sMSCzO1y-Hzu2J5CKtTmyPR1_xztfm_hKEOYT8OwXnz7zmGRDSrleLU8kFJMbSuhghL0s_279uuEeucYsTHa0kowGCM/s1600/stepfunctionsPath.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="753" data-original-width="676" height="640" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgnermAe4i03kNE0YFXBUUXRb0aFOJcKcAaIiYR9WpD-pF-PrN0sMSCzO1y-Hzu2J5CKtTmyPR1_xztfm_hKEOYT8OwXnz7zmGRDSrleLU8kFJMbSuhghL0s_279uuEeucYsTHa0kowGCM/s640/stepfunctionsPath.png" width="574" /></a></div>
My Step Function basically generates the below car configuration based on customer input.
<br />
<div style="background: #f8f8f8; border-width: 0.1em 0.1em 0.1em 0.8em; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="line-height: 125%; margin: 0;">{
<span style="color: green; font-weight: bold;">"car"</span>:{
<span style="color: green; font-weight: bold;">"make"</span>:<span style="color: #bb4444;">"Honda"</span>,
<span style="color: green; font-weight: bold;">"model"</span>:<span style="color: #bb4444;">"Accord"</span>
},
<span style="color: green; font-weight: bold;">"customerChoice"</span>:{
<span style="color: green; font-weight: bold;">"color"</span>:<span style="color: #bb4444;">"silver"</span>,
<span style="color: green; font-weight: bold;">"trim"</span>:<span style="color: #bb4444;">"LX"</span>
},
<span style="color: green; font-weight: bold;">"structure"</span>:{
<span style="color: green; font-weight: bold;">"door"</span>:<span style="color: #bb4444;">"4"</span>,
<span style="color: green; font-weight: bold;">"engine"</span>:<span style="color: #bb4444;">"front"</span>,
<span style="color: green; font-weight: bold;">"body"</span>:<span style="color: #bb4444;">"aluminium"</span>
},
<span style="color: green; font-weight: bold;">"engine"</span>:{
<span style="color: green; font-weight: bold;">"horsepower "</span>:<span style="color: #bb4444;">"192 to 252 hp"</span>,
<span style="color: green; font-weight: bold;">"type "</span>:<span style="color: #bb4444;">"gas"</span>,
<span style="color: green; font-weight: bold;">"brand"</span>:<span style="color: #bb4444;">"turbo"</span>
},
<span style="color: green; font-weight: bold;">"cylinders"</span>:{
<span style="color: green; font-weight: bold;">"transmission "</span>:<span style="color: #bb4444;">"6-speed manual"</span>,
<span style="color: green; font-weight: bold;">"type "</span>:<span style="color: #bb4444;">"Inline"</span>,
<span style="color: green; font-weight: bold;">"count"</span>:<span style="color: #666666;">4</span>
},
<span style="color: green; font-weight: bold;">"tyres"</span>:{
<span style="color: green; font-weight: bold;">"wheelDimension"</span>:<span style="color: #666666;">19</span>,
<span style="color: green; font-weight: bold;">"type "</span>:<span style="color: #bb4444;">"All Season"</span>,
<span style="color: green; font-weight: bold;">"MountedSpareAvailable"</span>:<span style="color: #aa22ff; font-weight: bold;">true</span>
}
}
</pre>
</div>
However this did not go as imagined.<br />
<span style="color: red;">Step Function xml is as below:</span><br />
<pre class="hljs" style="background: rgb(255 , 255 , 255); color: rgb(0 , 0 , 0); display: block; overflow-x: auto; padding: 0.5em;"></pre>
Notice that there are no Lambdas or Activities used here. The Step Function inly uses Pass states. Each state generates a configuration for the car. The step function includes a wait state before it reaches completion.<br />
<br />
<b>Learning No 1: Result field can only refer to static values.</b><br />
For the <span style="background-color: white;">PaintConfigurations state, I was attempting to generate the json based on the input 'color' attribute. </span><br />
<pre class="hljs" style="background: rgb(255 , 255 , 255); color: rgb(0 , 0 , 0); display: block; overflow-x: auto; padding: 0.5em;"> {
<span class="hljs-attr">"color"</span>:<span class="hljs-string" style="color: green; font-weight: 700;">"silver"</span>,
<span class="hljs-attr">"trim"</span>:<span class="hljs-string" style="color: green; font-weight: 700;">"LX"</span>
}</pre>
My Pass state looked something like this:
<br />
<pre class="hljs" style="background: rgb(255 , 255 , 255); color: rgb(0 , 0 , 0); display: block; overflow-x: auto; padding: 0.5em;"> <span class="hljs-string" style="color: green; font-weight: 700;">"PaintConfigurations"</span>: {
<span class="hljs-attr">"Type"</span>: <span class="hljs-string" style="color: green; font-weight: 700;">"Pass"</span>,
<span class="hljs-attr">"Result"</span>: {
<span class="hljs-attr">"color "</span>: <span class="hljs-string" style="color: green; font-weight: 700;">"$.color"</span>,
<span class="hljs-attr">"style "</span>: <span class="hljs-string" style="color: green; font-weight: 700;">"lustre"</span>
},
<span class="hljs-attr">"ResultPath"</span>: <span class="hljs-string" style="color: green; font-weight: 700;">"$.specs.paint"</span>,
<span class="hljs-attr">"Next"</span>: <span class="hljs-string" style="color: green; font-weight: 700;">"SpecialConfigurationsCheck"</span>
},</pre>
However on running this the output comes out as:
<br />
<pre class="hljs" style="background: rgb(255 , 255 , 255); color: rgb(0 , 0 , 0); display: block; overflow-x: auto; padding: 0.5em;"> <span class="hljs-string" style="color: green; font-weight: 700;">"paint"</span>: {
<span class="hljs-attr">"color "</span>: <span class="hljs-string" style="color: green; font-weight: 700;">"$.color"</span>,
<span class="hljs-attr">"style "</span>: <span class="hljs-string" style="color: green; font-weight: 700;">"lustre"</span>
},</pre>
I even tried the color.$ syntax - something I used for AWS Service Integrations, but that only changed my output to be
<br />
<pre style="line-height: 16.25px;"><span style="color: #bb4444;">"paint"</span><span style="border: 1px solid rgb(255, 0, 0);">:</span> {
<span style="color: green; font-weight: bold;">"color.$"</span>: <span style="color: #bb4444;">"$.color"</span>,
<span style="color: green; font-weight: bold;">"style "</span>: <span style="color: #bb4444;">"lustre"</span>
}<span style="border: 1px solid rgb(255, 0, 0);">,</span></pre>
As can be seen Result in Pass State cannot include any evaluation logic. This fits with AWS Docs definition
<br />
<div style="background: #f8f8f8; border-width: 0.1em 0.1em 0.1em 0.8em; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="line-height: 125%; margin: 0;">The Pass State (identified by "Type":"Pass") simply passes its input to its</pre>
<pre style="line-height: 125%; margin: 0;"> output, performing no work.</pre>
</div>
I had a similar requirement with ModelConfigurations where I wanted to include the "trim" information from user input into the result generated by the ModelConfigurations state. However being a Pass state no dynamic work was possible.<br />
<b>Learning 2: Parallel states executions leads to multiple outputs being sent to downstream</b><br />
This is something that I did not see coming. Consider the parallel state we added in our workflow.<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj5Qjmy-fbgSwm8T4tRL_7w6XtkJgi3w5KaDxWK1MoypbHXF8t52LYYdM4d8IkFri3oyz3Fb-TsH3hJJAIv_maD0RKwczvb_un1yTThJMBJ1jn_fuJU1K1qM69Hg-bicbmGFilKnkZPa8I/s1600/stepfunctionsParallelState.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1108" data-original-width="891" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj5Qjmy-fbgSwm8T4tRL_7w6XtkJgi3w5KaDxWK1MoypbHXF8t52LYYdM4d8IkFri3oyz3Fb-TsH3hJJAIv_maD0RKwczvb_un1yTThJMBJ1jn_fuJU1K1qM69Hg-bicbmGFilKnkZPa8I/s1600/stepfunctionsParallelState.png" /></a></div>
<br />
The step that follows the parallel state receives the input after all branches complete.
<br />
<pre class="hljs" style="background: rgb(255 , 255 , 255); color: rgb(0 , 0 , 0); display: block; overflow-x: auto; padding: 0.5em;">A Parallel state causes the interpreter to execute each branch starting with
the state named in its “StartAt” field, as concurrently as possible, and wait
until each branch terminates (reaches a terminal state) before processing
<span class="hljs-code"> the Parallel state's “Next” field.</span></pre>
The input received by the following state is :
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh1t5m6h4RawZGnPihyphenhyphenaYI-UbJLmb8bgRcQzvPM45wznBevAkSdfFINMiOrjjoOOrzrFunSS8c5hNOBEm8QbXseENJDhsNfxxWCnIelYNxa5qE91fXkyv98wg8oGhb7o0kwJBRtDRElATk/s1600/stepfunctionsParallelStateOutput.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1320" data-original-width="974" height="640" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh1t5m6h4RawZGnPihyphenhyphenaYI-UbJLmb8bgRcQzvPM45wznBevAkSdfFINMiOrjjoOOrzrFunSS8c5hNOBEm8QbXseENJDhsNfxxWCnIelYNxa5qE91fXkyv98wg8oGhb7o0kwJBRtDRElATk/s640/stepfunctionsParallelStateOutput.png" width="472" /></a></div>
This broke my Step Function. To solve this I decided to add a combine step here - a Lambda that took the input to generate the output.
Similarly to solve the Paint configuration problem, I decided the same Lambda which will create the Result that I need.
<br />
I also wanted to add some data to a previous pass state result - ModelConfigurations.<br />
Here is the Lambda:
<!-- HTML generated using hilite.me --><br />
<div style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="line-height: 125%; margin: 0px;"><span style="background-color: #f8f8f8; color: #aa22ff; font-weight: bold;">public</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #aa22ff; font-weight: bold;">class</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: blue;">PaintProcessor</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #aa22ff; font-weight: bold;">implements</span><span style="background-color: #f8f8f8;"> RequestHandler</span><span style="background-color: #f8f8f8; color: #666666;"><</span><span style="background-color: #f8f8f8;">List</span><span style="background-color: #f8f8f8; color: #666666;">,</span><span style="background-color: #f8f8f8;"> Map</span><span style="background-color: #f8f8f8; color: #666666;">></span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #666666;">{</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #aa22ff; font-weight: bold;">private</span><span style="background-color: #f8f8f8;"> ObjectMapper mapper </span><span style="background-color: #f8f8f8; color: #666666;">=</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #aa22ff; font-weight: bold;">new</span><span style="background-color: #f8f8f8;"> ObjectMapper</span><span style="background-color: #f8f8f8; color: #666666;">();</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #aa22ff; font-weight: bold;">private</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #aa22ff; font-weight: bold;">final</span><span style="background-color: #f8f8f8;"> String SPECS_KEY </span><span style="background-color: #f8f8f8; color: #666666;">=</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #bb4444;">"specs"</span><span style="background-color: #f8f8f8; color: #666666;">;</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #aa22ff;">@Override</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #aa22ff; font-weight: bold;">public</span><span style="background-color: #f8f8f8;"> Map </span><span style="background-color: #f8f8f8; color: #00a000;">handleRequest</span><span style="background-color: #f8f8f8; color: #666666;">(</span><span style="background-color: #f8f8f8;">List input</span><span style="background-color: #f8f8f8; color: #666666;">,</span><span style="background-color: #f8f8f8;"> Context context</span><span style="background-color: #f8f8f8; color: #666666;">)</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #666666;">{</span><span style="background-color: #f8f8f8;">
LambdaLogger logger </span><span style="background-color: #f8f8f8; color: #666666;">=</span><span style="background-color: #f8f8f8;"> context</span><span style="background-color: #f8f8f8; color: #666666;">.</span><span style="background-color: #f8f8f8; color: #bb4444;">getLogger</span><span style="background-color: #f8f8f8; color: #666666;">();</span><span style="background-color: #f8f8f8;">
logger</span><span style="background-color: #f8f8f8; color: #666666;">.</span><span style="background-color: #f8f8f8; color: #bb4444;">log</span><span style="background-color: #f8f8f8; color: #666666;">(</span><span style="background-color: #f8f8f8; color: #bb4444;">"In Handler: Executing "</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #666666;">+</span><span style="background-color: #f8f8f8;"> context</span><span style="background-color: #f8f8f8; color: #666666;">.</span><span style="background-color: #f8f8f8; color: #bb4444;">getFunctionName</span><span style="background-color: #f8f8f8; color: #666666;">()</span><span style="background-color: #f8f8f8;"> </span></pre>
<pre style="line-height: 125%; margin: 0px;"><span style="background-color: #f8f8f8; color: #666666;"> +</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #bb4444;">", "</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #666666;">+</span><span style="background-color: #f8f8f8;"> context</span><span style="background-color: #f8f8f8; color: #666666;">.</span><span style="background-color: #f8f8f8; color: #bb4444;">getFunctionVersion</span><span style="background-color: #f8f8f8; color: #666666;">());</span><span style="background-color: #f8f8f8;">
logger</span><span style="background-color: #f8f8f8; color: #666666;">.</span><span style="background-color: #f8f8f8; color: #bb4444;">log</span><span style="background-color: #f8f8f8; color: #666666;">(</span><span style="background-color: #f8f8f8;">input</span><span style="background-color: #f8f8f8; color: #666666;">.</span><span style="background-color: #f8f8f8; color: #bb4444;">toString</span><span style="background-color: #f8f8f8; color: #666666;">());</span><span style="background-color: #f8f8f8;">
Map</span><span style="background-color: #f8f8f8; color: #666666;"><</span><span style="background-color: #f8f8f8;">String</span><span style="background-color: #f8f8f8; color: #666666;">,</span><span style="background-color: #f8f8f8;"> Object</span><span style="background-color: #f8f8f8; color: #666666;">></span><span style="background-color: #f8f8f8;"> flatMap </span><span style="background-color: #f8f8f8; color: #666666;">=</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: yellow;">flattenMap<span style="color: #666666;">(</span>input<span style="color: #666666;">);</span></span><span style="background-color: #f8f8f8;">
</span><span style="background-color: yellow;">addPaintConfigurations<span style="color: #666666;">(</span>flatMap<span style="color: #666666;">);</span></span><span style="background-color: #f8f8f8;">
</span><span style="background-color: yellow;">updateTrimConfigurations<span style="color: #666666;">(</span>flatMap<span style="color: #666666;">);</span></span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #aa22ff; font-weight: bold;">return</span><span style="background-color: #f8f8f8;"> flatMap</span><span style="background-color: #f8f8f8; color: #666666;">;</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #666666;">}</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #aa22ff; font-weight: bold;">private</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #00bb00; font-weight: bold;">void</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #00a000;">addPaintConfigurations</span><span style="background-color: #f8f8f8; color: #666666;">(</span><span style="background-color: #f8f8f8;">Map</span><span style="background-color: #f8f8f8; color: #666666;"><</span><span style="background-color: #f8f8f8;">String</span><span style="background-color: #f8f8f8; color: #666666;">,</span><span style="background-color: #f8f8f8;"> Object</span><span style="background-color: #f8f8f8; color: #666666;">></span><span style="background-color: #f8f8f8;"> flatMap</span><span style="background-color: #f8f8f8; color: #666666;">)</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #666666;">{</span><span style="background-color: #f8f8f8;">
Map</span><span style="background-color: #f8f8f8; color: #666666;"><</span><span style="background-color: #f8f8f8;">String</span><span style="background-color: #f8f8f8; color: #666666;">,</span><span style="background-color: #f8f8f8;"> String</span><span style="background-color: #f8f8f8; color: #666666;">></span><span style="background-color: #f8f8f8;"> paintConfigurations </span><span style="background-color: #f8f8f8; color: #666666;">=</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #aa22ff; font-weight: bold;">new</span><span style="background-color: #f8f8f8;"> HashMap</span><span style="background-color: #f8f8f8; color: #666666;"><>();</span><span style="background-color: #f8f8f8;">
paintConfigurations</span><span style="background-color: #f8f8f8; color: #666666;">.</span><span style="background-color: #f8f8f8; color: #bb4444;">put</span><span style="background-color: #f8f8f8; color: #666666;">(</span><span style="background-color: #f8f8f8; color: #bb4444;">"color"</span><span style="background-color: #f8f8f8; color: #666666;">,</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #666666;">(</span><span style="background-color: #f8f8f8;">String</span><span style="background-color: #f8f8f8; color: #666666;">)</span><span style="background-color: #f8f8f8;"> flatMap</span><span style="background-color: #f8f8f8; color: #666666;">.</span><span style="background-color: #f8f8f8; color: #bb4444;">get</span><span style="background-color: #f8f8f8; color: #666666;">(</span><span style="background-color: #f8f8f8; color: #bb4444;">"color"</span><span style="background-color: #f8f8f8; color: #666666;">));</span><span style="background-color: #f8f8f8;">
paintConfigurations</span><span style="background-color: #f8f8f8; color: #666666;">.</span><span style="background-color: #f8f8f8; color: #bb4444;">put</span><span style="background-color: #f8f8f8; color: #666666;">(</span><span style="background-color: #f8f8f8; color: #bb4444;">"style"</span><span style="background-color: #f8f8f8; color: #666666;">,</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #bb4444;">"lustre"</span><span style="background-color: #f8f8f8; color: #666666;">);</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #666666;">((</span><span style="background-color: #f8f8f8;">Map</span><span style="background-color: #f8f8f8; color: #666666;"><</span><span style="background-color: #f8f8f8;">String</span><span style="background-color: #f8f8f8; color: #666666;">,</span><span style="background-color: #f8f8f8;"> Object</span><span style="background-color: #f8f8f8; color: #666666;">>)</span><span style="background-color: #f8f8f8;"> flatMap</span><span style="background-color: #f8f8f8; color: #666666;">.</span><span style="background-color: #f8f8f8; color: #bb4444;">get</span><span style="background-color: #f8f8f8; color: #666666;">(</span><span style="background-color: #f8f8f8;">SPECS_KEY</span><span style="background-color: #f8f8f8; color: #666666;">)).</span><span style="background-color: #f8f8f8; color: #bb4444;">put</span><span style="background-color: #f8f8f8; color: #666666;">(</span><span style="background-color: #f8f8f8; color: #bb4444;">"paint"</span><span style="background-color: #f8f8f8; color: #666666;">,</span></pre>
<pre style="line-height: 125%; margin: 0px;"><span style="background-color: #f8f8f8;"> paintConfigurations</span><span style="background-color: #f8f8f8; color: #666666;">);</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #666666;">}</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #aa22ff; font-weight: bold;">private</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #00bb00; font-weight: bold;">void</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #00a000;">updateTrimConfigurations</span><span style="background-color: #f8f8f8; color: #666666;">(</span><span style="background-color: #f8f8f8;">Map</span><span style="background-color: #f8f8f8; color: #666666;"><</span><span style="background-color: #f8f8f8;">String</span><span style="background-color: #f8f8f8; color: #666666;">,</span><span style="background-color: #f8f8f8;"> Object</span><span style="background-color: #f8f8f8; color: #666666;">></span><span style="background-color: #f8f8f8;"> flatMap</span><span style="background-color: #f8f8f8; color: #666666;">)</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #666666;">{</span><span style="background-color: #f8f8f8;">
Map</span><span style="background-color: #f8f8f8; color: #666666;"><</span><span style="background-color: #f8f8f8;">String</span><span style="background-color: #f8f8f8; color: #666666;">,</span><span style="background-color: #f8f8f8;"> String</span><span style="background-color: #f8f8f8; color: #666666;">></span><span style="background-color: #f8f8f8;"> carConfigurations </span><span style="background-color: #f8f8f8; color: #666666;">=</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #666666;">(</span><span style="background-color: #f8f8f8;">Map</span><span style="background-color: #f8f8f8; color: #666666;"><</span><span style="background-color: #f8f8f8;">String</span><span style="background-color: #f8f8f8; color: #666666;">,</span><span style="background-color: #f8f8f8;"> String</span><span style="background-color: #f8f8f8; color: #666666;">>)</span><span style="background-color: #f8f8f8;"> </span></pre>
<pre style="line-height: 125%; margin: 0px;"><span style="background-color: #f8f8f8; color: #666666;"> ((</span><span style="background-color: #f8f8f8;">Map</span><span style="background-color: #f8f8f8; color: #666666;"><</span><span style="background-color: #f8f8f8;">String</span><span style="background-color: #f8f8f8; color: #666666;">,</span><span style="background-color: #f8f8f8;"> Object</span><span style="background-color: #f8f8f8; color: #666666;">>) </span><span style="background-color: #f8f8f8;">flatMap</span><span style="background-color: #f8f8f8; color: #666666;">.</span><span style="background-color: #f8f8f8; color: #bb4444;">get</span><span style="background-color: #f8f8f8; color: #666666;">(</span><span style="background-color: #f8f8f8;">SPECS_KEY</span><span style="background-color: #f8f8f8; color: #666666;">)).</span><span style="background-color: #f8f8f8; color: #bb4444;">get</span><span style="background-color: #f8f8f8; color: #666666;">(</span><span style="background-color: #f8f8f8; color: #bb4444;">"car"</span><span style="background-color: #f8f8f8; color: #666666;">);</span><span style="background-color: #f8f8f8;">
carConfigurations</span><span style="background-color: #f8f8f8; color: #666666;">.</span><span style="background-color: #f8f8f8; color: #bb4444;">put</span><span style="background-color: #f8f8f8; color: #666666;">(</span><span style="background-color: #f8f8f8; color: #bb4444;">"trim"</span><span style="background-color: #f8f8f8; color: #666666;">,</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #666666;">(</span><span style="background-color: #f8f8f8;">String</span><span style="background-color: #f8f8f8; color: #666666;">)</span><span style="background-color: #f8f8f8;"> flatMap</span><span style="background-color: #f8f8f8; color: #666666;">.</span><span style="background-color: #f8f8f8; color: #bb4444;">get</span><span style="background-color: #f8f8f8; color: #666666;">(</span><span style="background-color: #f8f8f8; color: #bb4444;">"class"</span><span style="background-color: #f8f8f8; color: #666666;">));</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #666666;">}</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #aa22ff; font-weight: bold;">private</span><span style="background-color: #f8f8f8;"> Map</span><span style="background-color: #f8f8f8; color: #666666;"><</span><span style="background-color: #f8f8f8;">String</span><span style="background-color: #f8f8f8; color: #666666;">,</span><span style="background-color: #f8f8f8;"> Object</span><span style="background-color: #f8f8f8; color: #666666;">></span><span style="background-color: #f8f8f8;"> flattenMap</span><span style="background-color: #f8f8f8; color: #666666;">(</span><span style="background-color: #f8f8f8;">List</span><span style="background-color: #f8f8f8; color: #666666;"><</span><span style="background-color: #f8f8f8;">Map</span><span style="background-color: #f8f8f8; color: #666666;"><</span><span style="background-color: #f8f8f8;">String</span><span style="background-color: #f8f8f8; color: #666666;">,</span><span style="background-color: #f8f8f8;"> Object</span><span style="background-color: #f8f8f8; color: #666666;">>></span><span style="background-color: #f8f8f8;"> </span></pre>
<pre style="line-height: 125%; margin: 0px;"><span style="background-color: #f8f8f8;"> inputMessages</span><span style="background-color: #f8f8f8; color: #666666;">)</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #666666;">{</span><span style="background-color: #f8f8f8;">
Map</span><span style="background-color: #f8f8f8; color: #666666;"><</span><span style="background-color: #f8f8f8;">String</span><span style="background-color: #f8f8f8; color: #666666;">,</span><span style="background-color: #f8f8f8;"> Object</span><span style="background-color: #f8f8f8; color: #666666;">></span><span style="background-color: #f8f8f8;"> result </span><span style="background-color: #f8f8f8; color: #666666;">=</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #aa22ff; font-weight: bold;">new</span><span style="background-color: #f8f8f8;"> HashMap</span><span style="background-color: #f8f8f8; color: #666666;"><>();</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #aa22ff; font-weight: bold;">for</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #666666;">(</span><span style="background-color: #f8f8f8;">Map inputKey </span><span style="background-color: #f8f8f8; color: #666666;">:</span><span style="background-color: #f8f8f8;"> inputMessages</span><span style="background-color: #f8f8f8; color: #666666;">)</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #666666;">{</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #aa22ff; font-weight: bold;">for</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #666666;">(</span><span style="background-color: #f8f8f8;">Object key </span><span style="background-color: #f8f8f8; color: #666666;">:</span><span style="background-color: #f8f8f8;"> inputKey</span><span style="background-color: #f8f8f8; color: #666666;">.</span><span style="background-color: #f8f8f8; color: #bb4444;">keySet</span><span style="background-color: #f8f8f8; color: #666666;">())</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #666666;">{</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #aa22ff; font-weight: bold;">if</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #666666;">(</span><span style="background-color: #f8f8f8;">key</span><span style="background-color: #f8f8f8; color: #666666;">.</span><span style="background-color: #f8f8f8; color: #bb4444;">equals</span><span style="background-color: #f8f8f8; color: #666666;">(</span><span style="background-color: #f8f8f8;">SPECS_KEY</span><span style="background-color: #f8f8f8; color: #666666;">))</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #666666;">{</span><span style="background-color: #f8f8f8;">
Map specMap </span><span style="background-color: #f8f8f8; color: #666666;">=</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #666666;">(</span><span style="background-color: #f8f8f8;">Map</span><span style="background-color: #f8f8f8; color: #666666;">)</span><span style="background-color: #f8f8f8;"> inputKey</span><span style="background-color: #f8f8f8; color: #666666;">.</span><span style="background-color: #f8f8f8; color: #bb4444;">get</span><span style="background-color: #f8f8f8; color: #666666;">(</span><span style="background-color: #f8f8f8;">SPECS_KEY</span><span style="background-color: #f8f8f8; color: #666666;">);</span><span style="background-color: #f8f8f8;">
Map existingSpecMap </span><span style="background-color: #f8f8f8; color: #666666;">=</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #666666;">(</span><span style="background-color: #f8f8f8;">Map</span><span style="background-color: #f8f8f8; color: #666666;">)</span><span style="background-color: #f8f8f8;"> result</span><span style="background-color: #f8f8f8; color: #666666;">.</span><span style="background-color: #f8f8f8; color: #bb4444;">get</span><span style="background-color: #f8f8f8; color: #666666;">(</span><span style="background-color: #f8f8f8;">SPECS_KEY</span><span style="background-color: #f8f8f8; color: #666666;">);</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #aa22ff; font-weight: bold;">if</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #666666;">(</span><span style="background-color: #f8f8f8;">existingSpecMap </span><span style="background-color: #f8f8f8; color: #666666;">==</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #aa22ff; font-weight: bold;">null</span><span style="background-color: #f8f8f8; color: #666666;">)</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #666666;">{</span><span style="background-color: #f8f8f8;">
result</span><span style="background-color: #f8f8f8; color: #666666;">.</span><span style="background-color: #f8f8f8; color: #bb4444;">put</span><span style="background-color: #f8f8f8; color: #666666;">(</span><span style="background-color: #f8f8f8;">SPECS_KEY</span><span style="background-color: #f8f8f8; color: #666666;">,</span><span style="background-color: #f8f8f8;"> specMap</span><span style="background-color: #f8f8f8; color: #666666;">);</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #666666;">}</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #aa22ff; font-weight: bold;">else</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #666666;">{</span><span style="background-color: #f8f8f8;">
existingSpecMap</span><span style="background-color: #f8f8f8; color: #666666;">.</span><span style="background-color: #f8f8f8; color: #bb4444;">putAll</span><span style="background-color: #f8f8f8; color: #666666;">(</span><span style="background-color: #f8f8f8;">specMap</span><span style="background-color: #f8f8f8; color: #666666;">);</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #666666;">}</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #666666;">}</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #aa22ff; font-weight: bold;">else</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #666666;">{</span><span style="background-color: #f8f8f8;">
result</span><span style="background-color: #f8f8f8; color: #666666;">.</span><span style="background-color: #f8f8f8; color: #bb4444;">put</span><span style="background-color: #f8f8f8; color: #666666;">(</span><span style="background-color: #f8f8f8;">key</span><span style="background-color: #f8f8f8; color: #666666;">.</span><span style="background-color: #f8f8f8; color: #bb4444;">toString</span><span style="background-color: #f8f8f8; color: #666666;">(),</span><span style="background-color: #f8f8f8;"> inputKey</span><span style="background-color: #f8f8f8; color: #666666;">.</span><span style="background-color: #f8f8f8; color: #bb4444;">get</span><span style="background-color: #f8f8f8; color: #666666;">(</span><span style="background-color: #f8f8f8;">key</span><span style="background-color: #f8f8f8; color: #666666;">));</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #666666;">}</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #666666;">}</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #666666;">}</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #aa22ff; font-weight: bold;">return</span><span style="background-color: #f8f8f8;"> result</span><span style="background-color: #f8f8f8; color: #666666;">;</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #666666;">}</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #666666;">}</span><span style="background-color: #f8f8f8;">
</span></pre>
</div>
The Lambda gets the List of specs from the parallel execution which it merges and then adds the paint configuration. It then pulls in the trim information into car configuration.<br />
The output from My Lambda is:<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhetawMTlOaimA4M1QEnEu2rZaTqA5Kd8QUaQdjqIhS-vTFrjp9HRwiyf0iKnO170MBFkJn4jkHJ_iAWHwEZ1Tqx3i-cOZfrFbg0FNip7-pR5VREkXj_pW38Y6Gq3h_DpT-ygXIYbLM08A/s1600/LambdaResult.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1234" data-original-width="1547" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhetawMTlOaimA4M1QEnEu2rZaTqA5Kd8QUaQdjqIhS-vTFrjp9HRwiyf0iKnO170MBFkJn4jkHJ_iAWHwEZ1Tqx3i-cOZfrFbg0FNip7-pR5VREkXj_pW38Y6Gq3h_DpT-ygXIYbLM08A/s1600/LambdaResult.png" /></a></div>
<br />
<b>Learning 3: Wait States can be dynamic.</b><br />
I initially setup my wait state to be simple 2 second wait. But there is also a configuration to make the wait state wait depending on the input to the state. I updated my State machine input to include a wait field<br />
<!-- HTML generated using hilite.me --><br />
<div style="background: #f8f8f8; border-width: 0.1em 0.1em 0.1em 0.8em; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="line-height: 125%; margin: 0;"><span style="color: #bb4444;">"WaitForPublish"</span><span style="border: 1px solid #FF0000;">:</span> {
<span style="color: green; font-weight: bold;">"Type"</span>: <span style="color: #bb4444;">"Wait"</span>,
<span style="color: green; font-weight: bold;">"SecondsPath"</span>: <span style="color: #bb4444;">"$.pubDelay"</span>,
<span style="color: green; font-weight: bold;">"Next"</span>: <span style="color: #bb4444;">"FinishSpecs"</span>
}
</pre>
</div>
<b><br /></b>
<b>Learning 4: Output Path versus State Path</b><br />
There is a good pictorial representation on the various Path variables available to a State <a href="https://docs.aws.amazon.com/step-functions/latest/dg/concepts-input-output-filtering.html" target="_blank">here</a>.<!-- HTML generated using hilite.me --><br />
<div style="background: #f8f8f8; border-width: 0.1em 0.1em 0.1em 0.8em; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="line-height: 125%; margin: 0;">The output of a state can be a copy of its input, the result it produces (for </pre>
<pre style="line-height: 125%; margin: 0;">example, output from a Task state’s Lambda function), or a combination of its </pre>
<pre style="line-height: 125%; margin: 0;">input and result. Use ResultPath to control which combination of these is passed </pre>
<pre style="line-height: 125%; margin: 0;">to the state output.
</pre>
</div>
For most of my states I used the Result Path to include my State's input with the output (See <a href="https://docs.aws.amazon.com/step-functions/latest/dg/input-output-resultpath.html" target="_blank">here</a> for ResultPath capabilities).<br />
For my last state however, I did not want to output the whole Result till now, but only a selection of the result. This was achieved using the OutputPath<br />
<div style="background: #f8f8f8; border-width: 0.1em 0.1em 0.1em 0.8em; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="line-height: 125%; margin: 0;">OutputPath enables you to select a portion of the state output to pass to the </pre>
<pre style="line-height: 125%; margin: 0;">next state. This enables you to filter out unwanted information, and pass only </pre>
<pre style="line-height: 125%; margin: 0;">the portion of JSON that you care about.
</pre>
</div>
This is how the final state looks.
<!-- HTML generated using hilite.me --><br />
<div style="background: #f8f8f8; border-width: 0.1em 0.1em 0.1em 0.8em; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="line-height: 125%; margin: 0;"><span style="color: #bb4444;">"FinishSpecs"</span><span style="border: 1px solid #FF0000;">:</span> {
<span style="color: green; font-weight: bold;">"Type"</span>: <span style="color: #bb4444;">"Pass"</span>,
<span style="color: green; font-weight: bold;">"OutputPath"</span>: <span style="color: #bb4444;">"$.specs"</span>,
<span style="color: green; font-weight: bold;">"End"</span>: <span style="color: #aa22ff; font-weight: bold;">true</span>
}
</pre>
</div>
It takes the existing input to the state and only retains the value represented by my specs path.
<br />
<b>My final State Machine is as below:</b>
<!-- HTML generated using hilite.me --><br />
<div style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="line-height: 125%; margin: 0px;"><span style="background-color: #f8f8f8;">{
</span><span style="background-color: #f8f8f8; color: green; font-weight: bold;">"Comment"</span><span style="background-color: #f8f8f8;">: </span><span style="background-color: #f8f8f8; color: #bb4444;">"Honda Accord Assembly"</span><span style="background-color: #f8f8f8;">,
</span><span style="background-color: #f8f8f8; color: green; font-weight: bold;">"StartAt"</span><span style="background-color: #f8f8f8;">: </span><span style="background-color: #f8f8f8; color: #bb4444;">"ModelConfigurations"</span><span style="background-color: #f8f8f8;">,
</span><span style="background-color: #f8f8f8; color: green; font-weight: bold;">"States"</span><span style="background-color: #f8f8f8;">: {
</span><span style="background-color: #f8f8f8; color: green; font-weight: bold;">"ModelConfigurations"</span><span style="background-color: #f8f8f8;">: {
</span><span style="background-color: #f8f8f8; color: green; font-weight: bold;">"Type"</span><span style="background-color: #f8f8f8;">: </span><span style="background-color: #f8f8f8; color: #bb4444;">"Pass"</span><span style="background-color: #f8f8f8;">,
</span><span style="background-color: #f8f8f8; color: green; font-weight: bold;">"Result"</span><span style="background-color: #f8f8f8;">: {
</span><span style="background-color: #f8f8f8; color: green; font-weight: bold;">"make"</span><span style="background-color: #f8f8f8;">: </span><span style="background-color: #f8f8f8; color: #bb4444;">"Honda"</span><span style="background-color: #f8f8f8;">,
</span><span style="background-color: #f8f8f8; color: green; font-weight: bold;">"model"</span><span style="background-color: #f8f8f8;">: </span><span style="background-color: #f8f8f8; color: #bb4444;">"Accord"</span><span style="background-color: #f8f8f8;">
},
</span><span style="background-color: #f8f8f8; color: green; font-weight: bold;">"ResultPath"</span><span style="background-color: #f8f8f8;">: </span><span style="background-color: #f8f8f8; color: #bb4444;">"$.specs.car"</span><span style="background-color: #f8f8f8;">,
</span><span style="background-color: #f8f8f8; color: green; font-weight: bold;">"Next"</span><span style="background-color: #f8f8f8;">: </span><span style="background-color: #f8f8f8; color: #bb4444;">"StructureConfigurations"</span><span style="background-color: #f8f8f8;">
},
</span><span style="background-color: #f8f8f8; color: green; font-weight: bold;">"StructureConfigurations"</span><span style="background-color: #f8f8f8;">: {
</span><span style="background-color: #f8f8f8; color: green; font-weight: bold;">"Type"</span><span style="background-color: #f8f8f8;">: </span><span style="background-color: #f8f8f8; color: #bb4444;">"Pass"</span><span style="background-color: #f8f8f8;">,
</span><span style="background-color: #f8f8f8; color: green; font-weight: bold;">"Result"</span><span style="background-color: #f8f8f8;">: {
</span><span style="background-color: #f8f8f8; color: green; font-weight: bold;">"door"</span><span style="background-color: #f8f8f8;">: </span><span style="background-color: #f8f8f8; color: #bb4444;">"4"</span><span style="background-color: #f8f8f8;">,
</span><span style="background-color: #f8f8f8; color: green; font-weight: bold;">"engine"</span><span style="background-color: #f8f8f8;">: </span><span style="background-color: #f8f8f8; color: #bb4444;">"front"</span><span style="background-color: #f8f8f8;">,
</span><span style="background-color: #f8f8f8; color: green; font-weight: bold;">"body"</span><span style="background-color: #f8f8f8;">: </span><span style="background-color: #f8f8f8; color: #bb4444;">"aluminium"</span><span style="background-color: #f8f8f8;">
},
</span><span style="background-color: #f8f8f8; color: green; font-weight: bold;">"ResultPath"</span><span style="background-color: #f8f8f8;">: </span><span style="background-color: #f8f8f8; color: #bb4444;">"$.specs.structure"</span><span style="background-color: #f8f8f8;">,
</span><span style="background-color: #f8f8f8; color: green; font-weight: bold;">"Next"</span><span style="background-color: #f8f8f8;">: </span><span style="background-color: #f8f8f8; color: #bb4444;">"InternalsCombination"</span><span style="background-color: #f8f8f8;">
},
</span><span style="background-color: #f8f8f8; color: green; font-weight: bold;">"InternalsCombination"</span><span style="background-color: #f8f8f8;">: {
</span><span style="background-color: #f8f8f8; color: green; font-weight: bold;">"Type"</span><span style="background-color: #f8f8f8;">: </span><span style="background-color: #f8f8f8; color: #bb4444;">"Parallel"</span><span style="background-color: #f8f8f8;">,
</span><span style="background-color: #f8f8f8; color: green; font-weight: bold;">"Branches"</span><span style="background-color: #f8f8f8;">: [
{
</span><span style="background-color: #f8f8f8; color: green; font-weight: bold;">"StartAt"</span><span style="background-color: #f8f8f8;">: </span><span style="background-color: #f8f8f8; color: #bb4444;">"EngineConfigurations"</span><span style="background-color: #f8f8f8;">,
</span><span style="background-color: #f8f8f8; color: green; font-weight: bold;">"States"</span><span style="background-color: #f8f8f8;">: {
</span><span style="background-color: #f8f8f8; color: green; font-weight: bold;">"EngineConfigurations"</span><span style="background-color: #f8f8f8;">: {
</span><span style="background-color: #f8f8f8; color: green; font-weight: bold;">"Type"</span><span style="background-color: #f8f8f8;">: </span><span style="background-color: #f8f8f8; color: #bb4444;">"Pass"</span><span style="background-color: #f8f8f8;">,
</span><span style="background-color: #f8f8f8; color: green; font-weight: bold;">"Result"</span><span style="background-color: #f8f8f8;">: {
</span><span style="background-color: #f8f8f8; color: green; font-weight: bold;">"horsepower"</span><span style="background-color: #f8f8f8;">: </span><span style="background-color: #f8f8f8; color: #bb4444;">"192 to 252 hp"</span><span style="background-color: #f8f8f8;">,
</span><span style="background-color: #f8f8f8; color: green; font-weight: bold;">"type "</span><span style="background-color: #f8f8f8;">: </span><span style="background-color: #f8f8f8; color: #bb4444;">"gas"</span><span style="background-color: #f8f8f8;">,
</span><span style="background-color: #f8f8f8; color: green; font-weight: bold;">"brand"</span><span style="background-color: #f8f8f8;">: </span><span style="background-color: #f8f8f8; color: #bb4444;">"turbo"</span><span style="background-color: #f8f8f8;">
},
</span><span style="background-color: #f8f8f8; color: green; font-weight: bold;">"ResultPath"</span><span style="background-color: #f8f8f8;">: </span><span style="background-color: #f8f8f8; color: #bb4444;">"$.specs.engine"</span><span style="background-color: #f8f8f8;">,
</span><span style="background-color: #f8f8f8; color: green; font-weight: bold;">"Next"</span><span style="background-color: #f8f8f8;">: </span><span style="background-color: #f8f8f8; color: #bb4444;">"CylinderConfiguration"</span><span style="background-color: #f8f8f8;">
},
</span><span style="background-color: #f8f8f8; color: green; font-weight: bold;">"CylinderConfiguration"</span><span style="background-color: #f8f8f8;">: {
</span><span style="background-color: #f8f8f8; color: green; font-weight: bold;">"Type"</span><span style="background-color: #f8f8f8;">: </span><span style="background-color: #f8f8f8; color: #bb4444;">"Pass"</span><span style="background-color: #f8f8f8;">,
</span><span style="background-color: #f8f8f8; color: green; font-weight: bold;">"Result"</span><span style="background-color: #f8f8f8;">: {
</span><span style="background-color: #f8f8f8; color: green; font-weight: bold;">"transmission "</span><span style="background-color: #f8f8f8;">: </span><span style="background-color: #f8f8f8; color: #bb4444;">"6-speed manual"</span><span style="background-color: #f8f8f8;">,
</span><span style="background-color: #f8f8f8; color: green; font-weight: bold;">"type "</span><span style="background-color: #f8f8f8;">: </span><span style="background-color: #f8f8f8; color: #bb4444;">"Inline"</span><span style="background-color: #f8f8f8;">,
</span><span style="background-color: #f8f8f8; color: green; font-weight: bold;">"count"</span><span style="background-color: #f8f8f8;">: </span><span style="background-color: #f8f8f8; color: #666666;">4</span><span style="background-color: #f8f8f8;">
},
</span><span style="background-color: #f8f8f8; color: green; font-weight: bold;">"ResultPath"</span><span style="background-color: #f8f8f8;">: </span><span style="background-color: #f8f8f8; color: #bb4444;">"$.specs.cylinders"</span><span style="background-color: #f8f8f8;">,
</span><span style="background-color: #f8f8f8; color: green; font-weight: bold;">"End"</span><span style="background-color: #f8f8f8;">: </span><span style="background-color: #f8f8f8; color: #aa22ff; font-weight: bold;">true</span><span style="background-color: #f8f8f8;">
}
}
},
{
</span><span style="background-color: #f8f8f8; color: green; font-weight: bold;">"StartAt"</span><span style="background-color: #f8f8f8;">: </span><span style="background-color: #f8f8f8; color: #bb4444;">"TyreConfiguration"</span><span style="background-color: #f8f8f8;">,
</span><span style="background-color: #f8f8f8; color: green; font-weight: bold;">"States"</span><span style="background-color: #f8f8f8;">: {
</span><span style="background-color: #f8f8f8; color: green; font-weight: bold;">"TyreConfiguration"</span><span style="background-color: #f8f8f8;">: {
</span><span style="background-color: #f8f8f8; color: green; font-weight: bold;">"Type"</span><span style="background-color: #f8f8f8;">: </span><span style="background-color: #f8f8f8; color: #bb4444;">"Pass"</span><span style="background-color: #f8f8f8;">,
</span><span style="background-color: #f8f8f8; color: green; font-weight: bold;">"Result"</span><span style="background-color: #f8f8f8;">: {
</span><span style="background-color: #f8f8f8; color: green; font-weight: bold;">"wheelDimension"</span><span style="background-color: #f8f8f8;">: </span><span style="background-color: #f8f8f8; color: #666666;">19</span><span style="background-color: #f8f8f8;">,
</span><span style="background-color: #f8f8f8; color: green; font-weight: bold;">"type "</span><span style="background-color: #f8f8f8;">: </span><span style="background-color: #f8f8f8; color: #bb4444;">"All Season"</span><span style="background-color: #f8f8f8;">,
</span><span style="background-color: #f8f8f8; color: green; font-weight: bold;">"MountedSpareAvailable"</span><span style="background-color: #f8f8f8;">: </span><span style="background-color: #f8f8f8; color: #aa22ff; font-weight: bold;">true</span><span style="background-color: #f8f8f8;">
},
</span><span style="background-color: #f8f8f8; color: green; font-weight: bold;">"ResultPath"</span><span style="background-color: #f8f8f8;">: </span><span style="background-color: #f8f8f8; color: #bb4444;">"$.specs.tyres"</span><span style="background-color: #f8f8f8;">,
</span><span style="background-color: #f8f8f8; color: green; font-weight: bold;">"End"</span><span style="background-color: #f8f8f8;">: </span><span style="background-color: #f8f8f8; color: #aa22ff; font-weight: bold;">true</span><span style="background-color: #f8f8f8;">
}
}
}
],
</span><span style="background-color: #f8f8f8; color: green; font-weight: bold;">"Next"</span><span style="background-color: #f8f8f8;">: </span><span style="background-color: #f8f8f8; color: #bb4444;">"PaintConfigurations"</span><span style="background-color: #f8f8f8;">
},
</span><span style="background-color: #f8f8f8; color: green; font-weight: bold;">"PaintConfigurations"</span><span style="background-color: #f8f8f8;">: {
</span><span style="background-color: #f8f8f8; color: green; font-weight: bold;">"Type"</span><span style="background-color: #f8f8f8;">: </span><span style="background-color: #f8f8f8; color: #bb4444;">"Task"</span><span style="background-color: #f8f8f8;">,
</span><span style="background-color: #f8f8f8; color: green; font-weight: bold;">"Resource"</span><span style="background-color: #f8f8f8;">: </span><span style="color: #bb4444;"><span style="background-color: #f8f8f8;">"</span><span style="background-color: yellow;">arn:aws:lambda:us-east-1:ActNo:function:ConfigureAndPaint:$LATEST"</span></span><span style="background-color: #f8f8f8;">,
</span><span style="background-color: #f8f8f8; color: green; font-weight: bold;">"Next"</span><span style="background-color: #f8f8f8;">: </span><span style="background-color: #f8f8f8; color: #bb4444;">"SpecialConfigurationsCheck"</span><span style="background-color: #f8f8f8;">
},
</span><span style="background-color: #f8f8f8; color: green; font-weight: bold;">"SpecialConfigurationsCheck"</span><span style="background-color: #f8f8f8;">: {
</span><span style="background-color: #f8f8f8; color: green; font-weight: bold;">"Type"</span><span style="background-color: #f8f8f8;">: </span><span style="background-color: #f8f8f8; color: #bb4444;">"Choice"</span><span style="background-color: #f8f8f8;">,
</span><span style="background-color: #f8f8f8; color: green; font-weight: bold;">"Choices"</span><span style="background-color: #f8f8f8;">: [
{
</span><span style="background-color: #f8f8f8; color: green; font-weight: bold;">"Not"</span><span style="background-color: #f8f8f8;">: {
</span><span style="background-color: #f8f8f8; color: green; font-weight: bold;">"And"</span><span style="background-color: #f8f8f8;">: [
{
</span><span style="background-color: #f8f8f8; color: green; font-weight: bold;">"Variable"</span><span style="background-color: #f8f8f8;">: </span><span style="background-color: #f8f8f8; color: #bb4444;">"$.class"</span><span style="background-color: #f8f8f8;">,
</span><span style="background-color: #f8f8f8; color: green; font-weight: bold;">"StringEquals"</span><span style="background-color: #f8f8f8;">: </span><span style="background-color: #f8f8f8; color: #bb4444;">"EX"</span><span style="background-color: #f8f8f8;">
},
{
</span><span style="background-color: #f8f8f8; color: green; font-weight: bold;">"Variable"</span><span style="background-color: #f8f8f8;">: </span><span style="background-color: #f8f8f8; color: #bb4444;">"$.class"</span><span style="background-color: #f8f8f8;">,
</span><span style="background-color: #f8f8f8; color: green; font-weight: bold;">"StringEquals"</span><span style="background-color: #f8f8f8;">: </span><span style="background-color: #f8f8f8; color: #bb4444;">"VX"</span><span style="background-color: #f8f8f8;">
}
]
},
</span><span style="background-color: #f8f8f8; color: green; font-weight: bold;">"Next"</span><span style="background-color: #f8f8f8;">: </span><span style="background-color: #f8f8f8; color: #bb4444;">"StandardConfigurations"</span><span style="background-color: #f8f8f8;">
},
{
</span><span style="background-color: #f8f8f8; color: green; font-weight: bold;">"Variable"</span><span style="background-color: #f8f8f8;">: </span><span style="background-color: #f8f8f8; color: #bb4444;">"$.class"</span><span style="background-color: #f8f8f8;">,
</span><span style="background-color: #f8f8f8; color: green; font-weight: bold;">"StringEquals"</span><span style="background-color: #f8f8f8;">: </span><span style="background-color: #f8f8f8; color: #bb4444;">"EX"</span><span style="background-color: #f8f8f8;">,
</span><span style="background-color: #f8f8f8; color: green; font-weight: bold;">"Next"</span><span style="background-color: #f8f8f8;">: </span><span style="background-color: #f8f8f8; color: #bb4444;">"EXConfigurations"</span><span style="background-color: #f8f8f8;">
},
{
</span><span style="background-color: #f8f8f8; color: green; font-weight: bold;">"Variable"</span><span style="background-color: #f8f8f8;">: </span><span style="background-color: #f8f8f8; color: #bb4444;">"$.class"</span><span style="background-color: #f8f8f8;">,
</span><span style="background-color: #f8f8f8; color: green; font-weight: bold;">"StringEquals"</span><span style="background-color: #f8f8f8;">: </span><span style="background-color: #f8f8f8; color: #bb4444;">"VX"</span><span style="background-color: #f8f8f8;">,
</span><span style="background-color: #f8f8f8; color: green; font-weight: bold;">"Next"</span><span style="background-color: #f8f8f8;">: </span><span style="background-color: #f8f8f8; color: #bb4444;">"VXConfigurations"</span><span style="background-color: #f8f8f8;">
}
],
</span><span style="background-color: #f8f8f8; color: green; font-weight: bold;">"Default"</span><span style="background-color: #f8f8f8;">: </span><span style="background-color: #f8f8f8; color: #bb4444;">"SpecialConfigCheckErrorState"</span><span style="background-color: #f8f8f8;">
},
</span><span style="background-color: #f8f8f8; color: green; font-weight: bold;">"StandardConfigurations"</span><span style="background-color: #f8f8f8;">: {
</span><span style="background-color: #f8f8f8; color: green; font-weight: bold;">"Type"</span><span style="background-color: #f8f8f8;">: </span><span style="background-color: #f8f8f8; color: #bb4444;">"Pass"</span><span style="background-color: #f8f8f8;">,
</span><span style="background-color: #f8f8f8; color: green; font-weight: bold;">"Next"</span><span style="background-color: #f8f8f8;">: </span><span style="background-color: #f8f8f8; color: #bb4444;">"WaitForPublish"</span><span style="background-color: #f8f8f8;">,
</span><span style="background-color: #f8f8f8; color: green; font-weight: bold;">"Result"</span><span style="background-color: #f8f8f8;">: {
</span><span style="background-color: #f8f8f8; color: green; font-weight: bold;">"Alloy Wheels"</span><span style="background-color: #f8f8f8;">: </span><span style="background-color: #f8f8f8; color: #aa22ff; font-weight: bold;">false</span><span style="background-color: #f8f8f8;">,
</span><span style="background-color: #f8f8f8; color: green; font-weight: bold;">"WarrantyYears"</span><span style="background-color: #f8f8f8;">: </span><span style="background-color: #f8f8f8; color: #666666;">2</span><span style="background-color: #f8f8f8;">,
</span><span style="background-color: #f8f8f8; color: green; font-weight: bold;">"seatCover"</span><span style="background-color: #f8f8f8;">: </span><span style="background-color: #f8f8f8; color: #bb4444;">"cloth"</span><span style="background-color: #f8f8f8;">
},
</span><span style="background-color: #f8f8f8; color: green; font-weight: bold;">"ResultPath"</span><span style="background-color: #f8f8f8;">: </span><span style="background-color: #f8f8f8; color: #bb4444;">"$.specs.extra"</span><span style="background-color: #f8f8f8;">
},
</span><span style="background-color: #f8f8f8; color: green; font-weight: bold;">"EXConfigurations"</span><span style="background-color: #f8f8f8;">: {
</span><span style="background-color: #f8f8f8; color: green; font-weight: bold;">"Type"</span><span style="background-color: #f8f8f8;">: </span><span style="background-color: #f8f8f8; color: #bb4444;">"Pass"</span><span style="background-color: #f8f8f8;">,
</span><span style="background-color: #f8f8f8; color: green; font-weight: bold;">"Next"</span><span style="background-color: #f8f8f8;">: </span><span style="background-color: #f8f8f8; color: #bb4444;">"WaitForPublish"</span><span style="background-color: #f8f8f8;">,
</span><span style="background-color: #f8f8f8; color: green; font-weight: bold;">"Result"</span><span style="background-color: #f8f8f8;">: {
</span><span style="background-color: #f8f8f8; color: green; font-weight: bold;">"Alloy Wheels"</span><span style="background-color: #f8f8f8;">: </span><span style="background-color: #f8f8f8; color: #aa22ff; font-weight: bold;">true</span><span style="background-color: #f8f8f8;">,
</span><span style="background-color: #f8f8f8; color: green; font-weight: bold;">"WarrantyYears"</span><span style="background-color: #f8f8f8;">: </span><span style="background-color: #f8f8f8; color: #666666;">4</span><span style="background-color: #f8f8f8;">,
</span><span style="background-color: #f8f8f8; color: green; font-weight: bold;">"seatCover"</span><span style="background-color: #f8f8f8;">: </span><span style="background-color: #f8f8f8; color: #bb4444;">"leather"</span><span style="background-color: #f8f8f8;">,
</span><span style="background-color: #f8f8f8; color: green; font-weight: bold;">"Driver Seat"</span><span style="background-color: #f8f8f8;">: </span><span style="background-color: #f8f8f8; color: #bb4444;">"6-Way Power"</span><span style="background-color: #f8f8f8;">
},
</span><span style="background-color: #f8f8f8; color: green; font-weight: bold;">"ResultPath"</span><span style="background-color: #f8f8f8;">: </span><span style="background-color: #f8f8f8; color: #bb4444;">"$.specs.extra"</span><span style="background-color: #f8f8f8;">
},
</span><span style="background-color: #f8f8f8; color: green; font-weight: bold;">"VXConfigurations"</span><span style="background-color: #f8f8f8;">: {
</span><span style="background-color: #f8f8f8; color: green; font-weight: bold;">"Type"</span><span style="background-color: #f8f8f8;">: </span><span style="background-color: #f8f8f8; color: #bb4444;">"Pass"</span><span style="background-color: #f8f8f8;">,
</span><span style="background-color: #f8f8f8; color: green; font-weight: bold;">"Next"</span><span style="background-color: #f8f8f8;">: </span><span style="background-color: #f8f8f8; color: #bb4444;">"WaitForPublish"</span><span style="background-color: #f8f8f8;">,
</span><span style="background-color: #f8f8f8; color: green; font-weight: bold;">"Result"</span><span style="background-color: #f8f8f8;">: {
</span><span style="background-color: #f8f8f8; color: green; font-weight: bold;">"Alloy Wheels"</span><span style="background-color: #f8f8f8;">: </span><span style="background-color: #f8f8f8; color: #aa22ff; font-weight: bold;">true</span><span style="background-color: #f8f8f8;">,
</span><span style="background-color: #f8f8f8; color: green; font-weight: bold;">"WarrantyYears"</span><span style="background-color: #f8f8f8;">: </span><span style="background-color: #f8f8f8; color: #666666;">6</span><span style="background-color: #f8f8f8;">,
</span><span style="background-color: #f8f8f8; color: green; font-weight: bold;">"seatCover"</span><span style="background-color: #f8f8f8;">: </span><span style="background-color: #f8f8f8; color: #bb4444;">"leather"</span><span style="background-color: #f8f8f8;">,
</span><span style="background-color: #f8f8f8; color: green; font-weight: bold;">"Driver Seat"</span><span style="background-color: #f8f8f8;">: </span><span style="background-color: #f8f8f8; color: #bb4444;">"8-Way Power"</span><span style="background-color: #f8f8f8;">
},
</span><span style="background-color: #f8f8f8; color: green; font-weight: bold;">"ResultPath"</span><span style="background-color: #f8f8f8;">: </span><span style="background-color: #f8f8f8; color: #bb4444;">"$.specs.extra"</span><span style="background-color: #f8f8f8;">
},
</span><span style="background-color: #f8f8f8; color: green; font-weight: bold;">"WaitForPublish"</span><span style="background-color: #f8f8f8;">: {
</span><span style="background-color: #f8f8f8; color: green; font-weight: bold;">"Type"</span><span style="background-color: #f8f8f8;">: </span><span style="background-color: #f8f8f8; color: #bb4444;">"Wait"</span><span style="background-color: #f8f8f8;">,
</span><span style="background-color: #f8f8f8; color: green; font-weight: bold;">"SecondsPath"</span><span style="background-color: #f8f8f8;">: </span><span style="background-color: #f8f8f8; color: #bb4444;">"$.pubDelay"</span><span style="background-color: #f8f8f8;">,
</span><span style="background-color: #f8f8f8; color: green; font-weight: bold;">"Next"</span><span style="background-color: #f8f8f8;">: </span><span style="background-color: #f8f8f8; color: #bb4444;">"FinishSpecs"</span><span style="background-color: #f8f8f8;">
},
</span><span style="background-color: #f8f8f8; color: green; font-weight: bold;">"SpecialConfigCheckErrorState"</span><span style="background-color: #f8f8f8;">: {
</span><span style="background-color: #f8f8f8; color: green; font-weight: bold;">"Type"</span><span style="background-color: #f8f8f8;">: </span><span style="background-color: #f8f8f8; color: #bb4444;">"Fail"</span><span style="background-color: #f8f8f8;">
},
</span><span style="background-color: #f8f8f8; color: green; font-weight: bold;">"FinishSpecs"</span><span style="background-color: #f8f8f8;">: {
</span><span style="background-color: #f8f8f8; color: green; font-weight: bold;">"Type"</span><span style="background-color: #f8f8f8;">: </span><span style="background-color: #f8f8f8; color: #bb4444;">"Pass"</span><span style="background-color: #f8f8f8;">,
</span><span style="background-color: #f8f8f8; color: green; font-weight: bold;">"OutputPath"</span><span style="background-color: #f8f8f8;">: </span><span style="background-color: #f8f8f8; color: #bb4444;">"$.specs"</span><span style="background-color: #f8f8f8;">,
</span><span style="background-color: #f8f8f8; color: green; font-weight: bold;">"End"</span><span style="background-color: #f8f8f8;">: </span><span style="background-color: #f8f8f8; color: #aa22ff; font-weight: bold;">true</span><span style="background-color: #f8f8f8;">
}
}
}
</span></pre>
</div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj-wkXEc2ZNdepdWrIwQwwGEAdPWALiV2PSzV6_Y2S7RKNRafTapfv3EW4oItUa9UnBY8t-HbPXjVxsI658Tj9WKaRgXkoN-I_w2jDV0pmYi7jsPq45WnUz6qr_h-FbL73qNlFaXj9cwSg/s1600/StateMcResult.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="743" data-original-width="1600" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj-wkXEc2ZNdepdWrIwQwwGEAdPWALiV2PSzV6_Y2S7RKNRafTapfv3EW4oItUa9UnBY8t-HbPXjVxsI658Tj9WKaRgXkoN-I_w2jDV0pmYi7jsPq45WnUz6qr_h-FbL73qNlFaXj9cwSg/s1600/StateMcResult.png" /></a></div>
<br /></div>
Robin Varghesehttp://www.blogger.com/profile/00841917517097216645noreply@blogger.com1tag:blogger.com,1999:blog-6262128735818493674.post-59309030882183433402020-05-09T05:16:00.000+05:302020-05-10T00:01:16.343+05:30Step Functions - AWS Service Integrations 2<div dir="ltr" style="text-align: left;" trbidi="on">
In the previous post we successfully executed dynamo db, sqs and sns interactions directly from step functions. I wanted to try some tweaks to my state machine.<br />
<a name='more'></a><br />
Consider the SQS call from state machine<br />
<div style="background: #f8f8f8; border-width: 0.1em 0.1em 0.1em 0.8em; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="line-height: 125%; margin: 0;"><span style="color: #bb4444;">"InformOtherQueue"</span><span style="border: 1px solid #FF0000;">:</span> {
<span style="color: green; font-weight: bold;">"Type"</span>: <span style="color: #bb4444;">"Task"</span>,
<span style="color: green; font-weight: bold;">"Resource"</span>: <span style="color: #bb4444;">"arn:aws:states:::sqs:sendMessage"</span>,
<span style="color: green; font-weight: bold;">"Parameters"</span>: {
<span style="color: green; font-weight: bold;">"QueueUrl"</span>: <span style="color: #bb4444;">"https://sqs.us-east-1.amazonaws.com/466170491455/TeamCSqs"</span>,
<span style="color: green; font-weight: bold;">"MessageBody"</span>: {
<span style="color: green; font-weight: bold;">"Input.$"</span>: <span style="color: #bb4444;">"$.messageId"</span>
}
},
<span style="color: green; font-weight: bold;">"ResultPath"</span>: <span style="color: #bb4444;">"$.SQS"</span>,
<span style="color: green; font-weight: bold;">"Next"</span>: <span style="color: #bb4444;">"WaitSometime"</span>,
<span style="color: green; font-weight: bold;">"TimeoutSeconds"</span>: <span style="color: #666666;">10</span>,
<span style="color: green; font-weight: bold;">"Catch"</span>: [
{
<span style="color: green; font-weight: bold;">"ErrorEquals"</span>: [
<span style="color: #bb4444;">"States.ALL"</span>
],
<span style="color: green; font-weight: bold;">"Next"</span>: <span style="color: #bb4444;">"GracefulFail"</span>
}
]
}
</pre>
</div>
I changed my resource to "arn:aws:states:::sqs:sendMessage.sync" and reran the state machine. I expected no difference but it failed:
<br />
<div style="background: #f8f8f8; border-width: 0.1em 0.1em 0.1em 0.8em; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="line-height: 125%; margin: 0;">{
<span style="color: green; font-weight: bold;">"error"</span>: <span style="color: #bb4444;">"States.Runtime"</span>,
<span style="color: green; font-weight: bold;">"cause"</span>: <span style="color: #bb4444;">"An error occurred while scheduling the state 'InformOtherQueue'.</span></pre>
<pre style="line-height: 125%; margin: 0;"><span style="color: #bb4444;"> The provided ARN 'arn:aws:states:::sqs:sendMessage.sync' is not recognized."</span>
}
</pre>
</div>
Long story short, "sync" attribute is only supported by specific resources - AWS Batch and ECS. Which makes sense if we read the explanation:
<!-- HTML generated using hilite.me --><br />
<div style="background: #f8f8f8; border-width: 0.1em 0.1em 0.1em 0.8em; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="line-height: 125%; margin: 0;">For integrated services such as AWS Batch and Amazon ECS, Step Functions can wait</pre>
<pre style="line-height: 125%; margin: 0;">for a request to complete before progressing to the next state. To have Step </pre>
<pre style="line-height: 125%; margin: 0;">Functions wait, specify the "Resource" field in your task state definition with </pre>
<pre style="line-height: 125%; margin: 0;">the .sync suffix appended after the resource URI.
</pre>
</div>
In case of SQS, the state machine gets a success response and then moves forward. There is no asynchronous behavior involved. But AWS batch jobs could be long running and waiting on their completion might not always be desired. So when we need the wait for compete behavior we can use the "sync" setting.
<br />
<br />
The other capability that AWS Integrations exposes is callbacks. Step Functions can pause workflow waiting for some information to be returned from a resource. For example in this case the state machine could wait after sending SQS message for a reply from dependent team.<br />
I changed my SQS step to be as below
<!-- HTML generated using hilite.me --><br />
<div style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="line-height: 125%; margin: 0px;"><span style="background-color: #f8f8f8; color: #bb4444;">"InformOtherQueue"</span><span style="background-color: #f8f8f8; border: 1px solid rgb(255, 0, 0);">:</span><span style="background-color: #f8f8f8;"> {
</span><span style="background-color: #f8f8f8; color: green; font-weight: bold;">"Type"</span><span style="background-color: #f8f8f8;">: </span><span style="background-color: #f8f8f8; color: #bb4444;">"Task"</span><span style="background-color: #f8f8f8;">,
</span><span style="background-color: #f8f8f8; color: green; font-weight: bold;">"Resource"</span><span style="background-color: #f8f8f8;">: </span><span style="color: #bb4444;"><span style="background-color: #f8f8f8;">"arn:aws:states:::sqs:sendMessage</span><span style="background-color: yellow;">.waitForTaskToken"</span></span><span style="background-color: #f8f8f8;">,
</span><span style="background-color: #f8f8f8; color: green; font-weight: bold;">"Parameters"</span><span style="background-color: #f8f8f8;">: {
</span><span style="background-color: #f8f8f8; color: green; font-weight: bold;">"QueueUrl"</span><span style="background-color: #f8f8f8;">: </span><span style="background-color: #f8f8f8; color: #bb4444;">"https://sqs.us-east-1.amazonaws.com/466170491455/TeamCSqs"</span><span style="background-color: #f8f8f8;">,
</span><span style="background-color: #f8f8f8; color: green; font-weight: bold;">"MessageBody"</span><span style="background-color: #f8f8f8;">: {
</span><span style="background-color: #f8f8f8; color: green; font-weight: bold;">"Input.$"</span><span style="background-color: #f8f8f8;">: </span><span style="background-color: #f8f8f8; color: #bb4444;">"$.messageId"</span><span style="background-color: #f8f8f8;">,
</span><span style="background-color: yellow;"><span style="color: green; font-weight: bold;">"TaskToken.$"</span>: <span style="color: #bb4444;">"$$.Task.Token"</span></span><span style="background-color: #f8f8f8;">
}
},
</span><span style="background-color: #f8f8f8; color: green; font-weight: bold;">"ResultPath"</span><span style="background-color: #f8f8f8;">: </span><span style="background-color: #f8f8f8; color: #bb4444;">"$.SQS"</span><span style="background-color: #f8f8f8;">,
</span><span style="background-color: #f8f8f8; color: green; font-weight: bold;">"Next"</span><span style="background-color: #f8f8f8;">: </span><span style="background-color: #f8f8f8; color: #bb4444;">"WaitSometime"</span><span style="background-color: #f8f8f8;">,
</span><span style="background-color: #f8f8f8; color: green; font-weight: bold;">"TimeoutSeconds"</span><span style="background-color: #f8f8f8;">: </span><span style="background-color: #f8f8f8; color: #666666;">10</span><span style="background-color: #f8f8f8;">,
</span><span style="background-color: #f8f8f8; color: green; font-weight: bold;">"Catch"</span><span style="background-color: #f8f8f8;">: [
{
</span><span style="background-color: #f8f8f8; color: green; font-weight: bold;">"ErrorEquals"</span><span style="background-color: #f8f8f8;">: [
</span><span style="background-color: #f8f8f8; color: #bb4444;">"States.ALL"</span><span style="background-color: #f8f8f8;">
],
</span><span style="background-color: #f8f8f8; color: green; font-weight: bold;">"Next"</span><span style="background-color: #f8f8f8;">: </span><span style="background-color: #f8f8f8; color: #bb4444;">"GracefulFail"</span><span style="background-color: #f8f8f8;">
}
]
}
</span></pre>
</div>
In this case the input generated in the SQS State is as below:
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgJr43jR9U4XKqA0Ll4ATE1n8IxPhyphenhyphenMroM9WAw2vfjT_LHGdIYR5SgBjivJ9K3UtKXQY5sXamHOIf2ZTJ42b9zMGEneJpNL3NJTaus68NtJWxi1I3GmXNJWPppPr8oG5yVmX4uNZt3rVVk/s1600/SQSMessageSent.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="370" data-original-width="849" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgJr43jR9U4XKqA0Ll4ATE1n8IxPhyphenhyphenMroM9WAw2vfjT_LHGdIYR5SgBjivJ9K3UtKXQY5sXamHOIf2ZTJ42b9zMGEneJpNL3NJTaus68NtJWxi1I3GmXNJWPppPr8oG5yVmX4uNZt3rVVk/s1600/SQSMessageSent.png" /></a></div>
<br />
The task token is now a part of the SQS Message. The state machine will wait for 10 seconds and then timeout if it does not receive a callback from the Queue consumer
<!-- HTML generated using hilite.me --><br />
<div style="background: #f8f8f8; border-width: 0.1em 0.1em 0.1em 0.8em; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="line-height: 125%; margin: 0;">When it's complete, the external service calls SendTaskSuccess or SendTaskFailure</pre>
<pre style="line-height: 125%; margin: 0;">with the taskToken included. Only then does the workflow continue to the next state.</pre>
</div>
I will setup a Lambda here that can consume from the SQS and return the callback information needed here.
<!-- HTML generated using hilite.me --><br />
<div style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="line-height: 125%; margin: 0px;"><span style="background-color: #f8f8f8; color: #aa22ff; font-weight: bold;">public</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #aa22ff; font-weight: bold;">class</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: blue;">NotificationHandler</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #aa22ff; font-weight: bold;">implements</span><span style="background-color: #f8f8f8;"> RequestHandler</span><span style="background-color: #f8f8f8; color: #666666;"><</span><span style="background-color: #f8f8f8;">SQSEvent</span><span style="background-color: #f8f8f8; color: #666666;">,</span><span style="background-color: #f8f8f8;"> Void</span><span style="background-color: #f8f8f8; color: #666666;">></span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #666666;">{</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #aa22ff; font-weight: bold;">private</span><span style="background-color: #f8f8f8;"> ObjectMapper mapper </span><span style="background-color: #f8f8f8; color: #666666;">=</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #aa22ff; font-weight: bold;">new</span><span style="background-color: #f8f8f8;"> ObjectMapper</span><span style="background-color: #f8f8f8; color: #666666;">();</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #aa22ff; font-weight: bold;">private</span><span style="background-color: #f8f8f8;"> AWSStepFunctions awsStepFunctionsClient</span><span style="background-color: #f8f8f8; color: #666666;">;</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #aa22ff; font-weight: bold;">public</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #00a000;">NotificationHandler</span><span style="background-color: #f8f8f8; color: #666666;">()</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #666666;">{</span><span style="background-color: #f8f8f8;">
AWSCredentials awsCredentials </span><span style="background-color: #f8f8f8; color: #666666;">=</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #aa22ff; font-weight: bold;">new</span><span style="background-color: #f8f8f8;"> BasicAWSCredentials</span><span style="background-color: #f8f8f8; color: #666666;">(</span><span style="background-color: #f8f8f8; color: #bb4444;">"AccessKey"</span><span style="background-color: #f8f8f8; color: #666666;">,</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #bb4444;">"SecretKey"</span><span style="background-color: #f8f8f8; color: #666666;">);</span><span style="background-color: #f8f8f8;">
AWSCredentialsProvider awsCredentialsProvider </span><span style="background-color: #f8f8f8; color: #666666;">=</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #aa22ff; font-weight: bold;">new</span><span style="background-color: #f8f8f8;"> AWSStaticCredentialsProvider</span><span style="background-color: #f8f8f8; color: #666666;">(</span><span style="background-color: #f8f8f8;">awsCredentials</span><span style="background-color: #f8f8f8; color: #666666;">);</span><span style="background-color: #f8f8f8;">
ClientConfiguration clientConfiguration </span><span style="background-color: #f8f8f8; color: #666666;">=</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #aa22ff; font-weight: bold;">new</span><span style="background-color: #f8f8f8;"> ClientConfiguration</span><span style="background-color: #f8f8f8; color: #666666;">();</span><span style="background-color: #f8f8f8;">
clientConfiguration</span><span style="background-color: #f8f8f8; color: #666666;">.</span><span style="background-color: #f8f8f8; color: #bb4444;">setClientExecutionTimeout</span><span style="background-color: #f8f8f8; color: #666666;">(2</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #666666;">*</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #666666;">60</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #666666;">*</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #666666;">1000);</span><span style="background-color: #f8f8f8; color: #008800; font-style: italic;">//2 mins</span><span style="background-color: #f8f8f8;">
awsStepFunctionsClient </span><span style="background-color: #f8f8f8; color: #666666;">=</span><span style="background-color: #f8f8f8;"> AWSStepFunctionsClient</span><span style="background-color: #f8f8f8; color: #666666;">.</span><span style="background-color: #f8f8f8; color: #bb4444;">builder</span><span style="background-color: #f8f8f8; color: #666666;">()</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #666666;">.</span><span style="background-color: #f8f8f8; color: #bb4444;">withCredentials</span><span style="background-color: #f8f8f8; color: #666666;">(</span><span style="background-color: #f8f8f8;">awsCredentialsProvider</span><span style="background-color: #f8f8f8; color: #666666;">)</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #666666;">.</span><span style="background-color: #f8f8f8; color: #bb4444;">withRegion</span><span style="background-color: #f8f8f8; color: #666666;">(</span><span style="background-color: #f8f8f8;">Regions</span><span style="background-color: #f8f8f8; color: #666666;">.</span><span style="background-color: #f8f8f8; color: #bb4444;">US_EAST_1</span><span style="background-color: #f8f8f8; color: #666666;">)</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #666666;">.</span><span style="background-color: #f8f8f8; color: #bb4444;">withClientConfiguration</span><span style="background-color: #f8f8f8; color: #666666;">(</span><span style="background-color: #f8f8f8;">clientConfiguration</span><span style="background-color: #f8f8f8; color: #666666;">)</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #666666;">.</span><span style="background-color: #f8f8f8; color: #bb4444;">build</span><span style="background-color: #f8f8f8; color: #666666;">();</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #666666;">}</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #aa22ff; font-weight: bold;">public</span><span style="background-color: #f8f8f8;"> Void </span><span style="background-color: #f8f8f8; color: #00a000;">handleRequest</span><span style="background-color: #f8f8f8; color: #666666;">(</span><span style="background-color: #f8f8f8;">SQSEvent request</span><span style="background-color: #f8f8f8; color: #666666;">,</span><span style="background-color: #f8f8f8;"> Context context</span><span style="background-color: #f8f8f8; color: #666666;">)</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #666666;">{</span><span style="background-color: #f8f8f8;">
LambdaLogger logger </span><span style="background-color: #f8f8f8; color: #666666;">=</span><span style="background-color: #f8f8f8;"> context</span><span style="background-color: #f8f8f8; color: #666666;">.</span><span style="background-color: #f8f8f8; color: #bb4444;">getLogger</span><span style="background-color: #f8f8f8; color: #666666;">();</span><span style="background-color: #f8f8f8;">
logger</span><span style="background-color: #f8f8f8; color: #666666;">.</span><span style="background-color: #f8f8f8; color: #bb4444;">log</span><span style="background-color: #f8f8f8; color: #666666;">(</span><span style="background-color: #f8f8f8; color: #bb4444;">"In Handler: Executing "</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #666666;">+</span><span style="background-color: #f8f8f8;"> context</span><span style="background-color: #f8f8f8; color: #666666;">.</span><span style="background-color: #f8f8f8; color: #bb4444;">getFunctionName</span><span style="background-color: #f8f8f8; color: #666666;">()</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #666666;">+</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #bb4444;">", "</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #666666;">+</span><span style="background-color: #f8f8f8;"> context</span><span style="background-color: #f8f8f8; color: #666666;">.</span><span style="background-color: #f8f8f8; color: #bb4444;">getFunctionVersion</span><span style="background-color: #f8f8f8; color: #666666;">());</span><span style="background-color: #f8f8f8;">
logger</span><span style="background-color: #f8f8f8; color: #666666;">.</span><span style="background-color: #f8f8f8; color: #bb4444;">log</span><span style="background-color: #f8f8f8; color: #666666;">(</span><span style="background-color: #f8f8f8;">request</span><span style="background-color: #f8f8f8; color: #666666;">.</span><span style="background-color: #f8f8f8; color: #bb4444;">toString</span><span style="background-color: #f8f8f8; color: #666666;">());</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #aa22ff; font-weight: bold;">for</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #666666;">(</span><span style="background-color: #f8f8f8;">SQSEvent</span><span style="background-color: #f8f8f8; color: #666666;">.</span><span style="background-color: #f8f8f8; color: #bb4444;">SQSMessage</span><span style="background-color: #f8f8f8;"> snsRecord </span><span style="background-color: #f8f8f8; color: #666666;">:</span><span style="background-color: #f8f8f8;"> request</span><span style="background-color: #f8f8f8; color: #666666;">.</span><span style="background-color: #f8f8f8; color: #bb4444;">getRecords</span><span style="background-color: #f8f8f8; color: #666666;">())</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #666666;">{</span><span style="background-color: #f8f8f8;">
String message </span><span style="background-color: #f8f8f8; color: #666666;">=</span><span style="background-color: #f8f8f8;"> snsRecord</span><span style="background-color: #f8f8f8; color: #666666;">.</span><span style="background-color: #f8f8f8; color: #bb4444;">getBody</span><span style="background-color: #f8f8f8; color: #666666;">();</span><span style="background-color: #f8f8f8;">
Map</span><span style="background-color: #f8f8f8; color: #666666;"><</span><span style="background-color: #f8f8f8;">String</span><span style="background-color: #f8f8f8; color: #666666;">,</span><span style="background-color: #f8f8f8;"> String</span><span style="background-color: #f8f8f8; color: #666666;">></span><span style="background-color: #f8f8f8;"> map </span><span style="background-color: #f8f8f8; color: #666666;">=</span><span style="background-color: #f8f8f8;"> getMap</span><span style="background-color: #f8f8f8; color: #666666;">(</span><span style="background-color: #f8f8f8;">message</span><span style="background-color: #f8f8f8; color: #666666;">,</span><span style="background-color: #f8f8f8;"> logger</span><span style="background-color: #f8f8f8; color: #666666;">);</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #aa22ff; font-weight: bold;">if</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #666666;">(</span><span style="background-color: #f8f8f8;">map</span><span style="background-color: #f8f8f8; color: #666666;">.</span><span style="background-color: #f8f8f8; color: #bb4444;">containsKey</span><span style="background-color: #f8f8f8; color: #666666;">(</span><span style="background-color: #f8f8f8; color: #bb4444;">"TaskToken"</span><span style="background-color: #f8f8f8; color: #666666;">))</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #666666;">{</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #008800; font-style: italic;">//Respond back to Step Fn that work is completed</span><span style="background-color: #f8f8f8;">
SendTaskSuccessResult successResult </span><span style="background-color: #f8f8f8; color: #666666;">=</span><span style="background-color: #f8f8f8;"> callbackStepFunction</span><span style="background-color: #f8f8f8; color: #666666;">(</span><span style="background-color: #f8f8f8;">map</span><span style="background-color: #f8f8f8; color: #666666;">.</span><span style="background-color: #f8f8f8; color: #bb4444;">get</span><span style="background-color: #f8f8f8; color: #666666;">(</span><span style="background-color: #f8f8f8; color: #bb4444;">"TaskToken"</span><span style="background-color: #f8f8f8; color: #666666;">),</span><span style="background-color: #f8f8f8;"> map</span><span style="background-color: #f8f8f8; color: #666666;">.</span><span style="background-color: #f8f8f8; color: #bb4444;">get</span><span style="background-color: #f8f8f8; color: #666666;">(</span><span style="background-color: #f8f8f8; color: #bb4444;">"Input"</span><span style="background-color: #f8f8f8; color: #666666;">));</span><span style="background-color: #f8f8f8;">
logger</span><span style="background-color: #f8f8f8; color: #666666;">.</span><span style="background-color: #f8f8f8; color: #bb4444;">log</span><span style="background-color: #f8f8f8; color: #666666;">(</span><span style="background-color: #f8f8f8;">successResult</span><span style="background-color: #f8f8f8; color: #666666;">.</span><span style="background-color: #f8f8f8; color: #bb4444;">toString</span><span style="background-color: #f8f8f8; color: #666666;">());</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #666666;">}</span><span style="background-color: #f8f8f8;">
System</span><span style="background-color: #f8f8f8; color: #666666;">.</span><span style="background-color: #f8f8f8; color: #bb4444;">out</span><span style="background-color: #f8f8f8; color: #666666;">.</span><span style="background-color: #f8f8f8; color: #bb4444;">println</span><span style="background-color: #f8f8f8; color: #666666;">(</span><span style="background-color: #f8f8f8;">message</span><span style="background-color: #f8f8f8; color: #666666;">);</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #666666;">}</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #aa22ff; font-weight: bold;">return</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #aa22ff; font-weight: bold;">null</span><span style="background-color: #f8f8f8; color: #666666;">;</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #666666;">}</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #aa22ff; font-weight: bold;">private</span><span style="background-color: #f8f8f8;"> Map </span><span style="background-color: #f8f8f8; color: #00a000;">getMap</span><span style="background-color: #f8f8f8; color: #666666;">(</span><span style="background-color: #f8f8f8;">String message</span><span style="background-color: #f8f8f8; color: #666666;">,</span><span style="background-color: #f8f8f8;"> LambdaLogger logger</span><span style="background-color: #f8f8f8; color: #666666;">)</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #666666;">{</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #aa22ff; font-weight: bold;">try</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #666666;">{</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #aa22ff; font-weight: bold;">return</span><span style="background-color: #f8f8f8;"> mapper</span><span style="background-color: #f8f8f8; color: #666666;">.</span><span style="background-color: #f8f8f8; color: #bb4444;">readValue</span><span style="background-color: #f8f8f8; color: #666666;">(</span><span style="background-color: #f8f8f8;">message</span><span style="background-color: #f8f8f8; color: #666666;">,</span><span style="background-color: #f8f8f8;"> Map</span><span style="background-color: #f8f8f8; color: #666666;">.</span><span style="background-color: #f8f8f8; color: #bb4444;">class</span><span style="background-color: #f8f8f8; color: #666666;">);</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #666666;">}</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #aa22ff; font-weight: bold;">catch</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #666666;">(</span><span style="background-color: #f8f8f8;">JsonProcessingException e</span><span style="background-color: #f8f8f8; color: #666666;">)</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #666666;">{</span><span style="background-color: #f8f8f8;">
logger</span><span style="background-color: #f8f8f8; color: #666666;">.</span><span style="background-color: #f8f8f8; color: #bb4444;">log</span><span style="background-color: #f8f8f8; color: #666666;">(</span><span style="background-color: #f8f8f8;">e</span><span style="background-color: #f8f8f8; color: #666666;">.</span><span style="background-color: #f8f8f8; color: #bb4444;">getMessage</span><span style="background-color: #f8f8f8; color: #666666;">()</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #666666;">+</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #bb4444;">" while processing "</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #666666;">+</span><span style="background-color: #f8f8f8;"> message</span><span style="background-color: #f8f8f8; color: #666666;">);</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #666666;">}</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #aa22ff; font-weight: bold;">return</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #aa22ff; font-weight: bold;">new</span><span style="background-color: #f8f8f8;"> HashMap</span><span style="background-color: #f8f8f8; color: #666666;"><>();</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #666666;">}</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #aa22ff; font-weight: bold;">private</span><span style="background-color: #f8f8f8;"> SendTaskSuccessResult </span><span style="background-color: #f8f8f8; color: #00a000;">callbackStepFunction</span><span style="background-color: #f8f8f8; color: #666666;">(</span><span style="background-color: #f8f8f8;">String taskToken</span><span style="background-color: #f8f8f8; color: #666666;">,</span><span style="background-color: #f8f8f8;"> String message</span><span style="background-color: #f8f8f8; color: #666666;">)</span><span style="background-color: #f8f8f8;"> </span><span style="background-color: #f8f8f8; color: #666666;">{</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: yellow;"> SendTaskSuccessRequest sendTaskSuccessRequest <span style="color: #666666;">=</span> <span style="color: #aa22ff; font-weight: bold;">new</span> SendTaskSuccessRequest<span style="color: #666666;">();</span>
sendTaskSuccessRequest<span style="color: #666666;">.</span><span style="color: #bb4444;">setTaskToken</span><span style="color: #666666;">(</span>taskToken<span style="color: #666666;">);</span>
sendTaskSuccessRequest<span style="color: #666666;">.</span><span style="color: #bb4444;">setOutput</span><span style="color: #666666;">(</span><span style="color: #bb4444;">"{\"Output\": \"message\",\"ack\":\"success\"}"</span><span style="color: #666666;">);</span>
<span style="color: #aa22ff; font-weight: bold;">return</span> awsStepFunctionsClient<span style="color: #666666;">.</span><span style="color: #bb4444;">sendTaskSuccess</span><span style="color: #666666;">(</span>sendTaskSuccessRequest<span style="color: #666666;">);</span>
</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #666666;">}</span><span style="background-color: #f8f8f8;">
</span><span style="background-color: #f8f8f8; color: #666666;">}</span><span style="background-color: #f8f8f8;">
</span></pre>
</div>
The Lambda reads the message, if it detects a task token it responds back with TaskSuccess Message.
The Step Function executed successfully. The states results can be seen here.
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjL5mt36hNJxxWczD079N8CTqoi8eo7yAArTeFosaeun_J_nns2T6m_N-QwsfEwxjSmif9x2bMy0AK973XpbMMZhh4ku0qpVtZHGEdTqD8u_sVpcZs1WUASbvzdxLNzyjKLK6TXvmUf3Ug/s1600/WaitFortaskToken.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="941" data-original-width="1199" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjL5mt36hNJxxWczD079N8CTqoi8eo7yAArTeFosaeun_J_nns2T6m_N-QwsfEwxjSmif9x2bMy0AK973XpbMMZhh4ku0qpVtZHGEdTqD8u_sVpcZs1WUASbvzdxLNzyjKLK6TXvmUf3Ug/s1600/WaitFortaskToken.png" /></a></div>
<br /></div>
Robin Varghesehttp://www.blogger.com/profile/00841917517097216645noreply@blogger.com1