Playing with ElastiCache

I have never used Elastic Cache yet. My caching use cases were solved either via an in memory cache or a hosted cache. I finally got the chance to venture beyond - to cloud caches.

AWS offers two cache solutions (or cache engines) - Redis and Memcached. I have used Memcached before and it has performed well for my use cases.

memcached is a high-performance, distributed memory object caching system, 

generic in nature, but originally intended for use in speeding up dynamic web 

applications by alleviating database load.memcached allows you to take memory 

from parts of your system where you have more than you need and make it 

accessible to areas where you have less than you need

Redis is far more powerful and versatile. From Redis website

Redis is an open source (BSD licensed), in-memory data structure store, used as 
a database, cache and message broker. It supports data structures such as 

strings, hashes, lists, sets, sorted sets with range queries, bitmaps, 

hyperloglogs, geospatial indexes with radius queries and streams. Redis has 

built-in replication, Lua scripting, LRU eviction, transactions and different 

levels of on-disk persistence, and provides high availability via Redis Sentinel 

and automatic partitioning with Redis Cluster

Wanting to play with the new shiny toy, I decided to experiment with Redis first. I setup a simple cluster that has 1 replica nodes.

As seen my cache is up and running with 1 primary node and 1 replica node. Similar to Kinesis, the data key range can be distributed between shards, In this case I have chosen to setup a single shard for my elasticache (i.e. cluster mode disabled). I have also added a single replica node. So my Cluster is composed of 2 nodes - a primary and a replica or backup node

If the cluster with replicas has Multi-AZ with Automatic Failover enabled and 

the primary node fails, the primary fails over to a read replica. Because the 

data is updated on the replica nodes asynchronously, there may be some data loss 

due to latency in updating the replica nodes

Our cache is using 'Multi-AZ with Automatic Failover enabled' (this is default option). AW will also take daily backups of my cluster during a scheduled maintenance window. It creates a single .rdb file per shard (One in this case). This AWS Link provides good guidelines on when to use which option.
The backup node can also serve read requests ( providing a read scaling behavior).
All the ElastiCache instances are launched within an Amazon VPC. This means you need to provide a subnet group ''

A subnet group is a collection of subnets (typically private) that you can 

designate for your clusters running in an Amazon Virtual Private Cloud (VPC) 

environment. When you create a cluster in an Amazon VPC, you must specify a 

subnet group. ElastiCache uses that subnet group to choose a subnet and IP 

addresses within that subnet to associate with your nodes.
(A subnet is a range of IP addresses in your VPC.)

The endpoints are for connecting to our ElastiCache

Redis (cluster mode disabled) clusters, use the Primary Endpoint for all write 

operations. Use the Reader Endpoint to evenly split incoming connections to the 

endpoint between all read replicas. Use the individual Node Endpoints for read 

operations

Now I want to start writing to the cache. Being a serverless fan, I wanted to try out the integration with a Lambda function.
My function is pretty straight forward. It runs every 1 second, adding resetting the tokens in a bucket.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26

public class TokenBucketRefresherLambdaFn implements RequestHandler<String, Void> { private final Jedis jedis; public TokenBucketRefresherLambdaFn() { jedis = new Jedis(new HostAndPort( "redis-test-cache.ql6jf2.ng.0001.use1.cache.amazonaws.com", 6379)); } @Override public Void handleRequest(String input, Context context) { LambdaLogger logger = context.getLogger(); logger.log("In Handler: Executing " + context.getFunctionName() + ", " + context.getFunctionVersion()); logger.log(input); //This input is not really used now String token = "TokenCount"; int capacity = 100; String value = jedis.get(token); if (value == null) { logger.log("No value detected for Key, adding"); } else { logger.log("Value detected for " + token + " is " + capacity); } jedis.set(token, String.valueOf(capacity)); return null; } }

Amazon does not provide any AWS library to connect with Redis. I used the Jedis Redis Client instead.

      <dependency>
          <groupId>redis.clients</groupId>
          <artifactId>jedis</artifactId>
          <version>3.2.0</version>
          <type>jar</type>
          <scope>compile</scope>
      </dependency>

The code is pretty straight forward. It adds tokens to the Bucket.  However to run this code, we need the Lambda to be VPC aware. Since my Elasti Cache is setup in the VPC.

I tested this code through the Lambda console. The Cloud Watch logs were as below:
Run 1:

In Handler: Executing TokenBucketRefresherLambda, $LATEST
EXAMPLE
No value detected for Key, adding
END RequestId: 00d372e1-acfa-4d5d-9099-7131e24e7533
REPORT RequestId: 00d372e1-acfa-4d5d-9099-7131e24e7533 Duration: 116.45 ms Billed Duration: 200 ms Memory Size: 512 MB Max Memory Used: 90 MB Init Duration: 371.09 ms 

Run 2:

START RequestId: 0fa896fd-0931-4390-b4db-63eaf442bb4c Version: $LATEST
In Handler: Executing TokenBucketRefresherLambda, $LATEST
EXAMPLE
Value detected for TokenCount is 100
END RequestId: 0fa896fd-0931-4390-b4db-63eaf442bb4c
REPORT RequestId: 0fa896fd-0931-4390-b4db-63eaf442bb4c Duration: 3.86 ms Billed Duration: 100 ms Memory Size: 512 MB Max Memory Used: 91 MB

Now for Token Bucket, I would schedule this Lambda to run every second refershing the tokens or providing a capacity of 100 Tokens/second