If you’ve read the blog posts on CloudJourney.io before, you’ve likely read the term “Continuous Verification”. If you haven’t that’s okay too. There’s an amazing artiIf you’ve read the blog posts on CloudJourney.io before, you’ve likely read the term “Continuous Verification”. If you haven’t that’s okay too. There’s an amazing article from Dan Illson and Bill Shetti on The New Stack explaining in detail what Continuous Verification is. To make sure we’re all on the same page, though, I’ll quickly go over it as well. As a definition, Continuous Verification is “A process of querying external system(s) and using information from the response to make decision(s) to improve the development and deployment process.” The definition comes down to making sure that DevOps teams put as many checks as possible into their CI/CD pipelines. Adding checks into a pipeline means there are fewer manual tasks and that means you have access to more data tot smooth out and improve your development and deployment process. The CloudJourney.io team built the ACME Fitness Shop to showcase the power of continuous verification in a containerized world. There are deployments for Kubernetes, Docker, and AWS Fargate. In this blog series, we’ll look at how the premise of Continuous Verification works in a serverless world, and how we built the components that make up the ACME Serverless Fitness Shop.

What is the ACME Serverless Fitness Shop

First off, I should probably introduce the ACME Serverless Fitness Shop. The shop combines the concepts of Serverless and Fitness, which are two of my favorite things, because combining two amazing things can only lead to more amazing outcomes. The shop contains seven different domains, which all contain one or more serverless functions:

  • Shipment: A shipping service, because what is a shop without a way to ship your purchases? 🚚
  • Payment: A payment service, because nothing in life is really free… 💰
  • Order: An order service, because what is a shop without actual orders to be shipped? 📦
  • Cart: A cart service, because what is a shop without a cart to put stuff in? 🛒
  • Catalog: A catalog service, because what is a shop without a catalog to show off our awesome red pants? 📖
  • User: A user service, because what is a shop without users to buy our awesome red pants? 👨‍💻
  • Point-of-Sales: A point-of-sales app to sell our products in brick-and-mortar stores! 🛍️

Some of these services are event-driven, while others have an HTTP API. The services that have APIs are built using the same API specifications as their containerized counterparts. We wanted to make sure that the serverless version of the ACME Fitness Shop is compatible with the frontend of the original ACME Fitness shop.

Deciding on Data stores

With Functions-as-a-Service you cannot maintain state inside of the function. Once the function is done processing, it’ll shut down and you would lose state. Most functions, though, will need to keep their information somewhere. Once you decide to go down the route of serverless for everything, there are quite a few options you have when it comes to storing data.

  • AWS DynamoDB if you want to go for a NoSQL database with single-digit millisecond latency at any scale
  • Amazon Aurora Serverless if you want a MySQL compatible relational database
  • Amazon RDS Proxy if you want to use AWS Lambda with traditional RDS relational databases

For the ACME Serverless Fitnesss Shop, most of the queries that are executed are “gets” and “puts", and we always know the type of data a function needs to access and which keys are associated with the data. The functions that exist right now, don’t need joins or schemas to preserve referential integrity. AWS strongly believes in using purpose built databases, so with that in mind you can pick a database which provides that functionality and in this case that would be DynamoDB. While the ACME Serverless Fitness Shop doesn’t need single-digit millisecond latency, DynamoDB is a completely managed service. That gives DynamoDB users the benefit of not having to worry about upgrade windows, patching, or any other operations tasks.

Deciding on Application integration

Serverless also means that your apps are event-driven so that you don’t have to worry about keeping servers running. That means the next step is to decide which service to use to handle the events. There are a few options here too:

  • Amazon SNS for publish/subscribe style messaging
  • Amazon SQS which is a managed queueing service
  • Amazon EventBridge which is a serverless event bus

In a queueing service, like SQS, receivers of the messages will have to poll to receive the messages and a single message can only be received by a single receiver. In publish-subscribe services, like SNS, messages are sent to all subscribers. The benefit of publish-subscribe services is that they “push” messages to the receivers, meaning that it’s usually a little faster. The biggest different, though, comes in the type of use cases these systems are a fit for. Queueing services are a great fit to decouple apps and allow them to communicate asynchronously. Publish-subscribe services are awesome when you need to let multiple different systems work on a single message.The ACME Serverless Fitness Shop has functions that take care of different messages and the asynchronous nature of the functions mean that SQS is a great fit.

Deciding on Compute

The last step is to decide where the apps will run. Within the AWS ecosystem you have, again, a few options to choose from:

  • AWS Lambda which lets you run code without provisioning servers and is almost synonymous with serverless
  • Lambda@Edge which allows you to run Lambda functions at edge locations
  • AWS Fargate which allows you to run containers in a serverless fashion

AWS Fargate is great and during re:Invent 2019 AWS introduced the ability to use Fargate to run Kubernetes pods too. This would be the easiest method to get the ACME Fitness Shop into the cloud, but even when our containers aren’t handling traffic they’ll incur a cost. As there already is a Fargate and Kubernetes version, and because the ideas is to pay as little as possible when functions aren’t running, we’ll go with AWS Lambda and the Go 1.x runtime.

From Microservices to Serverless

Moving from traditional microservices to anything event-driven requires you to refactor and rearchitect your code. Refactoring in itself is all about updating the code without changing the functional behavior of the service. To give some insights into how we changed the services, I’ll break down how we changed the Payment service to be serverless. In this case, it’ll be quite a massive change though as the service changes from an HTTP-based microservice to an SQS-based Lambda function. There are two major requirements for this change, though:

  • The service must still be able to validate the credit card payment and respond with the status of the validation (no change in functionality)
  • The input and output must not introduce or remove any fields that would alter the behavior of the service (no change to inputs or outputs)

Creating events

Event-driven architectures require events. Events should describe what has happened to the system. In the case of the payment service, there are two events. One that starts the service and one that is the response. The event that starts the payment service is sent by the order service when an order needs to be paid. So let’s call that event “PaymentRequested”. The event sent by the payment service is called “CreditCardValidated” because that is exactly what just happened, the credit card was validated. With event-driven architectures, it does become a lot harder to keep track of all the events that flow in the system. To make it a little easier to keep track, a good practice is to add some metadata to events. That makes the PaymentRequested event like the one below:

{
    "metadata": {
        "domain": "Order", // Domain represents the the event came from like Payment or Order
        "source": "CLI", // Source represents the function the event came from
        "type": "PaymentRequested", // Type respresents the type of event this is
        "status": "success" // Status represents the current status of the event
    },
    "data": {
        "orderID": "12345",
        "card": {
            "Type": "Visa",
            "Number": "4222222222222",
            "ExpiryYear": 2022,
            "ExpiryMonth": 12,
            "CVV": "123"
        },
        "total": "123"
    }
}

Similarly, for the CreditCardValidated event, it will look like:

{
    "metadata": {
        "domain": "Payment",
        "source": "CLI",
        "type": "CreditCardValidated",
        "status": "success"
    },
    "data": {
        "success": "true",
        "status": 200,
        "message": "transaction successful",
        "amount": 123,
        "transactionID": "3f846704-af12-4ea9-a98c-8d7b37e10b54"
    }
}

Functional behavior

From a behavior point of view, the Payment service will do three things:

  • Receive a message from Amazon SQS
  • Validate the credit card
  • Send the validation result to Amazon SQS

Translating that into Go, you end up with the below code. For clarity, I’ve removed the parts that send tracing data to Sentry.

package main

// removed imports for clarity

// handler handles the SQS events and returns an error if anything goes wrong.
// The resulting event, if no error is thrown, is sent to an SQS queue.
func handler(request events.SQSEvent) error {
	// Unmarshal the PaymentRequested event to a struct
	req, err := payment.UnmarshalPaymentRequested([]byte(request.Records[0].Body))
	if err != nil {
		return handleError("unmarshaling payment", err)
	}

	// Generate the event to emit
	evt := payment.CreditCardValidated{
		Metadata: payment.Metadata{
			Domain: payment.Domain,
			Source: "ValidateCreditCard",
			Type:   payment.CreditCardValidatedEvent,
			Status: "success",
		},
		Data: payment.PaymentData{
			Success:       true,
			Status:        http.StatusOK,
			Message:       payment.DefaultSuccessMessage,
			Amount:        req.Data.Total,
			OrderID:       req.Data.OrderID,
			TransactionID: uuid.Must(uuid.NewV4()).String(),
		},
	}

	// Check the creditcard is valid.
	// If the creditcard is not valid, update the event to emit
	// with new information
	check := validator.New()
	err = check.Creditcard(req.Data.Card)
	if err != nil {
		evt.Metadata.Status = "error"
		evt.Data.Success = false
		evt.Data.Status = http.StatusBadRequest
		evt.Data.Message = payment.DefaultErrorMessage
		evt.Data.TransactionID = "-1"
		handleError("validating creditcard", err)
	}

	// Create a new SQS EventEmitter and send the event
	em := sqs.New()
	err = em.Send(evt)
	if err != nil {
		return handleError("sending event", err)
	}

	return nil
}

// handleError takes the activity where the error occured and the error object and sends a message to sentry.
// The original error is returned so it can be thrown.
func handleError(activity string, err error) error {
	log.Printf("error %s: %s", activity, err.Error())
	return err
}

// The main method is executed by AWS Lambda and points to the handler
func main() {
	lambda.Start(handler)
}

Infrastructure as Code

Continuous Integration, Continuous Delivery, and Continuous Verification all rely on doing as much as possible in automated pipelines so developers and system engineers can focus on building business value. Moving as much as possible into a pipeline also means that the infrastructure should be created within the pipeline too and that means you want Infrastructure as Code. There are a ton of options when it comes to Infrastructure as Code:

  • Terraform, which lets you write HCL to create Infrastructure as Code
  • Serverless Framework, which was one of the first companies that made building and deploying functions easier
  • AWS CloudFormation (and the Serverless Application Model), which is the AWS specific configuration language
  • Pulumi, which is an open source Infrastructure as Code tool that works across clouds and allows you to create all sorts of resources

Ideally, I want to use a tool that that doesn’t have a custom domain-specific-language. As a developer, I’m most definitely not a YAML expert and I enjoy writing Go. If I could pick a tool for my Infrastructure as Code, I’d rather have my entire toolset be Go based because that is what I’m familiar with. This is where Pulumi shines. It allows me to use the Go toolchain, while still deploying to Amazon Web Services and leveraging all the constructs that exist in the AWS ecosystem. All the services, the DynamoDB table, and the SQS queues are deployed using Pulumi. The code below walks through how you create a DynamoDB table using the Pulumi Go SDK. For clarity, I’ve removed the “tags” you can add to AWS resources but you can find the complete code on GitHub.

package main

import (
	"fmt"

	"github.com/pulumi/pulumi-aws/sdk/go/aws/dynamodb"
	"github.com/pulumi/pulumi/sdk/go/pulumi"
	"github.com/pulumi/pulumi/sdk/go/pulumi/config"
)

// DynamoConfig contains the key-value pairs for the configuration of Amazon DynamoDB in this stack
type DynamoConfig struct {
	// Controls how you are charged for read and write throughput and how you manage capacity
	BillingMode pulumi.String `json:"billingmode"`

	// The number of write units for this table
	WriteCapacity pulumi.Int `json:"writecapacity"`

	// The number of read units for this table
	ReadCapacity pulumi.Int `json:"readcapacity"`
}

func main() {
	pulumi.Run(func(ctx *pulumi.Context) error {
		// Read the configuration data from Pulumi.<stack>.yaml
		conf := config.New(ctx, "awsconfig")

		// Create a new DynamoConfig object with the data from the configuration
		var dynamoConfig DynamoConfig
		conf.RequireObject("dynamodb", &dynamoConfig)

		// The table attributes represent a list of attributes that describe the key schema for the table and indexes
		tableAttributeInput := []dynamodb.TableAttributeInput{
			dynamodb.TableAttributeArgs{
				Name: pulumi.String("PK"),
				Type: pulumi.String("S"),
			}, dynamodb.TableAttributeArgs{
				Name: pulumi.String("SK"),
				Type: pulumi.String("S"),
			},
		}

		// The set of arguments for constructing an Amazon DynamoDB Table resource
		tableArgs := &dynamodb.TableArgs{
			Attributes:    dynamodb.TableAttributeArray(tableAttributeInput),
			BillingMode:   pulumi.StringPtrInput(dynamoConfig.BillingMode),
			HashKey:       pulumi.String("PK"),
			RangeKey:      pulumi.String("SK"),
			Name:          pulumi.String(fmt.Sprintf("%s-%s", ctx.Stack(), ctx.Project())),
			ReadCapacity:  dynamoConfig.ReadCapacity,
			WriteCapacity: dynamoConfig.WriteCapacity,
		}

		// NewTable registers a new resource with the given unique name, arguments, and options
		table, err := dynamodb.NewTable(ctx, fmt.Sprintf("%s-%s", ctx.Stack(), ctx.Project()), tableArgs)
		if err != nil {
			return err
		}

		// Export the ARN and Name of the table
		ctx.Export("Table::Arn", table.Arn)
		ctx.Export("Table::Name", table.Name)

		return nil
	})
}

Continuous Anything

As I was building out the services for the ACME Serverless Fitness Shop, I read Stackery’s Road to Serverless Ubiquity Guide. In that report, they have a few awesome bits of information. There was one paragraph that stood out for me on serverless developer experience.

“But developers are human beings, too—and their experience of these tools and technologies is extremely important if we want to encourage sustainable and repeatable development practices.”

I think that sustainable and repeatable development practices, regardless of whether that is for serverless or not, is incredibly important. As a developer you want to have repeatable processes and repeatable builds and luckily there are tons of options for that too. One of my friends introduced me to CircleCI. CircleCI has a concept of Orbs, which are reusable snippets of code that help automate repeated processes, speed up project setup, and make it easy to integrate with third-party tools. That saves a ton of work configuring and writing deployment scripts. All services, including DynamoDB and SQS, have their CircleCI pipeline and each of those pipelines is only 35 lines of configuration. Of those 35 lines, most are copied and pasted from the starter template.

Wrapping up

To wrap up, in this part of the series on the ACME Serverless Fitness Shop we looked at choosing:

  • A data store, DynamoDB, because it is the best purpose built database for the access patterns that the ACME Serverless Fitness Store needs
  • The application integration service, SQS, because it allows the functions to operate asynchronously
  • The compute resources, Lambda, for it’s event-driven and cost aspects
  • The Infrastructure as Code tool, Pulumi, so that I can write Go to deploy my Go functions
  • The CI/CD tool, CircleCI, because using Orbs drops the amount of configuration drastically

This post also touched a bit on moving from microservice to serverless. Next time I’ll dive a bit more into what Continuous Verification means for serverless workloads. In the meanwhile, let me know your thoughts and send me or the team a note on Twitter.

Photo by Humphrey Muleba on Unsplash