If you’ve read the blog posts on CloudJourney.io before, you’ve likely read the term “Continuous Verification”. If you haven’t that’s okay too. There’s an amazing article from Dan Illson and Bill Shetti on The New Stack explaining in detail what Continuous Verification is. In a nutshell, the Continuous Verification comes down to making sure that DevOps teams put as many checks as possible into their CI/CD pipelines. Adding checks into a pipeline means there are fewer manual tasks and that means you have access to more data tot smooth out and improve your development and deployment process.

In part one of this series, we’ve gone over the tools and technologies that make up the ACME Serverless Fitness Shop. Now, it’s time to dive into the depths. Let’s check out Continuous Everything!

What is the ACME Serverless Fitness Shop

Let’s go over what the ACME Serverless Fitness Shop is just one more time. It’s a shop that combines the concepts of Serverless and Fitness, which are two of my favorite things, because combining two amazing things can only lead to more amazing outcomes. There are seven distinct domains that all contain one or more serverless functions. Some of these services are event-driven, while others have an HTTP API and all of them are written in Go.

Continuous Anything

Continuous Anything is more than just the title of this blog post. It captures the idea that all the practices that start with “Continuous” should be able to work together in a single run from code to production. It should capture the integration (building and testing code), deployment (getting the builds to staging and production), and verification (are we sure this deployment is the right thing to do).

There are tons of options when it comes to choosing a CI/CD tool. You can choose products like Jenkins, Travis CI, or CircleCI. To help differentiate between them, I’ve come up with a few requirements that are important to the ACME Serverless Fitness Shop. Serverless means not running your own servers, so the number one requirement is that the CI/CD tool has to be a managed service. There are multiple service domains, with their own repositories, but some variables will be the same across all repositories and should be set in a single place.

If you’ve ever done something with CI/CD, you’ll likely know Jenkins. Jenkins is a great tool, but pretty much everything requires a plugin that you need to install on the server. If you’re building or testing code or parsing output from a command-line tool, you need to make sure that those tools are available on the machine that runs the builds. As far as I know, there is no managed service for Jenkins available.

Travis CI is another well-known tool. Builds in Travis CI run in a container, or virtual machine, with almost no additional tools installed other than the language you’ve chosen. You also cannot set environment variables across multiple projects.

That brings me to CircleCI. First of all, CircleCI is a managed service with a pretty generous free tier. CircleCI offers orbs, which are essentially reusable pieces of YAML code. Those can be configuration, commands, or even entire CI jobs. It’s the idea of plugins, without requiring the system engineers to install them first. The ACME Serverless Fitness Shop has seven domains of services and each of those services has its own GitHub repository. They also provide “Build Contexts”, which make it possible to share environment variables across builds. I only have to update my Sentry DSN in a single place and it’ll be made available to all builds.

Continuous Integration

Back in 1991, Grady Booch coined the term Continuous Integration as “the practice of merging all developers’ working copies to a shared mainline several times a day”. In today’s world that makes a lot of sense as more people get involved in software development. Instead of letting everyone build features and patch issues individually and have them merge their codebases at the end of the day, it suggests developers integrate (or merge) their code multiple times per day. The rationale behind that idea is that the longer a developer works on a separate copy of a codebase, the higher the chance something else changes that creates a conflict. These conflicts can come from simple things like library versions or very complex things like different developers updating the same method in the same file. A typical workflow for Continuous Integration runs builds and tests and gives feedback when the build or tests fail. To do that in CircleCI, you’ll need 10 lines of YAML. The image used is the new next-gen convenience image from CircleCI which has a bunch of useful Go tools already embedded in it.

# The version of the CircleCI configuration language to use (2.1 is needed for most orbs)
version: 2.1
jobs:
  build:
    docker:
      # The ACME source code is in Go, so we'll rely on the image provided by the CircleCI team (1.14 is the latest version at the time of writing)
      - image: cimg/go:1.14
    steps:
      # Get the sources from the repository
      - checkout
      # Get all dependencies for both code execution and running tests (-t) and downloaded packages shouldn't be installed (-d)
      - run: go get -t -d ./...
      # Run gotestsum for all packages, which is a great tool to run tests and see human friendly output
      - run: gotestsum --format standard-quiet
      # Compile and build executables and store them in the .bin folder
      - run: GOBIN=$(pwd)/.bin go install ./...

Continuous Delivery

While there is a great definition of what Continuous Integration is, finding a proper definition of Continuous Delivery is a lot harder. Thanks to a lot of feedback from Matty Stratton, Laura Santamaria, and Aaron Aldrich the general consensus seems to be that Continuous Delivery is being able to test the entire code and prepare it to be deployed to production. The final stage of getting it into production has to be approved manually. Testing the code also takes into account the ability to put this into production. For the ACME Serverless Fitness Shop, I decided to use Pulumi to do that. I want to use a tool that doesn’t have a custom domain-specific-language. As a developer, I’m most definitely not a YAML expert and I enjoy writing Go. To make the integration between CircleCI and Pulumi even easier, the Pulumi team built an orb. That allows me to use Pulumi in my CircleCI workflows, without having to take care of the installation and boilerplate code. In a future post, we’ll dive into how the Pulumi scripts are built.

version: 2.1
# Register the Pulumi orb
orbs:
  pulumi: pulumi/pulumi@1.2.0
jobs:
  build:
    docker:
      - image: circleci/golang:1.14
    steps:
      - checkout
      - run: go get -t -d ./...
      - run: gotestsum --format standard-quiet
      - run: GOBIN=$(pwd)/.bin go install ./...
      # Log in to Pulumi using a specific version of the CLI
      - pulumi/login:
          version: 1.12.1
      - pulumi/preview:
          stack: retgits/dev
          working_directory: ~/project/pulumi
workflows:
  version: 2
  deploy:
    jobs:
      - build:
          # The context gives the ability to set environment variables that are shared across pipelines.
          # In this case the ACMEServerless context has the Pulumi Token that is needed by the Pulumi orb
          context: ACMEServerless

Right now, the YAML file is up to 23 lines and already automated a large part of getting the ACME Serverless Fitness Shop into production. The pipeline completely builds and tests the code for both the app and the infrastructure. It also sets the context, which is a set of environment variables, so all builds have the same context.

Continuous Verification

To make sure we’re all on the same page, as a definition, Continuous Verification is “A process of querying external system(s) and using information from the response to make decision(s) to improve the development and deployment process.” Within Continuous Verification, there are many things you can add but let’s focus on four important ones:

  • Security: We want to make sure that we use safe Go modules and that things like IAM settings are right
  • Performance: We want to make sure that the performance of the components is acceptable
  • Utilization: We want to set a baseline for the amount of memory used by the components
  • Cost: We want to have an estimate of what running these components will cost

Let’s start with security because at the end of the day it is a shared responsibility to make sure we all build safe and secure software. I’ve written about using Snyk before, so I’ll not go over the amazing technology they offer. To scan the Go modules literally takes two lines of additional YAML. You have to add the orb (snyk: snyk/snyk@0.0.10) and you have to add the scan step (snyk/scan).

After you’ve invoked the AWS Lambda function a few times, you can use the AWS CLI to get some performance statistics. To do that, you’ll need four steps:

  • Add the AWS CLI orb to your YAML file: aws-cli: circleci/aws-cli@0.1.22
  • Add the setup step: aws-cli/setup (if you’re not using a context, you really should, because it saves you from adding additional configuration in your YAML)
  • Add a run step to get statistics:
export FUNCTION=AllCarts && export ENDDATE=`date -u '+%Y-%m-%dT%TZ'` && export STARTDATE=`date -u -d "1 day ago" '+%Y-%m-%dT%TZ'` && export DURATION=`aws cloudwatch get-metric-statistics --metric-name Duration --start-time $STARTDATE --end-time $ENDDATE --period 3600 --namespace AWS/Lambda --statistics Average --dimensions Name=FunctionName,Value=$FUNCTION | jq '.Datapoints | map(.Average) | add'` && if (($DURATION > 3000)); then echo "Alert" && exit 1; else echo "Within range. Continuing"; fi
  • Decide what a proper time for the $DURATION should be. In the above example, the acceptable duration is set to 3000 milliseconds.

In a blog on tracing, we’ll dive a bit more into using tools like VMware Tanzu Observability by Wavefront, to check what running your functions might cost you and which percentage of memory is actually used.

The end result of adding the steps and orbs:

version: 2.1
orbs:
  pulumi: pulumi/pulumi@1.2.0
  snyk: snyk/snyk@0.0.10
  aws-cli: circleci/aws-cli@0.1.22
jobs:
  build:
    docker:
      - image: circleci/golang:1.14
    steps:
      - checkout
      - run: go get -t -d ./...
      - run: gotestsum --format standard-quiet
      - run: GOBIN=$(pwd)/.bin go install ./...
      - snyk/scan
      - pulumi/login:
          version: 1.12.1
      # We're updating the stack instead of only showing the preview
      - pulumi/update:
          stack: retgits/dev
          working_directory: ~/project/pulumi
          skip-preview: true
      - aws-cli/setup
      - run: export FUNCTION=AllCarts && export ENDDATE=`date -u '+%Y-%m-%dT%TZ'` && export STARTDATE=`date -u -d "1 day ago" '+%Y-%m-%dT%TZ'` && export DURATION=`aws cloudwatch get-metric-statistics --metric-name Duration --start-time $STARTDATE --end-time $ENDDATE --period 3600 --namespace AWS/Lambda --statistics Average --dimensions Name=FunctionName,Value=$FUNCTION | jq '.Datapoints | map(.Average) | add'` && if (($DURATION > 3000)); then echo "Alert" && exit 1; else echo "Within range. Continuing"; fi
workflows:
  version: 2
  deploy:
    jobs:
      - build:
          context: ACMEServerless

Right now, we’re up to 29 lines of YAML to do:

  • Build and test the Go code for the Lambda functions
  • Scan the Go modules for security vulnerabilities
  • Validate the infrastructure deployment
  • Update the development environment
  • Run a performance check

To me, the best part is that all builds can use the exact same steps and I don’t have to share any credentials thanks to the CircleCI context.

What’s next?

We’ve looked at what role CircleCI plays in the ACME Serverless Fitness Shop. Next time I’ll dive a bit more into the Infrastructure as Code using Pulumi. In the meanwhile, let me know your thoughts and send me or the team a note on Twitter.

Cover photo by Magda Ehlers from Pexels