With more and more applications using public cloud (AWS/Azure/GCP) and the ever changing number of features services that are available on these hyperscalers, how do you maintain a stable process of deploying and managing resources and applications in the public cloud?

We know that most enterprise companies have a shift in organizational boundaries to start and “Grapple” with this shift. That shift is a segmenting of their IT organization into two distinct parts

  1. Traditional IT - tasked with managing “on-prem” data centers and usually requires the individual in this organization to have a balance between HW and SW knowledge and skills. SW usually relates to HW and virtualization and networking configurations.
  2. Cloud Operations - tasked with deploying and managing environments in public clouds and/or K8S conformant services. These individuals need to have heavy skill sets in development (coding with APIs), CI/CD, security for applications and managed services.

Cloud Operations is where the cross between traditional Line of Business DevOps merges with individuals with skill sets to manage the deployments (CD) on public cloud and manage the overall guardrails and health of the public cloud environment (general AWS/Azure/GCP management) and application deployments.

In a previous set of blogs we talked about a few concepts:

“Fences and Gates” which outlined organizational guardrails, and interconnection points between these areas. Guardrails can be multiple things:

  1. Cost and Resource Guardrails - Managing the amount of capacity used by your developers during the CI/CD process. We explored this during our blog “Budget check during CI/CD”
  2. Continuous Security - Managing several potential threats during the CD process which included network risks, application risks, and general compliance and governance. We explored the general compliance and governance during this blog “Continuous Security”
  3. Performance validation - Managing your application performance by analyzing past history to ensure the new deployment will meet existing needs. We will be detailing this in a future blog.

In this particular blog we will explore the aspects of Application security and part of Continuous Security, and in particular look at how to manage application level vulnerabilities in containers (or general VM based applications).

Application Security

As a cloud Ops person the maintaining the guardrails mentioned above is difficult enough when dealing with all the aspects surrounding hyper-scaler operations. Application development and security adds another twist. While its generally been relegated to individual projects the use of containers and the ability to run an application on ANY CLOUD requires the Cloud Ops person to understand application security during the deployment process and how to use the status in managing new and existing deployments.

First lets understand how to check for application security. There are many different steps that can be taken to ensure more robust application security.

  1. Code level analysis - This is known in the industry as static testing. Developers generally check their code as they are writing it to ensure security issues are NOT being introduced during development. The tools which help in this are called Static Application Security Testing (SAST) - There is a great list here https://www.owasp.org/index.php/Source_Code_Analysis_Tools

  2. Dependency checking - While checking static code is important, so is ensuring libraries and dependencies during the build process is not affected. These tools help identify project dependencies and checks if there are any known, publicly disclosed, vulnerabilities. Its a guarantee that the libraries included in any code (Ruby, Java, Python, etc) will have an issue. Most recently - Equifax got hacked based on an Apache Vulnerability that was not detected when they deployed the latest version of their service. - https://nvd.nist.gov/vuln/detail/CVE-2017-5638. The most popular project know for this is “OSWAP Dependency Check”

  3. Image Scanning - If static analysis and dependency checks pass, then most likely the image will be built. In our modern world of container based applications, this is known as container scanning. This will alert of known CVEs. Some popular tools in this category are “Clair”, etc.

  4. Analyze running code - This is known in the industry as dynamic testing. Vs SAST - these tools can simulate attacks on production systems and alert you to complex attack patterns that use a combination of systems. These tools are generally known as Dynamic Application Security Testing (DAST) or Vulnerability Scanning Tools. There is a great list here https://www.owasp.org/index.php/Category:Vulnerability_Scanning_Tools

All of these steps, which are supported by known tools, can also be integrated into the CI/CD process. In some cases its a manual integration while in other cases its part of the CI/CD tool/service.

Pipeline Stages

In one of our previous blogs we explored the use of “Gitlab” with respect to managing budget in deploying into a K8S environment. We showed how to add a python script that detected the potential over-run of resources in a project using K8S clusters on GKE. The script pinged “Cloudhealth by VMware”. We used simple variables and files to transfer the state from one stage to another. However this was all in the “Deployment” part of the process.

However, Gitlab provides its end users with the ability to run different tools (from the specific categories mentioned above) in their project pipelines. In particular it enables users to run tools to support:

  1. Source_Code_Analysis_Tools
  2. DAST (Vulnerability_Scanning_Tools)
  3. Container scanning
  4. etc.

NOTE: While we can describe and review how to implement all of these, the most important and BASIC step is to scan created images (container scanning). Its as basic as “brushing your teeth” - just good hygiene.

In this blog, I will highlight how to run and analyze a Container Vulnerability using Gitlab. In particular using “Clair”.

We picked “Gitlab” because of our previous blogs and it provides continuity. However you can also solve this with other options. Here are some iterations that are also available:

Manual scanning via - CI/CD:

  1. Jenkins + Anchore engine
  2. Codefresh + Clair
  3. etc.

Registry scanning support:

  1. “Harbor Container Scanning” - (CNCF Container registry by VMware) also automatically checks for vulnerabilities during a push using CVE databases like Debian Sec Bug Tracker, Ubuntu CVE Tracker, Red Hat Security Data, Oracle Linux Sec Data, Alphine SecDB and NIST NVD.
  2. “Quay.io” - while Quay.io provides the ability to also build containers, its main role is a to support a managing a docker registry. It has a capability to automatically scan for vulnerabilities during a push using CVE databases.
  3. etc.

There are numerous options and combinations available for manual scanning, and registry scanning. Hence there is NO EXCUSE to NOT use this vulnerabilities with your containers.

Testing Container for vulnerabilities

Setting container scanning in “Gitlab” is fairly straight forward, with the right additions to the CI/CD script to ensure scanning is actually executed. However, there are a few things we should cover prior to covering the configuration.

  1. How “Clair” works to check for security violations
  2. When to appropriately test the container image.
  3. Install and run the configuration with “Clair” on “Gitlab”

Claire Security Violations

Per CoreOS site (https://coreos.com/clair/docs/latest/)

Clair is an open source project for the static analysis of vulnerabilities in application containers (currently including appc and docker).

In regular intervals, Clair ingests vulnerability metadata from a configured set of sources and stores it in the database. Clients use the Clair API to index their container images; this creates a list of features present in the image and stores them in the database. Clients use the Clair API to query the database for vulnerabilities of a particular image; correlating vulnerabilities and features is done for each request, avoiding the need to rescan images. When updates to vulnerability metadata occur, a notification can be sent to alert systems that a change has occured. Our goal is to enable a more transparent view of the security of container-based infrastructure. Thus, the project was named Clair after the French term which translates to clear, bright, transparent.

Clair is generally deployed as a container with either docker compose or K8S as a container and a postgress database(s) - generally redundant.

  1. The Clair container has three basic components -
  1. Postgress DB
  • Stores all of the vulnerabilities in a database and results of analysis of uploaded docker image layers
  • Generally configured as redundant

Clair Architecture

  1. Clair updates its vulnerabilities database.
  2. Clair container APIs are called and individual docker image layers are sent to the Clair container for analysis.
  3. Clair checks against its vulnerabilities with each layer by accessing the vulnerabilities from Postgress
  4. The issues from each vulnerability is stored in Postgress

Once the detected vulnerabilities are stored, they can then be listed and analyzed further.

Where to test for CVEs?

Depending on how you your processes are configured

Example vulnerability check using a Gitlab configuration

In creating the example in Gitlab we used the following two components:

  1. We used the famous Kubernetes application example “Guestbook” which uses redis and php. And in this scanning example we check for vulnerabilities from a standard prebuilt php:5-apache docker image used in the Dockerfile.

  2. We use Gitlab’s “Security” capabilities which are detailed here - “Security Dashboard”. While it shows information about DAST, SAST, container vulnerabilities, etc. We used the container vulnerabilities only.

Gitlab Security Dashboard

  1. We used a configuration where we created a docker runner on a Ubuntu box which was registered to the project. This runner allowed for Docker in docker to be run. Important since it will allow us to not only build the container, but also test it against the vulnerabilities

  2. We used Gitlab’s pre-defined container scanning template Container-Scanning.gitlab-ci.yml which runs Claire against the image in question.

Here is the configuration:

  1. For the example we created a project in Gitlab called guestbook-ci and copied the Github content from “Guestbook” into this project.

Guestbook Project

  1. Next we created created a .gitlab-ci.yaml file with a scanning stage.
include:
  template: Container-Scanning.gitlab-ci.yml

stages:
  - docker_build
  - scanning


docker_build:
  stage: docker_build
  image: docker:stable
  variables:
    DOCKER_HOST: tcp://docker:2375/
    DOCKER_DRIVER: overlay2
  services:
    - docker:dind
  before_script:
    - docker info
    - docker login registry.gitlab.com -u "$REGISTRY_USER" -p "$REGISTRY_PASSWORD"
  script:
    - docker info
    - docker build -t gitlab-cart .
    - docker tag gitlab-cart registry.gitlab.com/bshetti/guestbook-ci/guestbook-fe:latest
    - docker push registry.gitlab.com/bshetti/guestbook-ci/guestbook-fe:latest

container_scanning:
  stage: scanning

Note the inclusion of the following:

template: Container-Scanning.gitlab-ci.yml

This is a template from gitlab that will do the following parts during the scanning stage noted in the script above:

  1. Create a container_scanning job in your CI/CD pipeline.
  2. Pull the already built Docker image from your project’s Container Registry (see requirements) and scan it for possible vulnerabilities.

Here is the execution:

$ mv clair-scanner_linux_amd64 clair-scanner
$ chmod +x clair-scanner
$ touch clair-whitelist.yml
$ retries=0
$ echo "Waiting for clair daemon to start"
Waiting for clair daemon to start
$ while( ! wget -T 10 -q -O /dev/null http://${DOCKER_SERVICE}:6060/v1/namespaces ) ; do sleep 1 ; echo -n "." ; if [ $retries -eq 10 ] ; then echo " Timeout, aborting." ; exit 1 ; fi ; retries=$(($retries+1)) ; done
$ ./clair-scanner -c http://${DOCKER_SERVICE}:6060 --ip $(hostname -i) -r gl-container-scanning-report.json -l clair.log -w clair-whitelist.yml ${CI_APPLICATION_REPOSITORY}:${CI_APPLICATION_TAG} || true
2019/06/26 21:00:07 [INFO] ▶ Start clair-scanner
2019/06/26 21:00:22 [INFO] ▶ Server listening on port 9279
2019/06/26 21:00:22 [INFO] ▶ Analyzing 08bf86d6624450c487db18071224c88003d970848fb8c5b2b07df27e3f6869b2
2019/06/26 21:00:23 [INFO] ▶ Analyzing ea9d1f8d70cd781e358b2d95e8a921107f6e695f01e8aeef83cf7c48e177e2d1
2019/06/26 21:00:23 [INFO] ▶ Analyzing b630615155cae79cd4ea2cdd02512a00f398d3a7ae1c4df3c369174db851452e
2019/06/26 21:00:24 [INFO] ▶ Analyzing 801875422a8dcfecd990d09c03333f5b749736e1cfd33ff2ee8e985309bd6323
2019/06/26 21:00:24 [INFO] ▶ Analyzing ecd2ce6aeb56ad93f3b7bdd66e966612269587eac40e9a0c35b25372c80b0259
2019/06/26 21:00:24 [INFO] ▶ Analyzing 48659871a5886384ef397e9259ff7c4f12b255e914c113fb3cca453fbff52410
2019/06/26 21:00:24 [INFO] ▶ Analyzing ec9a30fde3b864a97e5b3e531cf7d7c5570a0190536f8dec9929cf5e6aa96747
2019/06/26 21:00:24 [INFO] ▶ Analyzing 1fdf3fa400d39907deb23653b363a508144a8001321c524b7aea774a3da698bb
2019/06/26 21:00:24 [INFO] ▶ Analyzing f8978a6adea0ac4be384e8838330415e39fe185b2ea54f3b86b6c6d3ca6924ca
2019/06/26 21:00:25 [INFO] ▶ Analyzing 93795926c8793cd24f83fc6dbb5f8899a5a4b4ab524053393c56b622c7f67093
...
...

The output of this finds a significant number of vulnerabilities

Guestbook Docker Image Scan

Conclusion

Using container vulnerability is a simple step to significantly increase application security. Its a simple setup in a CI/CD tool like Gitlab and/or a simple feature in repository (like Harbor). Hopefully the configuration and output shown above give you confidence to implement this security step in your CI/CD pipeline to ensure your developers are not causing un-necessary harm through their development process.