News about data breaches, leaked customer information and stolen passwords for critical infrastructure are becoming very common. Many of these incidents seem to be related to mismanagement of credentials, unencrypted passwords, secrets being pushed to git repositories or secrets being hard coded within the application, leaving no room for rotation.
This has led to increasing demand for Secrets Management tools like AWS Secrets Manager, HashiCorp Vault, Confidant and others.
In this blog, we will look at
1. The need for secrets management
2. Overview - AWS Secrets Manager and HashiCorp Vault
3. AWS secrets manager vs HashiCorp Vault
4. Using AWS secrets manager and HashiCorp Vault within your GO application
5. Other use cases for Secret Management tools
Now, let’s look at some of the reasons for the adoption of secrets management.
The need for secrets management
Some of the common issues occurring within application and infrastructure teams with secrets are:
1. Adoption of Microservices architecture means that these services would communicate with each other and external services by using API keys and other secrets (database passwords), etc. In such a scenario, passing them individually via environment variables can get difficult.
2. Hardcoding of secrets within an application is NOT a good practice as there is the risk of these secrets being exposed on git repositories.
3. Hackers usually tend to exploit unencrypted secrets and keys. This is one of the main reasons for data breaches.
4. Teams working in shared public cloud environments, need strict access controls on who can access what portion of the infrastructure. This might mean sharing of API tokens, SSH keys or need for temporary credentials. This will allow for centralized secrets management.
The solution to all of this is a secrets management tool. These tools tend to provide capabilities like key rotation, encryption, role-based access controls, and API access. All of which makes it very attractive to DevOps engineers.
In the following sections, we will look at 2 of the most popular secrets management tool, AWS Secrets Manager and HashiCorp Vault.
Overview - AWS Secrets Manager
AWS Secrets Manager is a managed regional service that enables you to easily rotate, manage, and retrieve database credentials, API keys, and other secrets throughout their lifecycle. It works seamlessly with native services like RDS, RedShift, and DocumentDB.
It can also store various other kinds of secrets like API keys or SSH keys. You can provide any KEY VALUE Pairs.
You can also add password rotation for other services and NON-AWS databases by using lambda functions. For RDS, RedShift, and DocumentDB, you DON’T have to write any scripts and functions.
Access control for users can be enabled by using IAM.
Overview - HashiCorp Vault
Vault is an Opensource tool with a freemium model (You can buy enterprise version with more capabilities). The user has to host and maintain this service. It works with all the major cloud providers like AWS, GCP, and Azure to generate dynamic credentials for user access. It also has integrations to store keys for various databases like Mongo, MySQL, etc.
Vault also provides an access control mechanism to restrict access to users. It has both a CLI and web interface. It also provides the ability to add lease duration to secrets after which the secrets would be expired and no longer valid.
In the next section, we will look at how we can use these tools within an application.
Using AWS Secrets Manager with a Go Application
I will be setting up AWS secrets manager and walk through the scenario of accessing database keys (MongoDB) within a Golang Service. This is a catalog service, that displays a catalog of items and exposes REST APIs. It is part of the broader ACME Shop App
The same steps can be followed to retrieve and store other kinds of API keys and secrets necessary for your application.
Steps:
1. Setup AWS secrets manager
The first step is to create a new secret. It offers various kinds of secret ‘s store, for example, RDS, RedShift, DocumentDB, etc.
I will be choosing Other type of secrets to store username and password for MongoDB. At the time of writing this blog, there is no native integration with MongoDB.
Add the key-value pairs that you would like to store and click Next.
2. Select the encryption key
Next step is to select an encryption key or use the default one provided by AWS. I will be choosing the default encryption key as shown above. But you can also create your encryption key by using AWS KMS. You can find more details here
3. Now provide a name and description for the secret. It’s always a best practice to add description and tags to all the resources, especially when working in a shared environment with other team members.
4. In this step, you can configure secret rotation. For Native services like RDS, RedShift, AWS provides built-in key rotation, which means that AWS will securely configure these database instances with new random secrets at the set rotation interval.
But in our scenario with MongoDB, we will have to write our lamdba function to create a new secret. This function should contain the following steps:
a. Access the current keys from AWS Secrets Manager - This is usually labeled as AWSCURRENT version (we will see this below). Generate a new strong password while keeping the username and connecting string the same. In our example, we won’t be storing the connection string in the secret’s manager for simplicity.
b. Execute the update secret keys command - Add this new key to store using a PUT call as mentioned in the API doc. This new secret will be labeled as AWSPENDING
NOTE: There will be no effect to the application at this time because the App always tries to retrieve the version labeled as AWSCURRENT. We will see this in the section below.
c. Set new password for user - Use the GET call to retrieve AWSPENDING version of the password and use the commands or APIs for the specific database to set the new password for the user.
```
use catalogDB
db.changeUserPassword("mongoadmin", "SOh3TbYhx8ypJPxmt1oOfL")
```
In case of MongoDB, the commands above will update the password for a specific user, in our case that’s mongoadmin.
d. Test the new password to ensure that the user can log in by using GET call for version labeled as AWSPENDING.
e. Finally, if the password works, then update the label for the new password from AWSPENDING to AWSCURRENT. Now the application will use the latest password.
f. Between steps d and e, the user password is updated but the application is still using the old password (labeled as AWSCURRENT). This might result in AccessDenied error.
I will be using NO rotation for this policy.
5. Finally store the secret. Now, this secret can be accessed from the secrets manager API.
After setting up the secrets manager, you need to create an IAM role that allows for LIST and READ access to secrets manager.
[secretsmanager:ListSecrets, secretsmanager:ListSecretVersionIds, secretsmanager:GetSecretValue]
Within the code itself, add the following AWS packages to enable secrets manager
package main
import (
"fmt"
"os"
"github.com/globalsign/mgo"
"github.com/sirupsen/logrus"
"github.com/aws/aws-sdk-go/service/secretsmanager"
"github.com/aws/aws-sdk-go/aws"
"github.com/aws/aws-sdk-go/aws/awserr"
"github.com/aws/aws-sdk-go/aws/session"
"encoding/json"
)
Next, initialize a variable that will be used to parse the data returned from the Secrets Manager. Also, initialize other variables needed for MongoDB, that will be used by the service.
// Variable to store data after parsing result from Secrets Manager
var res map[string]interface{}
// Variables for MongoDB
var (
// Mongo stores the mongodb connection string information
mongo *mgo.DialInfo
db *mgo.Database
collection *mgo.Collection
)
The function to retrieve the secrets is usually provided by AWS for various programming languages. But you still need to write logic to parse and store the creds within the app.
// Beginning of getSecret Function.
func getSecret() (string, string){
// Provide the name of the secret
secretName := "catalog/MongoDB"
region := "us-west-2"
// Obtain the keys from the environment variable. This can be either AWS Access keys and secret keys or STS IAM role. Refer to AWS documentation for more details
sess, err := session.NewSession(&aws.Config{
Region: aws.String(region)},
)
if err != nil {
logger.Errorf("Error from getSecret is %s", err.Error())
}
//Create a Secrets Manager client
svc := secretsmanager.New(sess)
input := &secretsmanager.GetSecretValueInput{
SecretId: aws.String(secretName),
VersionStage: aws.String("AWSCURRENT"), // VersionStage defaults to AWSCURRENT if unspecified
}
// In this sample we only handle the specific exceptions for the 'GetSecretValue' API.
// See https://docs.aws.amazon.com/secretsmanager/latest/apireference/API_GetSecretValue.html
result, err := svc.GetSecretValue(input)
if err != nil {
if aerr, ok := err.(awserr.Error); ok {
switch aerr.Code() {
case secretsmanager.ErrCodeDecryptionFailure:
// Secrets Manager can't decrypt the protected secret text using the provided KMS key.
fmt.Println(secretsmanager.ErrCodeDecryptionFailure, aerr.Error())
case secretsmanager.ErrCodeInternalServiceError:
// An error occurred on the server side.
fmt.Println(secretsmanager.ErrCodeInternalServiceError, aerr.Error())
case secretsmanager.ErrCodeInvalidParameterException:
// You provided an invalid value for a parameter.
fmt.Println(secretsmanager.ErrCodeInvalidParameterException, aerr.Error())
case secretsmanager.ErrCodeInvalidRequestException:
// You provided a parameter value that is not valid for the current state of the resource.
fmt.Println(secretsmanager.ErrCodeInvalidRequestException, aerr.Error())
case secretsmanager.ErrCodeResourceNotFoundException:
// We can't find the resource that you asked for.
fmt.Println(secretsmanager.ErrCodeResourceNotFoundException, aerr.Error())
}
} else {
// Print the error, cast err to awserr.Error to get the Code and
// Message from an error.
fmt.Println(err.Error())
}
return err.Error()
}
// Decrypts secret using the associated KMS CMK.
// Depending on whether the secret is a string or binary, one of these fields will be populated.
var secretString string
if result.SecretString != nil {
secretString = *result.SecretString
}
// Convert to byte
bytes := []byte(secretString)
// Unmarshal and store it in the variable *res* created in previous step
json.Unmarshal([]byte(bytes), &res)
// Retrieve username and password for the database
dbUser := res["username"].(string)
dbPass := res["password"].(string)
return dbUser, dbPass
}
Finally, add the code for connecting to the MongoDB. The entire application/service code is available on github
// ConnectDB accepts name of database and collection as a string
func ConnectDB(dbName string, collectionName string, logger *logrus.Logger) *mgo.Session {
// Retrieve Secret from AWS Secrets Manager
dbUsername, dbSecret := getSecret()
// Get ENV variable or set to default value. These vaules can also be store in secrets manager
dbIP := GetEnv("CATALOG_DB_HOST", "0.0.0.0")
dbPort := GetEnv("CATALOG_DB_PORT", "27017")
mongoDBUrl := fmt.Sprintf("mongodb://%s:%s@%s:%s/?authSource=admin", dbUsername, dbSecret, dbIP, dbPort)
Session, error := mgo.Dial(mongoDBUrl)
if error != nil {
fmt.Printf(error.Error())
logger.Fatalf(error.Error())
os.Exit(1)
}
db = Session.DB(dbName)
error = db.Session.Ping()
if error != nil {
logger.Errorf("Unable to connect to database %s", dbName)
}
collection = db.C(collectionName)
logger.Info("Connected to database and the collection")
return Session
}
As you can see that the setup is pretty quick with AWS Secrets Manager. It comes with a pretty decent set of pre-built lambda functions if you need them.
Using HashiCorp Vault within Go Application
We will now walk through the setup for HashiCorp Vault for the same go application shown above.
Steps:
1. Install and Start Vault
2. The secrets can be added to the vault either by using CLI or by using the APIs.
I will discuss adding the secrets using the CLI.
3. If you are using this in a production environment then enable authentication on vault. It can be either from 3rd party services like GitHub, simple username and password or a token based on policy.
In this blog, we will use policy to generate a token as the authentication mechanism.
NOTE: For other authentication mechanisms, review this link
Next, create a file called policy.hcl and add the following content to it. This will allow authenticated users to perform all the operations except for delete
path "secret/data/catalogdb" {
capabilities = ["create", "read", "update", "list"]
}
4. Upload this policy to the vault and verify.
vault policy write catalog-policy policy.hcl
Ensure that policy was uploaded by using the following commands
vault policy list
This is the expected output
default
catalog-policy
root
5. Generate a new vault token with the new policy and login using the new token
vault token create -policy=catalog-policy
vault login <ADD token here>
It should look similar to this.
6. Now add the secret to the vault.
vault kv put secret/catalogdb username=mongoadmin passsword=secret
Once this is done, the secret is now stored and can be accessed from location secret/data/catalogdb (This is for version 2 of the kv store)
Now let’s look at how we can access this from the application. Set the VAULT_ADDR and TOKEN (generated in step 6) as part of the application bring up.
Import the packages necessary for HashiCorp vault.
package main
import (
"fmt"
"os"
"github.com/globalsign/mgo"
"github.com/sirupsen/logrus"
"github.com/hashicorp/vault/api"
)
Next, initialize a variable that will be used to parse the data returned from the Vault. Also, initialize other variables needed for MongoDB, that will be used by the catalog service.
// Variable to store data after retrieving secrets from the vault
var res map[string]string
// Set vault address
var vaultAddress = os.Getenv("VAULT_ADDR")
// Set token obtained from previous steps
var token = os.Getenv("TOKEN")
// Variable needed for MongoDB
var (
// Mongo stores the mongodb connection string information
mongo *mgo.DialInfo
db *mgo.Database
collection *mgo.Collection
)
Now retrieve the secret from the vault.
func getSecret() (string, string) {
// Path to secret within Vault
secretName := "secret/data/catalogdb"
// Start a new connection with the vault
client, err := api.NewClient(&api.Config{
Address: vaultAddress,
})
// Set Token - This token can be based on the role and permissions set within the vault
client.SetToken(token)
if err != nil {
logger.Errorf("Error from getSecret is %s", err.Error())
}
// Read the secret - This is version 2 of the KV store
res, err := client.Logical().Read("secret/data/catalogdb")
if err !=nil {
logger.Errorf("Error retrieving Secret from vault %s", err.Error())
}
vaultData := res.Data["data"]
// Get the vaules for db username and password
dbUser := vaultData.(map[string]interface{})["username"].(string)
dbPass := vaultData.(map[string]interface{})["password"].(string)
return dbUser, dbPass
}
As you can see that vault needs some setup time and will incur operation costs to keep it running.
Now that we have an idea of how these two services function, let’s look at some of the differences.
AWS Secrets Manager vs HashiCorp Vault
Here are some of the differences between AWS Secrets Manager and HashiCorp Vault
| | AWS Secrets Manager | HashiCorp Vault |
|---- |----------------------------------------------------------------------------------------- |--------------------------------------------------------------------------------------------------------------------- |
| 1 | Fully Managed Service | Self Hosted and Operated |
| 2 | Seamless secrets rotation for native services like RDS, RedShift and DocumentDB | Secrets Rotation has to be done manually or with scripts |
| 3 | No lease periods for secrets | Duration can be set for how long the access to secrets are allowed |
| 4 | Coupled to AWS ecosystem | Cloud Agnostic i.e. Works with AWS, Azure, and GCP |
| 5 | Limited (MariaDb, MySQL, Postgres) or no native integration with Mongo and Kubernetes | Integrations for various opensource and licensed databases like Influx, Mongo, Hana as well as Kubernetes secrets |
| 6 | Paid service - Charged for API calls as well as storage | Freemium model - They have an Opensource version as well as Enterprise version | |
Other Use Cases for Secret Management Tools
As it is evident, that AWS Secrets manager is still very lightweight and relies on lambda functions for key rotation functionality. AWS might be adding more functionalities to this in the future. HashiCorp Vault is way more feature-rich.
One of the other areas, where secrets management becomes important outside the application, is in managing access to Public Cloud environments. You may want to give temporary access based on lease time to some users or you may want to rotate/revoke keys for them. AWS Secrets engine doesn’t directly provide this ability, but you can use AWS IAM to do all of this functionality. You can use STS to perform similar actions.
HashiCorp Vault provides this ability with Dynamic Secrets. We will look at it in the next section.
Dynamic Secrets
Another ability of HashiCorp Vault is to generate dynamic secrets. These are different from kv secrets seen earlier, where the user needs to provide static passwords or keys.
Dynamic secrets, on the other hand, are not stored until the user or client initiates a read or GET call.
This feature can be pretty useful in generating temporary Access and secret keys for cloud access.
It can be achieved in a few steps.
1. Enable vault for AWS Secrets Engine
vault secrets enable -path=aws aws
2. Now go to AWS console to get an Access Key and Secret Access key that has privileged account credentials. Meaning, it allows for the creation of new IAM roles.
NOTE: Do NOT use Root account keys
Add the key to vault configuration.
vault write aws/config/myaccount access_key=<ADD ACCESS KEY HERE> secret_key=<ADD SECRET KEY HERE> region=us-east-2
3. Now, create an IAM policy that Vault should use to attach to a role that will be used to generate Temporary Access key and Secret Key.
In this example, I will use a policy that allows full access to S3 buckets.
vault write aws/roles/my-s3-role \
credential_type=iam_user \
policy_document=-<<EOF
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "Stmt12345678910",
"Effect": "Allow",
"Action": [
"S3:*"
],
"Resource": [
"*"
]
}
]
}
EOF
4. Now when generating the keys, Vault will attach the policy created in Step 3 to the new user.
vault read aws/creds/my-s3-role
OUTPUT:
Key Value
--- -----
lease_id aws/creds/my-s3-role/0bce0782-32aa-25ec-f61d-cd26ef241050
lease_duration 768h
lease_renewable true
access_key AKIAJELDTIANQGRQQQQ
secret_key WOeSnj01D+hHoHJMCR7ETNTCqZmKesTL1/0dDh
security_token <nil>
These secrets can be immediately revoked, if needed, by using the following command.
vault lease revoke aws/creds/my-s3-role/0bce0782-32aa-25ec-f61d-cd26ef241050
This method allows for access to credentials only on a need basis. Additionally, the lease time attached to it reduces the potential for keys being compromised and the need for key rotation.
This same method can be used for generating credentials across Azure and Google Cloud Platform. As a result of which, Vault can become your centralized tool for all things related to secrets.
Conclusion
There is a plethora of secret management tools out there. The best tool for your environment depends on the application and kind of infrastructure being managed. If you have a K8s based app, then Vault makes more sense. But if you have a simple 3-tiered application then AWS secrets manager can be used.
AWS Secrets Manager is very easy to set up and get started with. While Vault will require setting up the infrastructure and following a lot of best practices to get the deployment correct.
Did you know?
Git Secrets is a free and open-source tool that prevents you from committing AWS Keys, passwords and other secrets to git. Find out more here