Have you ever tried creating an RDS cluster/instance using terraform?

If yes, then I’m sure the question to manage RDS credentials easily and securely has bogged your mind.

If not, well hang tight I’ve got a simple two-step process to improve the security of your infrastructure in terms of managing RDS credentials and other secrets.

A simple RDS cluster can be created using the following block

resource "aws_rds_cluster" "postgresql" {
  cluster_identifier      = "aurora-cluster-demo"
  engine                  = "aurora-postgresql"
  availability_zones      = ["us-west-2a", "us-west-2b", "us-west-2c"]
  database_name           = "mydb"

  # How should we manage the username and the password? 
  master_username         = "???"
  master_password         = "???"
}

The username and password are required arguments, and can’t be omitted. So, in this blog post, first we will start with briefly comparing different ways to store RDS credentials. Later we will be using AWS Secret manager to store RDS credentials, and will try to automate the entire process through terraform.

Different Ways to Manage RDS Secret

Method 0: Storing secrets in plain text.

Sample code:

resource "aws_rds_cluster" "postgresql" {
  cluster_identifier      = "aurora-cluster-demo"
  engine                  = "aurora-postgresql"
  availability_zones      = ["us-west-2a", "us-west-2b", "us-west-2c"]
  database_name           = "mydb"

  master_username         = "foo"
  master_password         = "bar"
}

Pros:

Can be done, if a single person is managing the infra or the code is not checked in any version control system.

Cons:

• Storing secrets in plain text is a bad idea because

  • Anyone having access to the checked-in repository will have the access to the database no matter what.

  • There is no way to control, revoke the access.

Method 1: Storing secrets in environment variables.

Sample code:

variable "username" {
  type        = string
}
variable "password" {
  type        = string
}

resource "aws_rds_cluster" "postgresql" {
  cluster_identifier      = "aurora-cluster-demo"
  engine                  = "aurora-postgresql"
  availability_zones      = ["us-west-2a", "us-west-2b", "us-west-2c"]
  database_name           = "mydb"

  master_username         = var.username
  master_password         = var.password
}

# Set username and password as environment variables
export TF_VAR_username=foo
export TF_VAR_password=bar

Pros:

Safer than storing secrets in plain text.

Cons:

This technique helps avoid storing secrets in plain text. But doesn’t provide any managing or rotating mechanism. For that, we need to rely on other third-party password management tools such as 1Password, or LastPass. Which adds another overhead to manage them.

Also, everyone working on the code has to take a few extra steps to either manually set these environment variables or run a wrapper script, to fetch values from the third-party password manager.

Method 2: Storing in Encrypted files.

In this technique, the credentials are encrypted, and the ciphertext is checked in the version control system. The problem here is for encrypting the credentials we need some key. The key is another secret, which needs to be protected. Now we are dealing with two secrets, instead of one.

The secret used for encrypting the credentials can be stored in AWS KMS, GCP KMS, etc.

Pros:

• No secrets in plain text.

• Secrets are encrypted and checked in the version control system.

• Everything can be done through terraform code from encrypting credentials to using it while creating the RDS cluster.

Cons:

• Secrets are encrypted, but the problem of rotating and revoking the access of secrets is still not solved.

• Also, AWS KMS is not free. For more on pricing please refer here.

Method 3: Storing secrets in AWS Secret Manager

In this method, the secrets are stored in a dedicated secret manager specially designed for managing secrets.

Storing and Managing secrets from the AWS console is pretty straightforward, and a bunch of blogs and articles can be found on the internet. Refer this doc for more information. Manage RDS Secrets through AWS Console

Here, we will go one step ahead and try to automate the entire process as much as possible. As discussed earlier, storing and managing secrets through terraform is a two-step process. In the first step, we will be creating an AWS secret for storing the credentials, and a rotation lambda function for rotating the password. Later in the second step will enable the RDS to read credentials from the secret manager instead of reading it from the tfvars file.

To create the secret, use the following snippet;

resource "aws_secretsmanager_secret" "secret" {
  description         = "Secrets for ${var.rds_name}"
  name                = "${var.rds_name}-postgres-secret"
  rotation_lambda_arn = aws_lambda_function.rotate_code_postgres.arn
  rotation_rules {
  // RDS password will be rotated after 30 days automatically.  
    automatically_after_days = "30"
  }
}

resource "aws_secretsmanager_secret_version" "secret" {
  lifecycle {
    ignore_changes = [
      secret_string
    ]
  }
  secret_id     = aws_secretsmanager_secret.secret.id
  secret_string = <<EOF
{
  "username": "${var.rds_master_username}",
  "password": "${var.rds_master_password}",
  "engine": "postgres",
  "host": "${var.rds_host}",
  "port": 5432,
  "dbClusterIdentifier": "${var.rds_cluster_identifier}",
  "db" : "${

This creates a secret, storing the following variables

• username

• password

• engine

• host

• port

• dbClusterIdentifier

• db

Note: Only the password will be rotated by the lambda function and variables like host, port, etc play a crucial role in rotation thus needs to be stored in the secret itself.

Once the secret is created let’s proceed further by creating a lambda function and attaching it to the same VPC, subnet as of the RDS instance/ cluster. Also, the lambda function needs to be given access to read the created RDS secret from the secret manager.

To create the rotation lambda function, use the following snippet.

resource "aws_lambda_function" "rotate_code_postgres" {
  filename         = "${path.module}/rotate.zip"
  function_name    = "rds-rotation-lambda"
  role             = aws_iam_role.lambda_rotation.arn
  handler          = "lambda_function.lambda_handler"
  source_code_hash = filebase64sha256("${path.module}/rotate.zip")
  runtime          = "python3.7"
  vpc_config {
    subnet_ids         = var.subnet_ids
    security_group_ids = [var.rds_sg]
  }
  timeout     = 30
  description = "Conducts an AWS SecretsManager secret rotation for RDS using single user rotation scheme"
  environment {
    variables = {
      SECRETS_MANAGER_ENDPOINT = "https://secretsmanager.${data.aws_region.current.name}.amazonaws.com"
    }
  }
}

For giving the appropriate permission to the lambda function. Please refer this link.

Once RDS secret, the rotation lambda function is created, and the required permissions are given, we can verify the created secret from the AWS Console. See the image below for reference.

Let’s proceed further with rotating the secret from the AWS Console. By Clicking the Rotate secret immediately button. This will rotate the password for the RDS. Else, it will be automatically rotated after 30 days as defined in the rotation rule.

Once, done let’s enable the RDS to read credentials from the secret manager.

variable "enable_rds_secret_rotation" {
  type = bool
}

data "aws_secretsmanager_secret" "by_name" {
  count = var.enable_rds_secrets_rotation ? 1 : 0
  name  = "${var.db_name}-postgres-secret"
}

data "aws_secretsmanager_secret_version" "creds" {
  count     = var.enable_rds_secrets_rotation ? 1 : 0
  secret_id = try(data.aws_secretsmanager_secret.by_name[0].id, "")
}

locals {
  username = try(jsondecode(data.aws_secretsmanager_secret_version.creds[0].secret_string)["username"], var.master_username)
  password = try(jsondecode(data.aws_secretsmanager_secret_version.creds[0].secret_string)["password"], var.master_password)
}

resource "aws_db_instance" "postgres" {
  instance_class          = var.db_instance
  engine                  = var.db_engine
  engine_version          = var.db_engine_version
  multi_az                = var.enable_multi_az
  storage_type            = var.db_storage_type
  allocated_storage       = var.db_allocated_storage
  name                    = var.db_name
  username                = local.username
  password                = local.password
  backup_window           = var.db_backup_window
  backup_retention_period = var.db_backup_retention_period
  db_subnet_group_name    = aws_db_subnet_group.db_subnet.name
  vpc_security_group_ids  = [aws_security_group.rds_sg.id]
  skip_final_snapshot     = var.enable_skip_final_snapshot
  publicly_accessible     = var.enable_public_access
}

Key Points:

• Lambda function needs to be given access to read the RDS secret.

• Lambda function must be attached to the same VPC subnet as that of the RDS cluster.

• Once the password is rotated, the backend code needs to be updated accordingly to read values from the secret manager instead of hard-coding the value or reading it from the env variable.

• For the first iteration enable_rds_secret_rotation variable should be false, after successfully creating the secret, modify it to true and run terraform apply.

Pros:

• Secrets are stored in a dedicated secret store that enforces encryption and strict access control.

• It supports secrets rotation, which is useful if ever our secret got compromised. Also, automatically rotates secret periodically.

• AWS secret manager supports audit logs to see who accessed the sercret & when it was accessed.

• Once the secret is created, AWS secret manager generates code snippets that show you exactly how to read your secrets from apps written in Java, Python, JavaScript, Ruby, Go, etc.

Cons:

• No version control system for secrets.

• AWS secret manager is not free. The cost is $0.40 per secret per month. For secrets that are stored for less than a month, the price is prorated (based on the number of hours.) For API calls $0.05 per 10,000 API calls.

Conclusion

• RDS credentials can be managed by any of the above discussed methods based on your requirement and budget.

• No matter which technique you are using to manage the secrets in the end everything is stored in plain text in the terraform state file. So the state file has to be stored securely.

Some Useful resources

• Github Link

• AWS Secret Manager Doc