AWS Secrets Manager
AWS Secrets Manager keeps secrets such as database passwords, API keys and OAuth tokens. It encrypts each one with an AWS KMS key and returns it only to callers that AWS IAM allows to read it. It can also rotate a secret on a schedule by invoking a Lambda function, so a database password changes every 30 days without anyone touching config or a .tfvars file.
How it works
A secret is a named value, usually a JSON string, with versions. Staging labels say which version is which: AWSCURRENT is what callers get by default, AWSPREVIOUS is the one before it, and AWSPENDING holds a new value while rotation runs. Applications call GetSecretValue, and CloudTrail records each call, so you can see which role read which secret and when.
Rotation is a Lambda function that Secrets Manager calls four times. createSecret generates a new password and stores it as AWSPENDING. setSecret changes the password in the database, testSecret logs in with it, and finishSecret moves the AWSCURRENT label to the new version.
For Amazon RDS the function has to reach the database port, so it runs in the same VPC subnets. The secret also keeps host, port and engine next to the password, because the function needs them to connect. RDS can also manage the master password in Secrets Manager itself, with no rotation function for you to maintain.
For example, 8 services in 5 environments need 40 secrets. At $0.40 per secret per month, that is $16. API calls cost $0.05 per 10,000. If 60 pods each call GetSecretValue once a minute, that is 2,592,000 calls in a 30-day month, or $12.96. Cache the value for an hour and it drops to 43,200 calls, about $0.22.
The cache has a cost, though: with single-user rotation the old password stops working the moment setSecret runs, and a pod holding a cached copy fails its next login. Refetch the secret when the database rejects the password.
Why it matters in production
Secrets Manager takes the password out of code and environment variables. It does not take it out of Terraform state. If Terraform writes the first value, as in the One2N RDS post, that value sits in plain text in the state file, so state needs encryption and tight access too.
Access is also the limit on reuse: when One2N looked at it for Kafka users across 17 environments, giving many application teams IAM access to it was too much risk and overhead. On Kubernetes, the External Secrets Operator copies values from Secrets Manager into Kubernetes Secrets so pods need no AWS SDK.
See also
On the One2N blog
References
