The secret was born when a company started; it’s still in production. Rotating it would break things. So nobody rotates. This is how breaches happen. This post is the working playbook.
The dual-secret pattern
At the heart of safe rotation:
Time 0: secret-v1 in use
Time T: deploy secret-v2; both v1 and v2 valid
Time T+window: v1 revoked; only v2 valid
The application accepts both during the overlap. Clients update at their own pace. After the window: old is dead.
def verify_token(token: str) -> bool:
return any(
hmac.compare_digest(token, key)
for key in (CURRENT_KEY, PREVIOUS_KEY)
)
PREVIOUS_KEY is non-empty during overlap; revoked after.
AWS Secrets Manager rotation
# Lambda-backed rotation
def lambda_handler(event, context):
secret_id = event["SecretId"]
step = event["Step"]
if step == "createSecret":
# Generate new password; store as AWSPENDING version
new_pw = secrets.token_urlsafe(32)
client.put_secret_value(
SecretId=secret_id,
ClientRequestToken=event["ClientRequestToken"],
VersionStages=["AWSPENDING"],
SecretString=json.dumps({"password": new_pw}),
)
elif step == "setSecret":
# Apply to RDS / wherever
...
elif step == "testSecret":
# Verify the new secret works
...
elif step == "finishSecret":
# Promote AWSPENDING to AWSCURRENT
client.update_secret_version_stage(...)
Schedule rotation; AWS Lambda runs your script. RDS, DocumentDB, Aurora support managed rotation. For other systems, write the rotation Lambda.
For secrets management broadly .
Application-side patterns
For every secret your app uses:
- Read at startup + periodic refresh (every 5–15 min).
- Accept v1 OR v2 during overlap.
- Log which version is in use (for debugging).
- Alert on staleness (haven’t refreshed in N hours).
class SecretCache:
def __init__(self, secret_id):
self.secret_id = secret_id
self.values = [] # most-recent first
self.refreshed_at = 0
async def get(self) -> list[str]:
if time.time() - self.refreshed_at > 300:
await self.refresh()
return self.values
async def refresh(self):
current = await secrets_manager.get_value(self.secret_id, "AWSCURRENT")
previous = await secrets_manager.get_value(self.secret_id, "AWSPREVIOUS")
self.values = [current, previous]
self.refreshed_at = time.time()
Application uses both to verify; rotation just needs to update the secret manager.
Per-secret-type cadence
| Cadence | |
|---|---|
| AWS access keys | 90 days (or use IRSA) |
| DB passwords | 90 days |
| TLS certs | LE: 60–90 days auto |
| API keys (internal) | 30 days |
| API keys (customer-facing) | Customer-controlled |
| OAuth tokens | Hours |
| JWT signing keys | 30–90 days |
| Webhook secrets | Per customer or quarterly |
Document. Automate. Don’t rely on humans remembering.
JWT signing key rotation
Common pain. Pattern:
# JWT issuance signs with current
def sign_jwt(payload):
return jwt.encode(payload, CURRENT_KEY, algorithm="ES256", headers={"kid": "v2"})
# Verification accepts multiple keys keyed by kid
def verify_jwt(token):
header = jwt.get_unverified_header(token)
kid = header.get("kid")
key = KEY_REGISTRY.get(kid)
if not key: raise InvalidToken
return jwt.decode(token, key, algorithms=["ES256"])
KEY_REGISTRY includes current + recent retired. Old tokens still verify; new tokens use new key.
Database password rotation
1. Create new user with new password.
2. Grant same permissions.
3. Update app config to use new user.
4. Wait for app to roll over.
5. Drop old user.
Or for RDS managed: AWS handles the rotation; app just refreshes from Secrets Manager.
Webhook secret rotation
Customers expect rotation:
POST /webhook
X-Webhook-Signature-v1: <sig with old secret>
X-Webhook-Signature-v2: <sig with new secret>
During overlap, send BOTH signatures. Customer verifies either. After overlap: only v2.
For webhook design .
Detecting leaks
Tools:
- gitleaks / trufflehog in pre-commit and CI.
- GitHub secret scanning for known patterns.
- Cloud-vendor anomaly detection.
When detected: rotate immediately. Don’t debug; rotate first; investigate after.
Common mistakes
1. Rotation breaks customers
No overlap window; old keys cut off; integrations fail. Always overlap.
2. Rotation never happens
“Too risky.” Make it routine; prove it works.
3. Rotation manual
Humans forget. Automate with Secrets Manager + Lambda or Vault rotation.
4. No version tracking
App can’t tell which secret it’s using. Log it.
5. No leak detection
Secret in a Slack screenshot; nobody notices. Run scans.
What I’d ship today
- AWS Secrets Manager for all secrets.
- 30–90 day rotation scheduled via Lambda.
- App refresh from Secrets Manager every 5 min.
- Dual-secret pattern during rotation windows.
- gitleaks in CI + GitHub secret scanning.
- Quarterly review of rotation cadence per secret type.
Read this next
- Secrets Management in 2026
- Authentication in 2026 — Passkeys, OAuth, OIDC
- Webhook Design 2026
- Software Supply Chain Security
If you want my secret rotation runbook + Lambda, it’s at rajpoot.dev .
Building something AI-, backend-, or data-heavy and want a second pair of eyes? I do consulting and freelance work — see my projects and ways to reach me at rajpoot.dev .