Secrets Management: Storage, Rotation & Access Control

Secrets Management

Why Engineers Care About This

Secrets (API keys, passwords, certificates) are sensitive data that must be protected. Leaked secrets enable attackers to access systems, steal data, or cause damage. But secrets are needed by applications, making them hard to manage. Understanding secrets management helps you protect sensitive data while enabling applications to access it securely.

When secrets are hardcoded in code, or committed to version control, or logged in plaintext, you're hitting secrets management problems. These problems compound. Leaked secrets enable attacks that are hard to detect and stop. Without proper secrets management, secrets are exposed through multiple vectors (code, logs, environment variables). Good secrets management solves these problems by securing secrets throughout their lifecycle.

In interviews, when someone asks "How would you handle API keys and passwords?", they're really asking: "Do you understand secrets management? Do you know how to store and access secrets securely? Do you understand secret rotation and access control?" Most engineers don't. They hardcode secrets in code, or store them in environment variables, or don't rotate them.

Core Intuitions You Must Build

• Secrets should never be in code or version control. Secrets hardcoded in code or committed to version control are exposed to anyone with code access. Even if you remove secrets later, they remain in Git history. Use secrets management tools (AWS Secrets Manager, HashiCorp Vault) to store secrets, and inject them at runtime. Don't hardcode secrets—they will be leaked.

• Secrets management tools provide secure storage and access. Secrets management tools (AWS Secrets Manager, HashiCorp Vault, Azure Key Vault) store secrets encrypted, provide access control (IAM, RBAC), and enable rotation. Use dedicated tools for secrets—don't store secrets in databases, files, or environment variables. Also, use tools that provide audit logs (who accessed what secrets when).

• Secret rotation prevents long-term exposure. Secrets should be rotated regularly (every 90 days for passwords, every year for certificates). Rotated secrets limit exposure if leaked—old secrets become invalid. Automate secret rotation—manual rotation is error-prone and often forgotten. Also, rotation should be transparent to applications (no downtime during rotation).

• Access control restricts who can access secrets. Not everyone needs access to all secrets. Use IAM policies, RBAC, or secret-specific access controls to restrict secret access. Principle of least privilege—grant minimum access needed. Also, audit secret access (who accessed what secrets when) to detect unauthorized access. Don't grant broad secret access—restrict to what's needed.

• Secrets should be encrypted at rest and in transit. Secrets should be encrypted when stored (at rest) and when transmitted (in transit). Secrets management tools encrypt secrets at rest. Use TLS for secret retrieval (encryption in transit). Don't store or transmit secrets in plaintext—they can be intercepted or accessed.

• Secret leaks must be detected and responded to quickly. Secret leaks happen (committed to Git, logged, exposed in errors). Detect leaks (scan code, logs, error messages) and respond quickly (rotate secrets, revoke access, investigate). Also, monitor secret access for unusual patterns (unusual access times, locations, frequencies) that might indicate compromise.

Subtopics (Taught Through Real Scenarios)

Secret Storage and Access

What people usually get wrong:

Engineers often store secrets in environment variables or configuration files. But environment variables can be logged, exposed in errors, or accessed by anyone with server access. Configuration files can be committed to version control or accessed by unauthorized users. Use dedicated secrets management tools (AWS Secrets Manager, HashiCorp Vault) that provide secure storage, encryption, and access control.

How this breaks systems in the real world:

A service stored API keys in environment variables. When the service logged errors, environment variables were included in logs (accidental logging). Logs were accessible to many team members, exposing API keys. An attacker accessed logs and used API keys to access external services. The fix? Use secrets management tool (AWS Secrets Manager)—secrets stored encrypted, accessed via API, and not exposed in logs. Now secrets are secure. But the real lesson is: environment variables aren't secure. Use dedicated secrets management tools.

What interviewers are really listening for:

They want to hear you talk about secret storage, dedicated tools, and why environment variables aren't secure. Junior engineers say "just use environment variables." Senior engineers say "use dedicated secrets management tools (AWS Secrets Manager, HashiCorp Vault) for secure storage, encryption, and access control—environment variables can be logged or accessed by unauthorized users." They're testing whether you understand that secrets require dedicated management, not just "storing values."

Secret Rotation

What people usually get wrong:

Engineers often don't rotate secrets, or rotate them manually. But secrets should be rotated regularly to limit exposure if leaked. Manual rotation is error-prone and often forgotten. Automate secret rotation—secrets management tools can rotate secrets automatically and update applications transparently. Don't rotate secrets manually—automate it.

How this breaks systems in the real world:

A service used API keys that were never rotated. When an API key was leaked (committed to Git, exposed in error message), it remained valid for years. Attackers used the leaked key to access external services, causing data breaches. The fix? Implement automatic secret rotation (rotate every 90 days)—old secrets become invalid, limiting exposure. Now leaked secrets expire quickly. But the real lesson is: secret rotation limits exposure. Automate rotation.

What interviewers are really listening for:

They want to hear you talk about secret rotation, automation, and exposure limitation. Junior engineers say "just rotate secrets when you remember." Senior engineers say "rotate secrets regularly (every 90 days) and automate rotation—manual rotation is error-prone, and automated rotation limits exposure if secrets are leaked." They're testing whether you understand that secret rotation is about limiting exposure, not just "changing passwords."

Preventing Secret Leaks

What people usually get wrong:

Engineers often don't think about secret leaks until they happen. But secret leaks are common (committed to Git, logged, exposed in errors). Prevent leaks—don't commit secrets to version control (use .gitignore, pre-commit hooks), don't log secrets (sanitize logs), and don't expose secrets in errors (sanitize error messages). Also, scan code and logs for secrets to detect leaks.

How this breaks systems in the real world:

A developer committed API keys to a public GitHub repository. The keys were exposed publicly. An attacker found the keys and used them to access external services, causing data breaches and financial losses. The fix? Use pre-commit hooks to detect secrets, scan repositories for secrets, and rotate exposed secrets immediately. But the real lesson is: secret leaks are common. Prevent them proactively.

What interviewers are really listening for:

They want to hear you talk about preventing secret leaks, scanning, and response. Junior engineers say "just don't commit secrets." Senior engineers say "prevent secret leaks—use .gitignore and pre-commit hooks, sanitize logs and error messages, scan code and logs for secrets, and rotate exposed secrets immediately." They're testing whether you understand that secret management is about prevention, not just "storing securely."

---

• Secrets should never be in code or version control—use secrets management tools
• Secrets management tools provide secure storage and access—encryption, access control, audit logs
• Secret rotation prevents long-term exposure—automate rotation regularly
• Access control restricts who can access secrets—principle of least privilege
• Secrets should be encrypted at rest and in transit—use TLS for retrieval
• Secret leaks must be detected and responded to quickly—scan code and logs, rotate exposed secrets
• Good secrets management protects sensitive data while enabling secure access

---

• Authentication & Authorization - Managing authentication secrets
• CI/CD Pipelines - Handling secrets in pipelines
• Infrastructure as Code - Managing secrets in infrastructure