Automate secret rotation in Kubernetes, then get out of the way!
Márk Sági-Kazár
2023-05-12 @ Open Source Summit NA 2023
whoami
Márk Sági-Kazár
Open Source Tech Lead @ Cisco
CNCF Ambassador
Once upon a time…
Your secrets WILL be compromised…
When?
What are you going to do about it?
Why is secret rotation important?
- Maintain security of sensitive information
- Meet compliance requirements
- Reduce the risk of a data breach
Challenges of secret rotation
- Complexity
- Time-consuming and error prone process
- Disruption of service availability
Secret rotation should be…
- possible
- automated
- periodic
Secret rotation flow
Secret rotation in Kubernetes
⚠️ Plug the holes first! ⚠️
- Turn on encryption at rest
- Configure least-privilege access to Secrets
Official guide: Good practices for Kubernetes Secrets
Deploying secrets to Kubernetes
- External Secrets Operator (ESO): https://external-secrets.io
- Synchronize secrets from an external store to Kubernetes
- Mount secrets as usual (env var, file)
Alternatives
Triggering workload rollout
- Reloader: https://github.com/stakater/Reloader
- Detects secret changes
- Triggers rollout for workloads referencing changed secrets
What could possibly go wrong?
Who knows, so monitor everything
Warning
Potential high cardinality labels (drop metrics/labels you don’t need)
Changes take effect with a delay
- Change some configuration ✏️
- Wait until the next secret sync period 🤞
- Hope nothing breaks 🙏
Solution: create (and modify) test secrets at the same time.
Cascading effect of an outage 1
Requirement: Use store validation.
- Provider goes down for a long time (ie. hours) ❌
- Store validation reaches a backoff of hours ⏳
- Secret synchronization essentially stops 😱
Solution: Bump every (Cluster)SecretStore after an outage.
To sum up ESO
- Understand how (and when) changes will take effect
- Monitor and alert for failures
Kubernetes without secrets 😱
Access secret store directly
- Integrated into the application
OR
- “Inject” secrets into the application
Secret injection in Kubernetes
- Inject a custom init into Pods using a mutating admission webhook
- Get secrets from secret store in the custom init
- Inject secrets as environment variables
Bank-Vaults
- Started at Banzai Cloud
- Vault Swiss Army knife
Bank-Vaults secret injection
- Secret references:
vault:path/to/secret#KEY - Mutating webhook
- Detect secret references
- Mutate Pods
- Custom init replaces secret references with actual values
Warning
Secret changes do not take effect (ie. trigger workload reload) at the moment.
Risks and mitigations
Risk: Secret store is a SPOF
Mitigation: Maintain a cluster-local instance
Risk: Webhook is a SPOF
Mitigation: Configure webhook according to best practices
Alternatives
Bank-Vaults Roadmap
- Moving to a new GitHub organization
- Workload reload on secret change
- Support for more providers
- Secret synchronization between providers
- Your desired feature (submit a new feature request)
Demo
https://github.com/sagikazarmark/demo-oss-na-2023-kube-secret-rotation
Final thoughts
It seems wisest to assume the worst from the beginning…and let anything better come as a surprise.
Thank you
Any questions?
Automate secret rotation in Kubernetes, then get out of the way! Márk Sági-Kazár 2023-05-12 @ Open Source Summit NA 2023