Multi‑cluster, multi‑tenant secret management
Márk Sági-Kazár
2023-05-25 @ ESO meetup
whoami
Márk Sági-Kazár
Open Source Tech Lead @ Cisco
CNCF Ambassador
Cisco ET&I
- Incubating org for emerging technology
- Engineers generally don’t have production operational experience
- Infrastructure needs are served by a central platform engineering (SRE) team
Challenges
- Managing configuration across clusters, envs, tenants
- Access control and resource management
- Reducing the blast radius
- Secret management and rotation (!)
Deployment model
- GitOps
- ArgoCD
Secret management (initial approach)
- Central Hashicorp Vault (each team gets its own namespace)
- ESO to sync secrets into clusters
- Manual secret rotation :(
Challenges of secret rotation
- Complexity
- Time-consuming and error prone process
- Disruption of service availability
Secret rotation flow
Secret management in Kubernetes
⚠️ Plug the holes first! ⚠️
- Turn on encryption at rest
- Configure least-privilege access to Secrets
Official guide: Good practices for Kubernetes Secrets
Deploying secrets to Kubernetes
- External Secrets Operator (ESO): https://external-secrets.io
- Synchronize secrets from an external store to Kubernetes
- Mount secrets as usual (env var, file)
Triggering workload rollout
- Reloader: https://github.com/stakater/Reloader
- Detects secret changes
- Triggers rollout for workloads referencing changed secrets
What could possibly go wrong?
Who knows, so monitor everything
Warning
Potential high cardinality labels (drop metrics/labels you don’t need)
Changes take effect with a delay
- Change some configuration ✏️
- Wait until the next secret sync period 🤞
- Hope nothing breaks 🙏
Solution: create (and modify) test secrets at the same time.
Cascading effect of an outage 1
Requirement: Use store validation.
- Provider goes down for a long time (ie. hours) ❌
- Store validation reaches a backoff of hours ⏳
- Secret synchronization essentially stops 😱
Solution: Bump every (Cluster)SecretStore after an outage.
To sum up ESO
- Understand how (and when) changes will take effect
- Monitor and alert for failures
Demo
https://github.com/sagikazarmark/demo-oss-na-2023-kube-secret-rotation
Thank you
Any questions?
Multi‑cluster, multi‑tenant secret management Márk Sági-Kazár 2023-05-25 @ ESO meetup