sequenceDiagram
actor Operator
participant Provider as Secret provider
participant Store as Secret store
participant Deploy as ???
participant Production
Deploy->>Store: Watch for changes
activate Deploy
Operator->>Provider: Generate new secret
Provider-->>Operator: Return new secret
Operator->>Store: Rotate secret in store
Store-->>Deploy: Notice secret change
deactivate Deploy
Deploy->>Production: Deploy new secret
Automating secret rotation in Kubernetes
Minimizing mistakes by removing the human element
Márk Sági-Kazár
2023-02-04 @ FOSDEM ’23
whoami
Márk Sági-Kazár
Engineering Technical Lead @ Cisco
Help engineering teams run their business on Kubernetes
Once upon a time..
Why is secret rotation important?
- Maintain security of sensitive information
- Meet compliance requirements
- Reduce the risk of a data breach
Challenges of secret rotation
- Complexity
- Time-consuming and error prone process
- Disruption of service availability
Secret rotation should be…
- possible
- automated
- periodic
Secret rotation flow
Secret rotation in Kubernetes
Deploying secrets to Kubernetes
- External Secrets: https://external-secrets.io
- Synchronize secrets from an external store to Kubernetes secrets
- Mount secrets as usual (env var, file)
Important
Turn on envelope encryption!!!
Triggering workload rollout
- Reloader: https://github.com/stakater/Reloader
- Detects secret changes
- Triggers rollout for workloads referencing changed secrets
sequenceDiagram
participant Store as Secret store
participant ExternalSecrets as External secrets
participant Kubernetes
participant Reloader
ExternalSecrets->>Store: Watch for changes
Reloader->>Kubernetes: Watch for changes
Store-->>ExternalSecrets: Notice secret change
ExternalSecrets->>Kubernetes: Deploy new secret
Kubernetes-->>Reloader: Notice secret change
Reloader->>Kubernetes: Trigger workload rollout
Demo
https://github.com/sagikazarmark/demo-fosdem23-kube-secret-rotation
Thank you
Any questions?