sequenceDiagram
participant Store as Secret store
participant ExternalSecrets as External secrets
participant Kubernetes
participant Reloader
ExternalSecrets->>Store: Watch for changes
Reloader->>Kubernetes: Watch for changes
Store-->>ExternalSecrets: Notice secret change
ExternalSecrets->>Kubernetes: Deploy new secret
Kubernetes-->>Reloader: Notice secret change
Reloader->>Kubernetes: Trigger workload rollout
Kubernetes secret management
Márk Sági-Kazár
2023-06-30 @ Let’s Code meetup
whoami
Márk Sági-Kazár
Open Source Tech Lead @ Cisco
CNCF Ambassador
What is secret management?
Sensitive information, necessary for an application to function, is made available to it.
Why is it important?
- Maintain security of sensitive information
- Reduce the risk of security (and other) incidents
- Meet compliance requirements
Common requirements
- Encryption: Secrets should be encrypted both at rest and in transit.
- Rotation: The ability to rotate secrets regularly.
- Access control: Only authorized users/applications can access the secrets.
- Auditability: Track who is accessing which secrets and when.
- Compliance: Adhere to regulatory requirements related to data security.
Challenges
- No one-size-fits-all solution
- Secret management infra can easily become a SPOF
- Solutions often become complex, increasing the chance of introducing a vulnerability
Kubernetes secrets
- Standard Kubernetes resource
- Values are base64 encoded
⚠️ Plug the holes! ⚠️
- Turn on encryption at rest
- Configure least-privilege access to Secrets
Official guide: Good practices for Kubernetes Secrets
Deploying secrets to Kubernetes
External Secrets Operator
- https://external-secrets.io
- Synchronize secrets from an external store to Kubernetes
- Mount secrets as usual (env var, file)
Alternatives
Triggering workload rollout
- Reloader: https://github.com/stakater/Reloader
- Detects secret changes
- Triggers rollout for workloads referencing changed secrets
What could possibly go wrong?
Who knows, so monitor everything
Warning
Potential high cardinality labels (drop metrics/labels you don’t need)
Changes take effect with a delay
- Change some configuration ✏️
- Wait until the next secret sync period 🤞
- Hope nothing breaks 🙏
Solution: create (and modify) test secrets at the same time.
To sum up ESO
- Understand how (and when) changes will take effect
- Monitor and alert for failures
Kubernetes without secrets 😱
Access secret store directly
- Integrated into the application
OR
- “Inject” secrets into the application
Secret injection in Kubernetes
- Inject a custom init into Pods using a mutating admission webhook
- Get secrets from secret store in the custom init
- Inject secrets as environment variables
Bank-Vaults
- Started at Banzai Cloud
- Vault Swiss Army knife
Bank-Vaults secret injection
- Secret references:
vault:path/to/secret#KEY - Mutating webhook
- Detect secret references
- Mutate Pods
- Custom init replaces secret references with actual values
Warning
Secret changes do not take effect (ie. trigger workload reload) at the moment.
Risks and mitigations
Risk: Secret store is a SPOF
Mitigation: Maintain a cluster-local instance
Risk: Webhook is a SPOF
Mitigation: Configure webhook according to best practices
Alternatives
Demo
https://github.com/sagikazarmark/demo-oss-na-2023-kube-secret-rotation
https://github.com/sagikazarmark/demo-bank-vaults-secret-injection
Final thoughts
It seems wisest to assume the worst from the beginning…and let anything better come as a surprise.
Thank you
Any questions?