Using AWS Secrets Manager and Kubernetes Secrets Store CSI Driver for Secure Secret Management in EKS
AWS Secrets Manager is used to securely store and retrieve sensitive information for the application running on an EKS cluster. Here's how it works step-by-step:
AWS Secrets Manager Storage:
Secrets, such as database credentials, API keys, and environment-specific variables, are stored securely in AWS Secrets Manager.
In the
SecretProviderClass
section, theobjects
parameter specifies theobjectName
(secret in Secrets Manager) and how the secret's keys are mapped (jmesPath
). For example,objectName: applicationname
refers to a secret stored in Secrets Manager, and the nestedjmesPath
maps the paths within that secret to alias names used in the Kubernetes environment.
CSI Driver for Secrets Store:
The Kubernetes deployment uses the Secrets Store CSI Driver, which enables the Kubernetes pod to directly fetch secrets from AWS Secrets Manager.
The
volumes
section in theDeployment
uses the CSI driver (secrets-store.csi.k8s.io
) to mount the secrets as a volume. TheSecretProviderClass
specifies how the secrets should be fetched from AWS Secrets Manager.
Secrets Injection into the Pod:
When the pod is created, the Secrets Store CSI Driver connects to AWS Secrets Manager using the provided IAM role (
eks.amazonaws.com/role-arn
in theServiceAccount
).The secrets defined in the
SecretProviderClass
(applicationname-secrets
) are fetched and made available in the pod. They are either mounted as files at the specifiedmountPath
(/mnt/secrets
) or directly injected into environment variables.
Environment Variables from Secrets:
In the
Deployment
YAML, the environment variables for theapplicationname
container are defined using thevalueFrom
attribute. This references the secrets provided by the CSI driver.Each environment variable references a key from the
applicationname-secrets
that the CSI driver fetched from AWS Secrets Manager.For example:
- name: DATABASE_HOST valueFrom: secretKeyRef: name: applicationname-secrets key: DATABASE_HOST
This means that the environment variable
DATABASE_HOST
is populated with the value ofDATABASE_HOST
key stored in theapplicationname-secrets
.
IAM Role for Secret Access:
The
ServiceAccount
namedapplicationname
has an annotation (eks.amazonaws.com/role-arn
) pointing to an IAM role (arn:aws:iam::xxxxxxxxxxx:role/applicationname-secrets
).This IAM role must have permissions to access the specified secrets in AWS Secrets Manager. When the pod is started, it assumes this IAM role to access the secrets.
Auto rotation of mounted contents and synced Kubernetes Secrets
When the secret/key is updated in external secrets store after the initial pod deployment, the updated secret will be periodically updated in the pod mount and the Kubernetes Secret.
Depending on how the application consumes the secret data:
Mount Kubernetes secret as a volume: Use auto rotation feature + Sync K8s secrets feature in Secrets Store CSI Driver, application will need to watch for changes from the mounted Kubernetes Secret volume. When the Kubernetes Secret is updated by the CSI Driver, the corresponding volume contents are automatically updated.
Application reads the data from container’s filesystem: Use rotation feature in Secrets Store CSI Driver, application will need to watch for the file change from the volume mounted by the CSI driver.
Using Kubernetes secret for environment variable: The pod needs to be restarted to get the latest secret as environment variable.
Use something like Reloader to watch for changes on the synced Kubernetes secret and do rolling upgrades on pods
Enable auto rotation
NOTE: This alpha feature is not enabled by default.
To enable auto rotation, enable the --enable-secret-rotation
feature gate for the secrets-store
container in the Secrets Store CSI Driver pods. The rotation poll interval can be configured using --rotation-poll-interval
. The default rotation poll interval is 2m
. If using helm to install the driver, set enableSecretRotation: true
and configure the rotation poll interval by setting rotationPollInterval
. The rotation poll interval can be tuned based on how frequently the mounted contents for all pods and Kubernetes secrets need to be resynced to the latest.
The Secrets Store CSI Driver will update the pod mount and the Kubernetes Secret defined in
secretObjects
of SecretProviderClass periodically based on the rotation poll interval to the latest value.If the
SecretProviderClass
is updated after the pod was initially createdAdding/deleting objects and updating keys in existing
secretObjects
- the pod mount and Kubernetes secret will be updated with the new objects added to theSecretProviderClass
.Adding new
secretObj
ect
to the existingsecretObjects
- the Kubernetes secret will be created by the controller.
resource "helm_release" "secrets-csi-driver" {
name = "secrets-store-csi-driver"
repository = "https://kubernetes-sigs.github.io/secrets-store-csi-driver/charts"
chart = "secrets-store-csi-driver"
namespace = "kube-system"
version = "1.4.3"
set {
name = "syncSecret.enabled"
value = true
}
set {
name = "enableSecretRotation"
value = true
}
set {
name = "rotationPollInterval"
value = "60s"
}
}
Reloader
Problem
We would like to watch if some change happens in ConfigMap
and/or Secret
; then perform a rolling upgrade on relevant DeploymentConfig
, Deployment
, Daemonset
, Statefulset
and Rollout
Solution
Reloader can watch changes in ConfigMap
and Secret
and do rolling upgrades on Pods with their associated DeploymentConfigs
, Deployments
, DaemonsetsStatefulsets
and Rollouts
.
secret
To perform rolling upgrade when change happens only on specific secrets use below annotation.
For a Deployment
called foo
have a Secret
called foo-secret
. Then add this annotation to main metadata of your Deployment
kind: Deployment
metadata:
annotations:
secret.reloader.stakater.com/reload: "foo-secret"
spec:
template:
metadata:
Use comma separated list to define multiple secrets.
kind: Deployment
metadata:
annotations:
secret.reloader.stakater.com/reload: "foo-secret,bar-secret,baz-secret"
spec:
template:
metadata:
Deploying Reloader to Kubernetes
resource "helm_release" "reloader" {
name = "reloader"
repository = "https://stakater.github.io/stakater-charts"
chart = "reloader"
namespace = "kube-system"
version = "1.1.0"
set {
name = "reloader.watchGlobally"
value = "true"
}
}
Refrenece :
https://docs.aws.amazon.com/secretsmanager/latest/userguide/integrating_csi_driver.html
https://secrets-store-csi-driver.sigs.k8s.io/topics/secret-auto-rotation.html
https://github.com/stakater/Reloader?tab=readme-ov-file
Stay Tuned!
Be sure to follow and subscribe for more updates and upcoming blogs.