Rotating tokens and keys
Overview
Below is the guidance for rotating the tokens and keys that are set as environment variables for your application container or your automated deployment pipeline. This guide covers only secrets provided by the Cloud Platform system (your application may have other secrets too)
ServiceAccount tokens
In Kubernetes versions < 1.24 deleting your Service Account secret using a kubectl command would signal to kubernetes to recreate the secret automatically. This feature has been removed in kubernetes version 1.24 and above see here for more details.
Using Cloud Platform Service Account Terraform module
Ensure you are on the latest version of the cloud-platform-terraform-serviceaccount module.
Update your Terraform code to include the serviceaccount_token_rotated_date variable and set it to the current date in the format dd-mm-yyyy, and raise a PR for this update. See the mode example code for more details.
Once applied Terraform will destroy and recreate your Service Account token. if you are pushing this token to GitHub no further action is required; if you are using this token statically in external services like CircleCI then you will need to update the relevant fields manually. See here for more details.
AWS credentials
In cases where you have a need for creating and storing AWS Access keys as Kubernetes Secrets, and wish for these to be rotated, please get in touch with the Cloud Platform team.
Certificate tls keys
Delete the secrets related to the certificates that are used to create the certificate resource. The secret name will be shown in your certificate manifest file.
$ kubectl get secrets -n <yournamespace>
c100-application-tls-long-domain kubernetes.io/tls 2 325d
c100-application-tls-short-domain kubernetes.io/tls 2 325d
circleci-token-msrhd kubernetes.io/service-account-token 3 17d
default-token-pztpp kubernetes.io/service-account-token 3 325d
$
$ kubectl delete secret c100-application-tls-long-domain -n <yournamespace>
After the secret is deleted the replacement secret is generated immediately. This will not change the duration of the certificate or the renew process.
You will receive below error message while the new secret is created, it might incur some service downtime (in tests, deleting a secret of a certificate resulted in approx. 7 seconds downtime)
SSL: no alternative certificate subject name matches target host name 'xxxxxxx.service.justice.gov.uk'
failed to verify the legitimacy of the server and therefore could not establish a secure connection to it.
AWS RDS DB password
Rotate RDS DB password using the terraform module. You will need to change your resources/rds.tf file and add a new variable "db_password_rotated_date" in the terraform code as below and update the "dd-mm-yyyy" value to the current date.
db_password_rotated_date = "dd-mm-yyyy"
Note: Make sure you are using the latest version of the RDS module.
Applying this will recreate the password for an RDS instance, and the database_password stored as Kubernetes secrets in your namespace will be updated with the new one.
You can verify the database password change by decoding its rds secret contents using the cloud platform cli:
cloud-platform decode-secret -n [your namespace] -s [rds secret name]
Here is a redacted example:
cloud-platform decode-secret -n dstest -s rds-instance-output
{
"apiVersion": "v1",
"data": {
"database_password": "new-password",
"database_username": "xxxxxxxxxx",
"url": "postgres://xxxxxxxxxx:new-password@cloud-platform-1111111111111111.cdwm328dlye6.eu-west-2.rds.amazonaws.com:5432/db2axxxxxxxxxxxxxx"
},
"kind": "Secret",
"metadata": {
"name": "rds-instance-output",
"namespace": "dstest",
},
"type": "Opaque"
}
You will need to delete all pods in the corresponding namespace to allow new pods to be created, which will pick up the new DB password from the kubernetes secret.
It is possible that applications might experience downtime if, for example, a pod which was launched with the old password drops a DB connection and tries to open a new one (which will fail, because the password is no longer valid). Similarly, a pod launched with the old password (e.g. a cron job), which then waits to open a DB connection will fail to connect if the password has been replaced since the pod was launched.
AWS Elasticache AUTH token
Important: You must be using Redis v5.0.5 or above to rotate your AUTH token. If you're not using Redis v5.0.5 or above, you must upgrade your Redis cluster before attempting to rotate your AUTH token.
Elasticache Redis server supports two AUTH tokens at a given time. Hence, rotating the current AUTH token involves multiple steps.
- Rotate Elasticache AUTH token using the terraform module
You will need to change your `resources/elasticache.tf` file and add a new variable
"auth_token_rotated_date" in the terraform code as below and update the "dd-mm-yyyy:hh:mm" value to the current date.
```
auth_token_rotated_date = "yyyy-mm-dd"
```
> **Note:** Make sure you are using the latest version of the [Elasticache module][elasticache-module-version]. You may also wish to check
whether your `kubernetes_secret` is already set to contain `access_key_id` and `secret_access_key` data, as shown in the example code [here](https://github.com/ministryofjustice/cloud-platform-terraform-elasticache-cluster/blob/main/examples/elasticache.tf).
This follow the **ROTATE** strategy and when applying this adds an additional AUTH token to the server and remove the the oldest AUTH token from the server.
Hence the previous token will be retained allowing a server to support up to two most recent AUTH tokens at a given time. This also updates the `auth_token` stored as a Kubernetes secret in your namespace.
You can fetch the latest auth token by decoding the contents of your kubernetes secret using the [cloud platform cli]:
```
cloud-platform decode-secret -n [your namespace] -s [elasticache-redis-secret-name]
```
Here is a redacted example:
```
cloud-platform decode-secret -n dstest -s elasticache-redis-output
{
"apiVersion": "v1",
"data": {
"access_key_id": "XXXXXXX",
"auth_token": "XXXXXXXXXXX",
"member_clusters": "[\"cp-1111111-111\",\"cp-222222222-222\"]",
"primary_endpoint_address": "master.cp-1111111.XXXX.XXX.cache.amazonaws.com"
"replication_group_id": "cp-1111111",
"secret_access_key": "XXXXXXX"
},
"kind": "Secret",
"metadata": {
"creationTimestamp": "2023-01-19T17:47:37Z",
"name": "elasticache-redis-output",
"namespace": "dstest",
},
"type": "Opaque"
}
```
2. Recycle the pods in your namespace
Elasticache support two AUTH tokens, you can delete your pods one by one to allow new pods to pick up the new AUTH token from the kubernetes secret.
While testing, there was no downtime experience as the previous AUTH token will still be supported.
3. Update the Redis server to support and use only the latest AUTH token
In the event of Security Incident when an AUTH token is exposed, you will have to use SET Strategy after rotating the secrets using Step 1.
This will update the server to support just a single AUTH token.
Terraform does not support **SET** strategy yet, see [open issue]. Hence you will have to use AWS CLI to update the elasticache.
To access to your Elasticache using AWS CLI, you will need the `access_key_id` and `secret_access_key` which are stored as a kubernetes secret explained in Step 1 and Set up an AWS profile using AWS CLI.
Decode `replication_group_id` and the latest `auth_token` from the kubernetes secret and run the below command
```
aws elasticache modify-replication-group \
--replication-group-id <replication_group_id from the secret> \
--auth-token <auth_token from the secret> \
--auth-token-update-strategy SET \
--apply-immediately
```
To read more on how to modify the AUTH token, see AWS documentation
Was this page useful?