1. Home
  2. Documentation
  3. Cloud Platform User Guide
  4. Migrating an Elasticsearch service

Migrating an Elasticsearch service

This guide covers one way of migrating an elasticsearch domain to another. The steps listed here are specific to migrating an elasticsearch domain to enable encryption of data at rest.

Step 1 - Configure s3 snapshot repository for backup and restore

Adding an s3 snapshot repository to the elasticsearch is a 2-step process.

  • Create an s3 bucket

    Use the below code block and update the values appropriately for your namespace. Raise a PR against the environments repository.


module "es_snapshots_s3_bucket" {
  source                 = "github.com/ministryofjustice/cloud-platform-terraform-s3-bucket?ref=4.5"
  team_name              = var.team_name
  acl                    = "private"
  versioning             = false
  business-unit          = var.business-unit
  application            = var.application
  is-production          = var.is-production
  environment-name       = var.environment-name
  infrastructure-support = var.infrastructure-support
  namespace              = var.namespace

  providers = {
    aws = aws.london
  }
}

resource "kubernetes_secret" "es_snapshots" {
  metadata {
    name      = "es-snapshot-bucket"
    namespace = var.namespace
  }

  data = {
    bucket_arn  = module.es_snapshots_s3_bucket.bucket_arn
    bucket_name = module.es_snapshots_s3_bucket.bucket_name
  }
}
  • Add s3 bucket arn to the source elasticsearch module as below:
module "source_es" {
  source                        = "github.com/ministryofjustice/cloud-platform-terraform-elasticsearch?ref=3.8.0"
  cluster_name                  = var.cluster_name
  cluster_state_bucket          = var.cluster_state_bucket
  application                   = var.application
  business-unit                 = var.business-unit
  environment-name              = var.environment-name
  infrastructure-support        = var.infrastructure-support
  is-production                 = var.is-production
  team_name                     = var.team_name
  elasticsearch-domain          = "source-example-es"
  namespace                     = var.namespace
  elasticsearch_version         = "7.9"
  aws-es-proxy-replica-count    = 2
  instance_type                 = "t2.medium.elasticsearch"
  s3_manual_snapshot_repository = module.es_snapshots_s3_bucket.bucket_arn
}

module "ns_annotation" {
  source              = "github.com/ministryofjustice/cloud-platform-terraform-ns-annotation?ref=0.0.2"
  ns_annotation_roles = [module.source_es.aws_iam_role_name]
  namespace           = var.namespace
}

This output es_snapshot_role will have permissions to enable the creation of a manual snapshot repo in s3 bucket

resource "kubernetes_secret" "es_snapshots_role" {
  metadata {
    name      = "es-snapshot-role"
    namespace = var.namespace
  }

  data = {
    snapshot_role_arn = module.example_es.snapshot_role_arn
  }
}

Step 2 - Spin up a new ES domain with “encrypt at rest” enabled.

  • Register the same snapshot repository on the source domain and the new domain
  • Add the aws_iam_role_name of source ES domain and new encrypted ES domain to the ns_annotation module.
  • Provide a different name to aws-es-proxy-service so it does not overwrite the existing source proxy.
module "encrypted_es" {
  source                          = "github.com/ministryofjustice/cloud-platform-terraform-elasticsearch?ref=3.8.0"
  cluster_name                    = var.cluster_name
  cluster_state_bucket            = var.cluster_state_bucket
  application                     = var.application
  business-unit                   = var.business-unit
  environment-name                = var.environment-name
  infrastructure-support          = var.infrastructure-support
  is-production                   = var.is-production
  team_name                       = var.team_name
  elasticsearch-domain            = "encrypted-example-es"
  encryption_at_rest              = true
  node_to_node_encryption_enabled = true
  namespace                       = var.namespace
  elasticsearch_version           = "7.9"
  aws-es-proxy-service            = "aws-es-proxy-service-encrypted"
  aws-es-proxy-replica-count      = 2
  instance_type                   = "t3.medium.elasticsearch"
  s3_manual_snapshot_repository   = module.es_snapshots_s3_bucket.bucket_arn
}

module "ns_annotation" {
  source              = "github.com/ministryofjustice/cloud-platform-terraform-ns-annotation?ref=0.0.2"
  ns_annotation_roles = [module.source_es.aws_iam_role_name, module.encrypted_es.aws_iam_role_name]
  namespace           = var.namespace
}

Step 3: Stop all apps connected to the Elasticsearch domain

Step 4: Register both Elasticsearch domains

  • Port forward the source ES domain using the proxy aws-es-proxy-service
kubectl \
  -n [your namespace] \
  port-forward \
  svc/aws-es-proxy-service 9200:9200

  • Register the snapshot repository with new encrypted Elasticsearch domain

    A sample template is given below:

  PUT http://localhost:9200/_snapshot/<repo name for snapshot>
{
  "type": "s3",
  "settings": {
    "bucket": "<s3 bucket for snapshots>",
    "region": "eu-west-2",
    "role_arn": "<role arn from es_snapshots_role secret>"
  }
}
  • Port forward the encrypted ES domain using the proxy aws-es-proxy-service-encrypted
kubectl \
  -n [your namespace] \
  port-forward \
  svc/aws-es-proxy-service-encrypted 9200:9200

  • Register the snapshot repository with encrypted Elasticsearch domain

    A sample template is given below:

  PUT http://localhost:9200/_snapshot/<repo-name-for-snapshot>
{
  "type": "s3",
  "settings": {
    "bucket": "<s3-bucket for snapshots>",
    "region": "eu-west-2",
    "role_arn": "<role arn from es_snapshots_role secret>"
  }
}

For more details on Elasticsearch REST APIs see ES API.

Step 5: Backup and Restore snapshots

  • Port forward the source ES domain using the proxy aws-es-proxy-service
kubectl \
  -n [your namespace] \
  port-forward \
  svc/aws-es-proxy-service 9200:9200
  • Create a snapshot
PUT http://localhost:9200/_snapshot/<repo-name-for-snapshot>/<snapshot-to-restore>
  • Port forward the encrypted Elasticsearch domain
kubectl \
  -n [your namespace] \
  port-forward \
  svc/aws-es-proxy-service-encrypted 9200:9200
  • Restore snapshot

NOTE: "include_global_state":true will restore all the key data

POST http://localhost:9200/_snapshot/<repo-name-for-snapshot>/<snapshot-to-restore>/_restore
{
  "include_global_state": true
}

Step 6: Connect apps to new ES

Update the application to use the new encrypted es_proxy service.

Step 7: Clean up

Cleanup the portforward pods and the unwanted elasticsearch service

For more details on backup snapshots and restore, see aws documentation

Last reviewed: 5 August 2025Review status: ✗ Review overdueOwner: #cloud-platformSource: View source on GitHub

Was this page useful?