Runbook: Manually Editing Terraform State

Sometimes, you might need to do some surgery on our terraform scripts. This can happen when a provider we use makes a breaking change that makes our current code definition incompatible with the current terraform state.

To resolve this, you will need to download terraform on your laptop, get the terraform state out of the S3 bucket, make changes to it either with a script or with an editor, and push it back up.

As a prerequisite, you need two things: backend.tfvars and a .env file containing the credentials to get into the S3 bucket.

The former looks like this (beware, the format differs slightly between projects):

bucket = "nycmesh-dns"
key = "terraform/state/dns-prod_sn3.tfstate"

The latter, you will need to get from Andrew Dickinson, Willard Nilges, or James Otten. There is an S3 bucket in Andrew's personal AWS account that stores the TF state. If given access, you will be able to copy credentials and add them to your environment.

Now you're ready to fix the state.

cd ~/Code/nycmesh/nycmesh-dns/infra/terraform/      # You need to be cd'ed into whatever IaC project you're working with.
                                                    # Refer to the actions in .github/ for inspiration.
source .env 
terraform init -backend-config=backend.tfvars
terraform state pull > terraform.tfstate.backup
cp terraform.tfstate.backup terraform.tfstate.fixed # ALWAYS KEEP YOUR BACKUP PRISTINE
nvim terraform.tfstate.fixed                        # Make your changes and increment the serial number at the top of the file
terraform state push terraform.tfstate.fixed