MeshDB Stateful Cutover Instruction

Check that the destination copy of MeshDB is running the latest copy of MeshDB, by using the Deployments Tab in GitHub Disable the celery workers on the destination copy of MeshDB by running kubectl -n meshdb scale statefulset meshdb-celery-worker --replicas=0 Place the destination copy of MeshDB into maintenance mode by running manage.py enable_flag MAINTENANCE_MODE inside of any of the meshweb Kubernetes pods Make an announcement on slack in the #meshdb channel

1. Template: Hey all, we're going to do some work on MeshDB over the next hour or so. The Admin panel, SQL Explorer, query form, and Mesh API, will all be unavailable. The website network map and join form will remain active. I will post back here when things are back to normal. Thank you for your patience

Disable the celery workers on the originating copy of MeshDB by running kubectl -n meshdb scale statefulset meshdb-celery-worker --replicas=0 Run curl https://db.nycmesh.net/api/v1/mapdata/nodes/ > prior_snapshot.json to create a snapshot of the map data, this approximates the overall state of the MeshDB data, and will be used later to confirm that the data was migrated succesfully Place the originating copy of MeshDB into maintenance mode by setting the MAINTENANCE_MODE feature flag to true via the admin UI or using the manage.py command as in step 1 above

1. More info here: https://wiki.nycmesh.net/books/6-services-software/page/maintenance-mode

Run pg_dump inside of the postgres pod on the originating copy of MeshDB to export the state of the database Use scp to copy the created file to your local machine Use scp to copy the created file to the destination copy of MeshDB Run psql inside the postgres pod on the destination copy of MeshDB to import the datadump
....Edit your /etc/hosts file to add the following entry <lb_ip> db.nycmesh.net  Run curl -k https://db.nycmesh.net/api/v1/mapdata/nodes/ > post_snapshot.json to create a post-data migration snapshot snapshot. Then run diff prior_snapshot.json post_snapshot.json to make sure the destination copy has correct data
Modify the DNS CNAME record to point db.nycmesh.net to the LB at the supernode hosting the destination copy of MeshDB

1. Example PR: https://github.com/nycmeshnet/nycmesh-dns/pull/210/files 

Wait for the DNS change to roll out. Then modify the LB configuration for the LB at the destination supernode to pull TLS certificates for the MeshDB domain

1. Example PR: https://github.com/nycmeshnet/k8s-infra/pull/99/files 

Disable maintenance mode on the destination copy of MeshDB by running manage.py disable_flag MAINTENANCE_MODE inside of any of the meshweb Kubernetes pods Send a slack message to the #meshdb channel indicating the work is complete

1. Template: Okay things should be back to normal now, please let us know if anything is not working correctly