Skip to main content

MeshDB Stateful Cutover Instruction

  1. Check that the destination copy of MeshDB is running the latest copy of MeshDB, by using the Deployments Tab in GitHub
  2. Disable the celery workers on the destination copy of MeshDB by running kubectl -n meshdb scale statefulset meshdb-celery-worker --replicas=0
  3. Place the destination copy of MeshDB into maintenance mode by running manage.py enable_flag MAINTENANCE_MODE inside of any of the meshweb Kubernetes pods
    1. Use a command like prod1 -n meshdb exec -it meshdb-meshweb-b956496c8-8wj5f -- bash to get the shell
  4. Scale meshweb to 0 on the destination copy of MeshDB with prod2 -n meshdb scale deploy meshdb-meshweb --replicas 0
  5. Make an announcement on slack in the #meshdb channel
    1. Template: Hey all, we're going to do some work on MeshDB over the next hour or so. The Admin panel, SQL Explorer, query form, and Mesh API, will all be unavailable. The website network map and join form will remain active. I will post back here when things are back to normal. Thank you for your patience
  6. Disable the celery workers on the originating copy of MeshDB by running kubectl -n meshdb scale statefulset meshdb-celery-worker --replicas=0
  7. Run curl https://db.nycmesh.net/api/v1/mapdata/nodes/ > prior_snapshot.json to create a snapshot of the map data, this approximates the overall state of the MeshDB data, and will be used later to confirm that the data was migrated succesfully
  8. Place the originating copy of MeshDB into maintenance mode by setting the MAINTENANCE_MODE feature flag to true via the admin UI or using the manage.py command as in step 1 above
    1. More info here: https://wiki.nycmesh.net/books/6-services-software/page/maintenance-mode
  9. Scale meshweb to 0 on the originating copy of MeshDB prod1 -n meshdb scale deploy meshdb-meshweb --replicas 0
  10. Run pg_dump inside of the postgres pod on the originating copy of MeshDB to export the state of the database
    1. prod2 -n meshdb exec meshdb-postgres-6956db77c8-8xjm9 -- pg_dump -d meshdb -U meshdb -Fc >> meshdb_20250520.sql
    2. Or get a bash shell and run pg_dump -d meshdb -U meshdb -Fc >> meshdb_20250520.sql directly & use scp to copy the file to your local host
    3. Doc reference: https://www.postgresql.org/docs/current/app-pgdump.html
  11. Use scp to copy the database dump to the destination postgres pod
  12. Run psql inside the postgres pod on the destination copy of MeshDB to import the datadump
    1. pg_restore -d meshdb -U meshdb -c meshdb_20250520.sql
    2. Doc reference: https://www.postgresql.org/docs/current/app-pgrestore.html 
  13. Edit your /etc/hosts file to add the following entry <lb_ip> db.nycmesh.net 
  14. Run curl -k https://db.nycmesh.net/api/v1/mapdata/nodes/ > post_snapshot.json to create a post-data migration snapshot snapshot. Then run diff prior_snapshot.json post_snapshot.json to make sure the destination copy has correct data
  15. Modify the DNS CNAME record to point db.nycmesh.net to the LB at the supernode hosting the destination copy of MeshDB
    1. Example PR: https://github.com/nycmeshnet/nycmesh-dns/pull/210/files 
  16. Wait for the DNS change to roll out. Then modify the LB configuration for the LB at the destination supernode to pull TLS certificates for the MeshDB domain
    1. Example PR: https://github.com/nycmeshnet/k8s-infra/pull/99/files 
  17. Disable maintenance mode on the destination copy of MeshDB by running manage.py disable_flag MAINTENANCE_MODE inside of any of the meshweb Kubernetes pods
  18. Send a slack message to the #meshdb channel indicating the work is complete
    1. Template: Okay things should be back to normal now, please let us know if anything is not working correctly
No Comments
Back to top