Skip to main content

MeshDB Stateful Cutover Instruction

  1. Check that the destination copy of MeshDB is running the latest copy of MeshDB, by using the Deployments Tab in GitHub
  2. Disable the celery workers on the destination copy of MeshDB by running kubectl -n meshdb scale statefulset meshdb-celery-worker --replicas=0
  3. Place the destination copy of MeshDB into maintenance mode by running manage.py enable_flag MAINTENANCE_MODE inside of any of the meshweb Kubernetes pods
    1. Use a command like prod1 -n meshdb exec -it meshdb-meshweb-b956496c8-8wj5f -- bash to get the shell
  4. Make an announcement on slack in the #meshdb channel
    1. Template: Hey all, we're going to do some work on MeshDB over the next hour or so. The Admin panel, SQL Explorer, query form, and Mesh API, will all be unavailable. The website network map and join form will remain active. I will post back here when things are back to normal. Thank you for your patience
  5. Disable the celery workers on the originating copy of MeshDB by running kubectl -n meshdb scale statefulset meshdb-celery-worker --replicas=0
  6. Run curl https://db.nycmesh.net/api/v1/mapdata/nodes/ > prior_snapshot.json to create a snapshot of the map data, this approximates the overall state of the MeshDB data, and will be used later to confirm that the data was migrated succesfully
  7. Place the originating copy of MeshDB into maintenance mode by setting the MAINTENANCE_MODE feature flag to true via the admin UI or using the manage.py command as in step 1 above
    1. More info here: https://wiki.nycmesh.net/books/6-services-software/page/maintenance-mode
  8. Run pg_dump inside of the postgres pod on the originating copy of MeshDB to export the state of the database
    1. pg_dump -d meshdb -U meshdb -Fc >> meshdb_20250520.sql
    2. Doc reference: https://www.postgresql.org/docs/current/app-pgdump.html
  9. Use scp to copy the created file to your local machine
  10. Use scp to copy the created file to the destination copy of MeshDB
  11. Run psql inside the postgres pod on the destination copy of MeshDB to import the datadump
    1. pg_restore -d meshdb -U meshdb -c meshdb_20250520.sql
    2. Doc reference: https://www.postgresql.org/docs/current/app-pgrestore.html 
  12. Edit your /etc/hosts file to add the following entry <lb_ip> db.nycmesh.net 
  13. Run curl -k https://db.nycmesh.net/api/v1/mapdata/nodes/ > post_snapshot.json to create a post-data migration snapshot snapshot. Then run diff prior_snapshot.json post_snapshot.json to make sure the destination copy has correct data
  14. Modify the DNS CNAME record to point db.nycmesh.net to the LB at the supernode hosting the destination copy of MeshDB
    1. Example PR: https://github.com/nycmeshnet/nycmesh-dns/pull/210/files 
  15. Wait for the DNS change to roll out. Then modify the LB configuration for the LB at the destination supernode to pull TLS certificates for the MeshDB domain
    1. Example PR: https://github.com/nycmeshnet/k8s-infra/pull/99/files 
  16. Disable maintenance mode on the destination copy of MeshDB by running manage.py disable_flag MAINTENANCE_MODE inside of any of the meshweb Kubernetes pods
  17. Send a slack message to the #meshdb channel indicating the work is complete
    1. Template: Okay things should be back to normal now, please let us know if anything is not working correctly
Back to top