Skip to main content

MeshDB Stateful Cutover Instruction

  1. Disable the celery workers on the destination copy of MeshDB by running kubectl -n meshdb scale statefulset meshdb-celery-worker --replicas=0
  2. Place the destination copy of MeshDB into maintenance mode by running manage.py enable_flag MAINTENANCE_MODE inside of any of the meshweb Kubernetes pods
  3. Make an announcement on slack in the #meshdb channel
    1. Template: Hey all, we're going to do some work on MeshDB over the next hour or so. The Admin panel, SQL Explorer, query form, and Mesh API, will all be unavailable. The website network map and join form will remain active. I will post back here when things are back to normal. Thank you for your patience
  4. Disable the celery workers on the originating copy of MeshDB by running kubectl -n meshdb scale statefulset meshdb-celery-worker --replicas=0
  5. Place the originating copy of MeshDB into maintenance mode by setting the MAINTENANCE_MODE feature flag to true via the admin UI or using the manage.py command as in step 1 above
    1. More info here: https://wiki.nycmesh.net/books/6-services-software/page/maintenance-mode
  6. Run pg_dump inside of the postgres pod on the originating copy of MeshDB to export the state of the database
  7. Use scp to copy the created file to your local machine
  8. Use scp to copy the created file to the destination copy of MeshDB
  9. Run psql inside the postgres pod on the destination copy of MeshDB to import the datadump
  10. ....
  11. Modify the DNS CNAME record to point db.nycmesh.net to the LB at the supernode hosting the destination copy of MeshDB
    1. Example PR: https://github.com/nycmeshnet/nycmesh-dns/pull/210/files 
  12. Wait for the DNS change to roll out. Then modify the LB configuration for the LB at the destination supernode to pull TLS certificates for the MeshDB domain
    1. Example PR: https://github.com/nycmeshnet/k8s-infra/pull/99/files 
  13. Disable maintenance mode on the destination copy of MeshDB by running manage.py disable_flag MAINTENANCE_MODE inside of any of the meshweb Kubernetes pods
  14. Send a slack message to the #meshdb channel indicating the work is complete
    1. Template: Okay things should be back to normal now, please let us know if anything is not working correctly
Back to top