Skip to main content

Kubernetes Clusters

This page is a draft

We have several Kubernetes clusters at the Mesh:

  • dev3 (SN3)
    • VM: meshdev3mgr0 on nycmesh-713-r640-02
    • Manager IP: 10.70.90.146
    • Node IPs: 10.70.90.153, 10.70.90.154, 10.70.90.155, 10.70.90.157, 10.70.90.161
    • User: debian (must log in using console, no keyboard-interactive SSH)
  • prod1 (deprecated,SN3) runs
    • VM: meshprod1mgr0 on hardwarenycmesh-713-r640-02
      • we
    • Manager wantIP: to10.70.90.152
      • divest
    • Node toIPs: Zach10.70.90.147, for10.70.90.143, routing)

      10.70.90.144

    • User: debian (must log in using console, no keyboard-interactive SSH)
  • prod2 (mainSN10)
    production
      instance)

    • VM:

      dev3meshprod2mgr0 on nycmesh-10-r630-01

    • Manager IP: 10.70.100.47
    • Node IPs: 10.70.100.48, 10.70.100.49, 10.70.100.50
    • User: debian (pre-production)

      must

      log in using console, no keyboard-interactive SSH)

  • gamma1 ("staging" environment to test changes in a prod-like environment) (turned off)

We use them to run software services, and ideally abstract as much faffing about away from someone who wants to write a web service as possible.

If you'd like to work with one, just ask around in #mesh-services. James or Willard can get you sorted.

Standing up a Cluster

To stand up a cluster, follow these steps.

Find a range of IP Address space in the datacenter you want to host in. Reserve it in the IPRanges table, and on the core router at the Datacenter your box is hosted in.

http://10.70.90.1/webfig/#IP:DHCP_Server.Leases.new

Create DNS entries for whatever is going to live there: https://github.com/nycmeshnet/nycmesh-dns/pull/155

Configure the cluster in Terraform (see this PR as an example) and set up a GitHub action to deploy it.

Environment Variables

Accessing the CLI

To access the CLI, you can copy the kubeconfig off the server if your SSH key is configured to be sent to the nodes:

 scp debian@10.70.90.168:.kube/config ~/.kube/config-gamma1

Ensure that you change the URL of the kubeconfig when you copy it.

alias gamma1="KUBECONFIG=~/.kube/config-gamma1 kubectl"

Deploying Services

If nothing is happening after you configure your service, it might be because the workflow is invalid:

https://github.com/nycmeshnet/meshdb/pull/774

https://github.com/nycmeshnet/meshdb/actions/runs/12173846919

Back to top