OSPF

Open Shortest Path First (OSPF) is a dynamic routing protocol. It uses a link state routing algorithm, thus, it performs functions such as detecting topology changes and link failures. It generally converges quickly and in a loop-free manner. OSPF is often used in corporate networks within a datacenter or building. While OSPF is not generally used as a mesh protocol across a city, it has properties similar to other mesh protocols such as use of link state routing algorithms and auto-convergence.

NYC Mesh OSPF Routing Methodology

This guide gives an overview of the OSPF methodology and topology currently deployed at NYC Mesh. If you are looking for specific configuration instructions, please refer to the OSPF Configuration Guide page.

Positives and negatives

OSPF is an interesting choice as an in-neighborhood routing protocol because of its ease of setup (auto convergence, no ASNs), and how ubiquitous it is -- nearly every cheap and expensive commercial and open device supports it. These two positives alone make OSPF worth considering.

On the down-side, it is not specifically designed for an adhoc mesh, it trusts blindly, and has very few tuneables. Additionally, there are a few technical challenges such as the lack of link-local address use, only advertising connected networks (not summaries), and some common defaults on various platforms.  

Many of these challenges can be overcome by taking some care to make good choices for options when setting up a network.

OSPF Selection

NYC Mesh has chosen to use OSPF as the standard mesh routing protocol of choice. This may be a controversial choice, as _most_ mesh networks in Europe are using custom mesh routing protocols, or encrypted routing protocols. We have chosen this path because:

NYC Mesh utilizes a wide range of hardware with differing capacities and weather resiliency characteristics. Being volunteer-driven and operated, it's important that the network be resilient, but also easy to maintain and scale. OSPFv2 Point-to-Multipoint allows us to modify routing tables and plan for expansion without overly-complicated configuration planning.

Important Note: making changes to default OSPF costs can have unintended effects, up to and including network-wide outages, and frequently requires modifying Hub configurations. Testing and learning should not be done in production environments. Before making changes to NYC Mesh routing configurations, discuss in our Slack #architecture channel and/or applicable hub channel and check the NYC Mesh Node Explorer tool for current routing information.

Designing the basic architecture

To standardize across the network, each router has a Mesh Bridge Interface on the OSPF Area with default cost of 10 to all adjacent neighbors. This ensures symmetry in link costs on both ends of the link, keeping bi-directional traffic following the same path. For each "hop" to an internet exit, each router incurs its link cost to transit to the next hop. By calculating the lowest cost to an internet exit, the local router sends its traffic on an Internet exit in (usually) the most efficient manner, automatically.

Example: Node path to internet exit with all default costs

Node A > 10 > Hub > 10 > Supernode > 1 > Public Internet

In the above example, the Node incurs cost 10+10+1=21 to exit to the Public Internet. Unless a lower cost exit becomes available, this will be the preferred route for all internet traffic to and from Node A.

Now that we've standardized route costs, we need to design priority and redundancy to take advantage of nodes clustered around each other while preferring higher-capacity links.

The WDS bridge: ensuring Hub-and-spoke routes are preferred over WDS routes

NYC Mesh uses Omnitik wireless routers at almost all member nodes to automatically connect to each other, providing numerous backup routes in case of hardware failure or network changes, but these connections are often slower and less reliable than point-to-point and point-to-multipoint connections in our Hub-and-Spoke model. To account for this, we put the Omnitik<>Omnitik WDS links on a separate "WDS Bridge" on every Omnitik router with default cost of 100. 

Example: Node preferring Mesh Bridge over "shorter" WDS links

Node A > 100 (WDS) > Node B > 10 > Supernode X > 1 > Public Internet 
Node A > 10 > Microhub > 10 > Hub > 10 >  Supernode Y > 1 > Public Internet

In this example, Node A prefers to exit via Supernode Y as the cost it incurs is 10+10+10+1=31, versus 100+10+1=111 via Supernode X. If we did not have higher WDS costs, Node A would instead prefer the shorter link to Supernode X, but would very likely experience poorer performance.

For more details on the hybrid Hub-and-Spoke + Mesh model we deploy, see the Mesh page.

Example: Prospect Lefferts Garden

image.png

In Figure 1, we see many nodes (marked as red dots) clustered around 2 Microhubs (marked as blue dots) in Prospect Lefferts Garden, as well as multiple exit routes to the north. While most of the nodes will automatically find the best exit, there are some that may have equal costs through multiple exits. To mitigate this, we set preferred routes (via hardware like SXTs, or software with virtual wireless interfaces) on the Mesh Bridge, as illustrated by the green lines in Figure 2. This ensures each node selects its fastest and most stable route to send and receive internet traffic.

image.png

We can see this in action on the Omnitik: 10.69.45.7 is on the the "Mesh" bridge interface, meaning it incurs cost 10 to transit. All other adjacent routers are on the "WDS" bridge interface, and incur cost 100 to transit. This setup ensures the local node prefers the 4507 Microhub as its exit route, but also has backup routes in case 4507 goes offline or one of its upstream links is broken.

By implementing this architecture across all routers on the network, we now have high resiliency to outages, scalability, and minimal configuration effort.

Bridge Filters

Before we go further, its important to mention bridge filters. These are required to make OSPF work properly and also ensure members within the same building are isolated from each other. There are 3 filters applied to standard Omnitik configurations.

[admin@nycmesh-xxxx-omni] > interface bridge filter print
Flags: X - disabled, I - invalid, D - dynamic 
 0   chain=forward action=drop in-bridge=mesh log=no log-prefix=""
 1   chain=forward action=drop in-bridge=wds
 2   chain=forward action=drop in-interface=wlan2 

It is critically important to ensure these filters stay enabled on each router to ensure individual routers don't "bridge" connected nodes and Hubs, leading to unintended routing paths. Further explanation and examples of bridging scenarios are covered below.

Sidebar: The case against OSPF automation and summarization

Given the scale of the problem and continued growth, we must consider an important question: why not implement automation to dynamically adjust OSPF costs based on link quality, and/or utilize summarization and redistribution to simplify planning?

As mentioned above, our network design is meant in part to balance the following three goals:

Further, NYC Mesh has no CEO, directors, or employees, and the board intentionally does not have decision-making authority over non-financial/legal matters; as documented in the NYC Mesh Commons License, the design, planning, maintenance and support of NYC Mesh is done solely by community members and volunteers. While we do have highly-skilled volunteer network engineers, the day-to-day maintenance and monitoring of the network is done by members with varied skill levels; we generally prefer easy-to-maintain solutions over highly customized configurations requiring extensive knowledge and training.

Finally, as we primarily rely on member donations to maintain and expand the network, we generally avoid high-end enterprise-grade hardware or software requiring recurring subscription fees and support contracts to minimize operational expenses. As NYC Mesh continues to grow, we may need to adopt more robust and dynamic routing and load-balancing techniques, and will look to our community to collectively decide on the path forward.

Scaling out the Hub-and-Spoke model

This baseline architecture works great in individual neighborhoods and on relatively linear routes, but with over a thousand nodes connecting to 60+ Hubs with links crisscrossing New York City, some planning and manual intervention is required to ensure stability and speed for all connected members. 

image.png

In the Bed-Stuy, Bushwick, Ridgewood, and Crown Heights neighborhoods show above in Figure 3, we have over a dozen Hubs serving hundreds of members. Efficient routing and redundancy across multiple wireless links requires further options for route cost between 10 and 100.

In efforts to minimize single points of failure in our network (hubs having only 1 exit route) and provide dedicated backup routes in cases of weather impacting high-frequency links, we deploy redundant links in a "triangle schema" so that each hub has multiple low-cost routes to exit. To see this deployed, let's remove the nodes from the above photo and focus on the Hubs.

image.png

As we can see in Figure 4, most Hubs have 2 or more exit routes so that an outage of an individual link or Hub will not isolate any other Hub. Additional routes leading off Figure 4 allow multiple exits from both Vernon and Hex House, as well as other lower-capacity links through smaller Microhubs and nodes.

image.png

In Figure 5, we observe a similar trend as we move southwest towards Prospect Park and Supernode 3 at Industry City.

Load-balancing across varied hardware 

NYC Mesh uses a broad range of purchased and donated Ubiquiti and Mikrotik hardware with varying capacity, capabilities, and rain fade resilience, and members are allowed to extend the network at will pursuant to the Network Commons License. Because our OSPF link costs are static and do not automatically increase or decrease based on link quality, limiting ourselves to just two options for link cost will quickly cause issues as the network grows. Here are just a few use cases to consider:

To meet these goals, we need to set up custom link costs on backup routes as well as between high-traffic Hubs.

Example: Microhubs between Major Hubs

Our Vernon and Prospect Heights Hubs collectively carry more than 80% of NYC Mesh network traffic in Brooklyn. By design, each Hub's primary exit is through different Supernodes to the public Internet (Vernon through Supernode 10 in Manhattan, and Prospect Heights through Supernode 3 in Industry City). To allow redundancy between their exits, a dedicated 60GHz link (in teal) is deployed between the two, but Vernon and Prospect Heights also have more preferrable secondary links (illustrated further below in Figure 7). This requires the link to have a slightly higher cost (in this case, 15) so that each Hub prefers other backup routes in case of primary exit link outages.

To make matters more complicated, Microhubs in between Vernon and Prospect Heights connect to both Hubs to provide their own redundancy, as illustrated in Figure 6.

image.pngNote: nodes and sector coverage have been omitted

To ensure each Microhub prefers the fastest route and they don't bridge Vernon and Prospect Heights by having additive link costs lower than the Vernon <> Prospect Heights 60GHz link, we need to manually set the backup links with higher costs.

Reminder: before setting up secondary links, double-check that the appropriate Bridge Filters are enabled on the local router. A misconfigured or disabled Mesh Bridge Filter will result in 0-cost links between neighboring nodes and Hubs, risking major network congestion or outages.

Note that there is no firm methodology or formula for calculating optimal custom link costs in this model, though backup links are generally set between 20 and 80 depending on upstream impacts. Sufficient buffer should be allocated between primary and secondary routes to allow expansion and updates with minimal changes required to upstream OSPF costs or routes.

The NYC Mesh Node Explorer tool generates live and historical mappings of nodes and link costs, allowing us to quickly determine primary exit routes and costs, as well as an outage simulator tool that is extremely helpful in validating configuration of secondary routes.

You can also determine current exit cost of any Mikrotik router running RouterOS v6 with the following command:

[admin@nycmesh-xxxx-core] > routing ospf route print where dst-address =0.0.0.0/0
 # DST-ADDRESS        STATE          COST          GATEWAY         INTERFACE     
 0 0.0.0.0/0          ext-1          20            10.70.253.xx    bond1.1010    

Note: identifying information has been removed from this terminal export

When selecting a custom link cost that may bridge segments of the network, the following factors should be taken into account:

Planning for Outages

Ok, to summarize, we've done the following:

To determine route costs to set on primary and backup routes, we need to understand more about the wireless hardware used in the Mesh. The table below details characteristics of the common equipment currently in use on the Mesh.

Brand Antenna Band Advertised Capacity* Typical Capacity* Rain Resilience Preferred Link Distance**
Ubiquiti Litebeam 5AC Gen2
Litebeam LR		 5GHz 225 Mbps+
(40 MHz width)		 75-175 Mbps

 Extremely High < 3.5km (PtP)
< 2km (PtMP)
Ubiquiti airFiber 5XHD
LTU Long-Range		 5GHz 425 Mbps+
(80MHz width)		 125-300 Mbps

 Extremely High < 5km (PtP)
< 3km (PtMP)
Ubiquiti

airFiber 24

 24GHz 750 Mbps
(100MHz width)		 750 Mbps Very High < 5km (PtP)
Ubiquiti

airFiber 60 LR
Wave LR

 60GHz 1Gbps
(1080MHz width)
 1Gbps Medium < 3.5km (PtP)
< 2km (PtMP)
Ubiquiti airFiber 60 XR 60GHz 2.7Gbps
(2160MHz width)
 2.7Gbps Low/Medium < 5km (PtP)
Mikrotik SXTsq 5AC 5GHz 200 Mbps+
(40 MHz width)		 75-100 Mbps Extremely High < 1.5km (PtP)
< 500m (PtMP)
Mikrotik LHG 5AC 5GHz 200 Mbps+
(40 MHz width)		 75-125 Mbps Extremely High < 3.5km (PtP)
< 1.5km (PtMP)
Mikrotik LHG 60 60GHz 1Gbps
(1080MHz width)		 300-600Mbps Low/Medium < 1.5km (PtP)
< 500m (PtMP)
Siklu Etherhaul Kilo 8010
(licensed band)		 70GHz
80GHz		 10Gbps
(2160MHz width)
 10Gbps High < 5km (PtP)

*  Capacity listed is single-direction speed; Typical Capacity indicates observed performance in New York City
** Preferred Link Distance is a subjective estimate of maximum distance in dense urban areas before performance is significantly degraded

As we can see, decisions on route priority depend on the capacity of individual links, as well link distance (for rain resiliency) and count of hops (for latency) to an internet exit. That's a lot of factors to consider! Let's see what this looks like in the real world.

Determining Primary and Backup Costs

To see how these factors are taken into account when planning for real-world deployments, let's return to Brooklyn.

image.pngNote: some additional links and hubs omitted for clarity; listed link speeds are production single-direction actuals; distances between Hubs are not to scale

Figure 7.1 illustrates links between larger Hubs in Brooklyn, with notations indicating deployed hardware, directional link capacity and physical distance. Supernodes in green serve as Internet Exits; all other Hubs are marked in blue. As mentioned earlier in this article, all OSPF link costs are symmetrical to ensure consistent bi-directional traffic flow and ease of configuration.

Because OSPF will always prefer the lowest-cost route to an exit, its easier to work from the outside-in (i.e., starting with the Supernodes adjacent to the Internet Exits and working our way deeper into the network). To begin planning our link costs, let's start by identifying the Public Internet exits:

image.png

With the Public Internet exits and costs identified in red in Figure 7.2, the network will operate mostly ok with default costs of 10 (noted in grey) for all other links... until we experience outages, whether from heavy rain or rare hardware failure. 

To optimize the network, we'll need to make some changes to some of our primary link costs.

With these changes in place, Vernon now prioritizes traffic correctly over its highest-capacity link, and has a dedicated high-capacity secondary failover link in case of hardware failure or a Grand outage. Additionally, Saratoga will now also follow Vernon's exit to Grand.

image.png

We now have traffic moving more efficiently, and can operate with plenty of overhead for traffic spikes and growth. However, if we experience heavy rainfall or suffer outages, we may still have problems. We'd also like to make sure we don't have to redesign the entire OSPF schema from the ground up very time a new Hub is stood up. We'll work to address that now:

Let's see how our network looks in Figure 7.4 now that we've put these changes in place.

image.png

We now have efficient routing in place with high-resiliency backups for rain as well as high-capacity backups for hardware failures and outages!

If you've made it this far, you should have a good understanding of how to safely manage routes and costs on the Mesh, and be able to plan for new nodes and Hubs armed with a better understanding of traffic flow across the network.

To learn how to configure OSPF routes on our Mikrotik routers, see the OSPF Configuration Guide to get started.

Appendix: Brooklyn Hub OSPF Costs

Note: the routing details in this document are accurate as of April 11th, 2024. Before making production routing changes, check the NYC Mesh Node Explorer tool and discuss in our Slack #architecture channel.

1340 - Saratoga

3461 - Prospect Heights:

5151- President

5916 - Vernon

image.png

Rules and Standards

As OSPF has some challenges in deployment outside of a datacenter environment, we will need to all adhere to some rules to prevent from encountering a routing problem. OSPF, unlike BGP or other protocols, may/will refuse to connect to a peer which has different parameters set and can be a source of confusion.  Please follow these rules unless there is a good reason not to:

  1. General    
    1. All OSPF usage will be on area 0.0.0.0 ( backbone area )     
    2. Set Router ID to the 10-69-net address of the Node
    3. All OSPF timers will be set to ( this is typically default ):
      • Link Cost 10
      • Retransmit Interval 5
      • Transmit Delay 1
      • Hello Interval 10
      • Dead Router Interval 40  
  2. Interfaces should be in PtMP mode (not broadcast)   
  3. OSPF "networks" should only be the 10-69-net, unless there is a special case
  4. Redistribute Default Route should be never, unless you are distributing a default route
    • If you are redistributing a default route, do so as type 1 
    • Check with the rest of the network what the correct metric should be
  5. Redistribute user networks via Redistribute Connected as type 1 
  6. Mikrotik Only: Filter VPN point-to-point /32 links. They cause trouble.   
  7. For RouterOS 6, do not also run BGP on the standard mesh bridge, unless there is a special case; this has been known to cause problems in the past
  8. For RouterOS 7, BGP is working on the mesh bridge in a few locations as of July 2024; as ROS7 is not widely utilized within NYC Mesh yet, thorough testing and monitoring should be done for ROS7 routers

OSPF Configuration Guide

This guide contains in-depth parameters on Layer 2 and 3 configurations. This guide is not meant to be a comprehensive guide on overall networking, but an extension of existing concepts to solve Mesh-specific problems.

It is encouraged that you familiarize yourself with the NYC Mesh OSPF Routing Methodology to better understand the concepts and techniques in this page prior to putting these methods into practice.

What is the "Mesh bridge"?

On the Mesh, almost all routers have a "mesh bridge", which is an isolated, virtualized switch that connects home/apartment routers and radios together. All of the wireless radios on the NYC Mesh network are configured to operate in "bridge" mode, which means that the routers themselves are responsible for sending and routing packets to their neighbors which will be visible through the connected radios. As detailed in our routing methodology, this bridge is set with a standardized cost of 10 across the network to facilitate simple deployment of new equipment and ease-of-use for the volunteers that support and maintain it.

To see what OSPF interfaces are configured on a Mikrotik router on ROS6, use the routing ospf interface print command.

[admin@nycmesh-8300-omni] > routing ospf interface print
Flags: X - disabled, I - inactive, D - dynamic, P - passive 
 #    INTERFACE    COST PRIORITY NETWORK-TYPE   AUTHENTICATION AUTHENTICATION-KEY
 0    mesh           10        1 ptmp           none                             
 1    wds           100        1 ptmp           none

image.png

Adjacent routers in an OSPF area see each other as "neighbors", and each router is aware of all other routers in its area. On Mikrotik routers, you can see the OSPF neighbors by using the routing ospf neighbor print command.

[admin@nycmesh-8300-omni] > routing ospf neighbor print
 0 instance=default router-id=10.69.83.1 address=10.69.83.1 interface=mesh 
   priority=1 dr-address=0.0.0.0 backup-dr-address=0.0.0.0 state="Full" 
   state-changes=4 ls-retransmits=0 ls-requests=0 db-summaries=0 
   adjacency=4h39m31s 

 1 instance=default router-id=10.69.5.44 address=10.69.5.44 interface=mesh 
   priority=1 dr-address=0.0.0.0 backup-dr-address=0.0.0.0 state="Full" 
   state-changes=5 ls-retransmits=0 ls-requests=0 db-summaries=0 
   adjacency=4h39m38s

image.png

In the above example, we see that this router has two neighbors, 10.69.5.44 and 10.69.83.1, and both are on the "mesh" interface which has a cost of 10.

Why would you need a non-standard OSPF interface?

Sometimes the need will arise that a particular link should have a different cost than the mesh bridge standard, such as:

The solution to this is to create a new dedicated OSPF interface (which is not on the mesh bridge) between two routers with a point-to-point address space, and a non-standard cost.

Considerations for Configuration

The implementation of an OSPF interface depends on two factors:

The following table shows the most common link types and their requirements for OSPF configuration. Note that each end of a link may have different characteristics, and therefore different requirements. Some routers and radios support VLAN tagging on egress; for the purposes of simplicity and conformity to NYC Mesh standards, VLAN interfaces should be used as follows


Sidebar: nuances of VLAN configurations with Mikrotik and Ubiquiti Hardware

Before describing implementation steps, it's important to understand how VLANs work on switch ports; in all PtMP links, or if either end of the link has a switch between the radio and router (extremely common at NYC Mesh hubs), you will need to create VLAN interfaces on the router and instruct the switch where to send the VLAN-tagged traffic.

The first concept to understand is Ingress vs Egress Traffic

The second concept is a VLAN interface on Mikrotik devices, which is a virtual interface associated with a physical port that "listens for" tagged traffic through the physical port only with the specified VLAN ID, and automatically untags that traffic before passing it onto the bridge or OSPF instance. This also works in reverse; untagged traffic on the bridge or OSPF Interface can "enter" the vlan interface and is tagged before egress out the physical port.

image.png
Multiple VLAN Interfaces on a single physical router port 

The third concept is VLAN behavior on switch ports. On a managed switch, generally you must define what ports are allowed to pass traffic with specific VLAN tags in the VLAN table, but you also have the ability to add, modify, and remove tags on traffic entering and exiting the switch port, which will be key for configuration later in this guide. Mikrotik and Ubiquiti use similar terminology on their switches:

[admin@nycmesh-219-netpower16p] /interface bridge vlan> print
Flags: X - disabled, D - dynamic 
 #   BRIDGE           VLAN-IDS  CURRENT-TAGGED          CURRENT-UNTAGGED         
 0   ;;; lbelr to PH
     bridge           1072      ether1                 
                                ether15                
 1   bridge           99        ether1                  sfp-sfpplus1             
 2   ;;; Revere 7985 (lbe)
     bridge           2301      ether1                 
                                ether14                
 3   ;;; mgmt
     bridge           50        ether1                 
                                ether15                
                                ether13                
 4   ;;; af60 to Vernon
     bridge           1092      ether1                 
                                ether13           

image.png
Bridge VLAN table on a Mikrotik Netpower 16P 

Implementation scenarios with directly-connected antennas

The following section details the configuration steps for the scenarios listed in the table above. While this will not cover every scenario you may find, it should give you the knowledge needed to create links of varying complexity.

Important Notes: Making changes to default OSPF costs can have unintended effects, up to and including network-wide outages, and frequently requires modifying Hub configurations. Testing and learning should not be done in production environments. Before making changes to NYC Mesh routing configurations, discuss in our Slack #architecture channel and/or applicable hub channel and check the NYC Mesh Node Explorer tool for current routing information.

Before setting up secondary links, double-check that the appropriate Bridge Filters are enabled on the local router. A misconfigured or disabled Mesh Bridge Filter may result in 0-cost links between neighboring nodes and Hubs, risking major network congestion or outages.

When implementing changes in production, be mindful of order of operations and availability of backup routes; removing a remote interface from the mesh bridge will cause the link to drop and you may lose remote access to the router if there is not another OSPF neighbor visible to the router

At a high level, this configuration is simpler than it sounds.

  1. Reserve a point-to-point address space (/31 for Mikrotik, /30 for Juniper) for the two routers to connect
  2. If the data will traverse through a switch or an Access Point/PtMP antenna on either end, create VLAN interfaces on both routers for tagged traffic to traverse the link
  3. On the non-AP side, create a second VLAN interface for management access to the radio, added to the mesh bridge, and set the radio's management VLAN to match
  4. Add the IP addresses from the point-to-point address space to the interfaces on both ends
  5. Add the physical (for PtP with no switches) or VLAN interfaces as OSPF interfaces on the routers as "PTMP" network type with bfd enabled
  6. Add the point-to-point OSPF address space to the backbone
  7. On the non-AP side(s), remove/disable the physical interface from the Mesh bridge so data can only traverse through the new OSPF interface
  8. If data will traverse through a switch, update the switch vlan table to allow tagged traffic to pass

This diagram shows how this would look logically for a Microhub on the left to connect to two different Hubs with separate exits; this grants automatic failover for Microhub A in the event that the primary link, or Hub B / Supernode X goes down.

image.png

Scenario 1: Point to Point with directly-attached antennas

Router A > untagged > Antenna A <> Antenna B < untagged < Router B

This is the simplest configuration, as each end of the link has an antenna (or hardwire) plugged directly into the routers on each end. Because all traffic through the link should be on the new interface, we do not need to create any VLAN interfaces; the physical port itself can be removed from the Mesh bridge and added as a standalone OSPF interface with adjusted cost. 

  1. If performing this on a remote site, ensure the link being adjusted is not a single point of failure before proceeding
  2. Reserve an unused /31 IP range to be used for the OSPF network (ask in the #architecture channel for access to the IP ranges sheet for more information on this)
    • For this example, I will use 10.70.253.200/31
    • If one or both of the routers is running Juniper OS, you will need a /30 reservation
  3. On each router, add the reserved IPs and network to the appropriate interfaces
    RouterA: /ip address add address=10.70.253.200 network=10.70.253.201 interface=ether1
RouterB: /ip address add address=10.70.253.201 network=10.70.253.200 interface=ether5

    You can verify they were added correctly with /ip address print

    [admin@nycmesh-8300-omni] > ip address print
Flags: X - disabled, I - invalid, D - dynamic 
 #   ADDRESS            NETWORK         INTERFACE                                
 0   10.104.27.1/26     10.104.27.0     mesh                                     
 1   10.69.83.0/16      10.69.0.0       mesh                                     
 2   10.68.83.0/16      10.68.0.0       wds                                      
 3   10.70.253.200/32   10.70.253.201   ether1 
  4. On each router, add the physical port as an OSPF interface with the correct cost, set for "PTMP", and enable BFD. For this example, I will use cost 9.
    RouterA: /routing ospf interface add cost=9 interface=ether1 network-type=ptmp use-bfd=yes
RouterB: /routing ospf interface add cost=9 interface=ether5 network-type=ptmp use-bfd=yes
    You can verify they were added correctly with /routing ospf interface print
    [admin@nycmesh-8300-omni] > routing ospf interface print
Flags: X - disabled, I - inactive, D - dynamic, P - passive 
 #    INTERFACE    COST PRIORITY NETWORK-TYPE   AUTHENTICATION AUTHENTICATION-KEY
 0    mesh           10        1 ptmp           none                             
 1    wds           100        1 ptmp           none                             
 2    ether1          9        1 ptmp           none
  5.  On each router, add the OSPF Network so the OSPF interface is routable. Note that this command is identical on both ends.
    RouterA: /routing ospf network add area=backbone network=10.70.253.200/31
RouterB: /routing ospf network add area=backbone network=10.70.253.200/31
    You can verify they were added correctly with /routing ospf network print
     
    [admin@nycmesh-8300-omni] > routing ospf network print
Flags: X - disabled, I - invalid 
 #   NETWORK            AREA                                                     
 0   10.69.0.0/16       backbone                                                 
 1   10.68.0.0/16       backbone                                                 
 2   10.70.253.200/31   backbone
  6. On each router, remove the PtP interface from the Mesh Bridge. This should be done on the remote router first, as you will lose connectivity to the remote device until the local device is updated and the neighbors are re-established
    Router B: /interface bridge port set 4 disabled=yes
Router A: /interface bridge port set 0 disabled=yes
    You can verify this is set correctly with /interface bridge port print. For 0 - ether1, the "X" indicates the port is disabled
    [admin@nycmesh-8300-omni] > interface bridge port print; 
Flags: X - disabled, I - inactive, D - dynamic, H - hw-offload 
 #     INTERFACE      BRIDGE         HW  PVID PR  PATH-COST INTERNA...    HORIZON
 0 XI   ether1         mesh                  1 0x         10         10       none
 1 I   ether2         mesh           no     1 0x         10         10       none
 2 I   ether3         mesh           no     1 0x         10         10       none
 3 I   ether4         mesh           no     1 0x         10         10       none
 4 I   ether5         mesh           no     1 0x         10         10       none
 5 I   wlan1          mesh                  1 0x         10         10       none
 6 I   wlan2          mesh                  1 0x         10         10       none
 7 I   wlan4          mesh                  1 0x         10         10       none
 8 I   wlan3          wds                   1 0x         10         10       none
 9     dynamic        wds            yes    1 0x        100        100       none
  7. Once the bridge ports are disabled on both ends, the OSPF link will activate and establish adjacency; you can validate this on either router with /routing ospf neighbor print
    [admin@nycmesh-8300-omni] > /routing ospf neighbor print
 0 instance=default router-id=10.69.83.1 address=10.70.253.201 interface=ether1 
   priority=1 dr-address=0.0.0.0 backup-dr-address=0.0.0.0 state="Full" 
   state-changes=9 ls-retransmits=0 ls-requests=0 db-summaries=0 
   adjacency=6m30s

    image.png


Now we have established a 9-cost OSPF link between the two routers! This cost can be changed at will as the network topology changes in the future.

Scenario 2: Point to Multipoint with directly attached antennas

Router A > tagged > Antenna A <> Access Point B < tagged < Router B

This set up is more complicated, because we need to be able to establish a new OSPF session between Router A and RouterB without impacting its existing mesh interface neighbors that are also connected to Access Point B. Unlike the previous example, we need a way for RouterB to know to only send RouterA's traffic to the new OSPF interface, but leave all the other traffic on the default mesh interface.

This is where VLAN interfaces come into play - tagging the traffic with a VLAN ID will allow the traffic to be routed correctly. In this case, we will only be removing the physical port from the mesh bridge on Router A, detailed below

  1. Reserve an unused /31 IP range and VLAN ID to be used for the OSPF network (ask in the #architecture channel for access to the IP ranges sheet for more information on this)
    1. For this example, I will again use 10.70.253.200/31, and VLAN ID 3001
  2. To ensure we don't lose access to Antenna A's management portal later, create a VLAN interface on RouterA for management and add it to the mesh bridge; this can be any VLAN ID, as long as the same ID isn't being used for management on the other end of the link (because we don't want the two routers to see each other on this vlan). We normally use a X01 VLAN ID related to the physical port. In our case, because the antenna is on ether1, I will use 101.
    RouterA: /interface vlan add interface=ether1 name=ether1.101 vlan-id=101
RouterA: /interface bridge port add bridge=mesh interface=ether1.101

    image.png


  3. On the antenna, set the Management VLAN ID to match the VLAN you just added and save; you should retain connectivity.

    image.png


  4. On each router, create a vlan interface for VLAN 3001 on the physical port the antennas are connected to. We need to use the same VLAN on both ends whenever we traverse through an Access Point to keep this traffic identifiable on both ends of the link. Note that you will not add these interfaces to the mesh bridge.
    RouterA: /interface vlan add interface=ether1 name=ether1.3001 vlan-id=3001
RouterB: /interface vlan add interface=ether5 name=ether1.3001 vlan-id=3001

    image.png


  5. On each router, add the reserved IPs and network to the appropriate VLAN 3001 interfaces
    RouterA: /ip address add address=10.70.253.200 network=10.70.253.201 interface=ether1.3001
RouterB: /ip address add address=10.70.253.201 network=10.70.253.200 interface=ether5.3001

    You can verify they were added correctly with /ip address print

    [admin@nycmesh-8300-omni] > ip address print
Flags: X - disabled, I - invalid, D - dynamic 
 #   ADDRESS            NETWORK         INTERFACE                                
 0   10.104.27.1/26     10.104.27.0     mesh                                     
 1   10.69.83.0/16      10.69.0.0       mesh                                     
 2   10.68.83.0/16      10.68.0.0       wds                                      
 3   10.70.253.200/32   10.70.253.201   ether1.3001 
    On each router, add the vlan interface as an OSPF interface with the correct cost, set for "PTMP", and enable BFD. For this example, I will again use cost 9.
    RouterA: /routing ospf interface add cost=9 interface=ether1.3001 network-type=ptmp use-bfd=yes
RouterB: /routing ospf interface add cost=9 interface=ether5.3001 network-type=ptmp use-bfd=yes
    You can verify they were added correctly with /routing ospf interface print
    [admin@nycmesh-8300-omni] > routing ospf interface print
Flags: X - disabled, I - inactive, D - dynamic, P - passive 
 #    INTERFACE    COST PRIORITY NETWORK-TYPE   AUTHENTICATION AUTHENTICATION-KEY
 0    mesh           10        1 ptmp           none                             
 1    wds           100        1 ptmp           none                             
 2    ether1.3001     9        1 ptmp           none
  6.  On each router, add the OSPF Network so the OSPF interface is routable. 
    RouterA: /routing ospf network add area=backbone network=10.70.253.200/31
RouterB: /routing ospf network add area=backbone network=10.70.253.200/31
    You can verify they were added correctly with /routing ospf network print
     
    [admin@nycmesh-8300-omni] > routing ospf network print
Flags: X - disabled, I - invalid 
 #   NETWORK            AREA                                                     
 0   10.69.0.0/16       backbone                                                 
 1   10.68.0.0/16       backbone                                                 
 2   10.70.253.200/31   backbone
  7. At this point, the two routers should establish adjacency on the VLAN interfaces, and if the cost is <10, then traffic will prefer that route. 
    [admin@nycmesh-8300-omni] > /routing ospf neighbor print
 0 instance=default router-id=10.69.83.1 address=10.70.253.201 
   interface=ether1.3001 priority=1 dr-address=0.0.0.0 
   backup-dr-address=0.0.0.0 state="Full" state-changes=4 ls-retransmits=0 
   ls-requests=0 db-summaries=0 adjacency=47s 

 1 instance=default router-id=10.69.83.1 address=10.69.83.1 interface=mesh 
   priority=1 dr-address=0.0.0.0 backup-dr-address=0.0.0.0 state="Full" 
   state-changes=4 ls-retransmits=0 ls-requests=0 db-summaries=0 
   adjacency=11m26s 

    image.png


  8. The last step is to remove RouterA's physical port from the mesh bridge, so that traffic can only route through the new VLAN interface you have created. Note that we do not do this step on RouterB, because we want the untagged traffic from other connected antennas to continue to enter the mesh bridge.
    Router A: /interface bridge port set 0 disabled=yes
    Now, OSPF Neighbors will only show the VLAN interface and you are done!

    image.png

Implementation scenarios where a switch exists between the antennas and routers

Scenario 3: Point to Multipoint with a switch at one or both ends

Router A > tagged > Antenna A <> Access Point B < tagged < Switch B < tagged < Router B

This is one of the most common scenarios you'll encounter in the NYC Mesh network; many multi-member nodes and smaller hubs have a high-capacity 60Ghz radio with a backup 5Ghz radio to the same or a different hub for failover in heavy rain. This scenario is almost identical to Scenario 2, with one important caveat: we have to configure the switch to pass the VLAN traffic to the appropriate switch port.

  1. Reserve an unused /31 IP range and VLAN ID to be used for the OSPF network (ask in the #architecture channel for access to the IP ranges sheet for more information on this)
    1. For this example, I will again use 10.70.253.200/31, and VLAN ID 3001
  2. To ensure we don't lose access to Antenna A's management portal later, create a VLAN interface on RouterA for management and add it to the mesh bridge; this can be any VLAN ID, as long as the same ID isn't being used for management on the other end of the link. We normally use a X01 VLAN ID related to the physical port. In our case, because the antenna is on ether1, I will use 101.
    RouterA: /interface vlan add interface=ether1 name=ether1.101 vlan-id=101
RouterA: /interface bridge port add bridge=mesh interface=ether1.101

    image.png


  3. On Antenna A, set the Management VLAN ID to match the VLAN you just added and save; you should retain connectivity.

    image.png


  4. On each router, create a vlan interface for VLAN 3001 on the physical port the antenna/switch is connected to. For this example, let's assume that RouterB uses the bond1 interface to connect to the Switch.
    RouterA: /interface vlan add interface=ether1 name=ether1.3001 vlan-id=3001
RouterB: /interface vlan add interface=bond1 name=bond1.3001 vlan-id=3001

    image.png

    Note: If there is a bonded/LAG interface (where multiple SFP/ethernet ports are used for uplink), then you should put the vlan interface on the bonded interface

  5. On each router, add the reserved IPs and network to the appropriate VLAN 3001 interfaces
    RouterA: /ip address add address=10.70.253.200 network=10.70.253.201 interface=ether1.3001
RouterB: /ip address add address=10.70.253.201 network=10.70.253.200 interface=bond1.3001
  6. On each router, add the vlan interface as an OSPF interface with the correct cost, set for "PTMP", and enable BFD. For this example, I will again use cost 9.
    RouterA: /routing ospf interface add cost=9 interface=ether1.3001 network-type=ptmp use-bfd=yes
RouterB: /routing ospf interface add cost=9 interface=bond1.3001 network-type=ptmp use-bfd=yes
  7.  On each router, add the OSPF Network so the OSPF interface is routable. 
    RouterA: /routing ospf network add area=backbone network=10.70.253.200/31
RouterB: /routing ospf network add area=backbone network=10.70.253.200/31
  8. Here is where we diverge from Scenario 2: we must configure Switch B to pass the tagged traffic to the correct egress port. In this example, let's assume the trunk port to the router is also bond1, and the Access Point is on ether8 on a Mikrotik switch
    SwitchB: /interface bridge vlan add bridge=bridge tagged=ether8,bond1 vlan-ids=3001
    If not already enabled, we also need to enable vlan-filtering on the SwitchB bridge and ether8. 
    SwitchB: /interface bridge set 0 vlan-filtering=yes ether-type=0x8100 pvid=1 frame-types=admit-all
SwitchB: /interface bridge port set 7 vlan-filtering=yes ingress-filtering=yes
    If the switch is a Ubiquiti S16, you would add the VLAN ID and set the trunk and AP ports as Tagged for VLAN 3001, and exclude all other ports

    image.png

    Note: If there is also a switch between Router A and Antenna A, repeat step 8 on Switch A to allow tagged traffic to pass to the appropriate port

  9. The two routers should establish adjacency on the VLAN interfaces, and if the cost is <10, then traffic will prefer that route. 
    [admin@nycmesh-8300-omni] > /routing ospf neighbor print
 0 instance=default router-id=10.69.83.1 address=10.70.253.201 
   interface=ether1.3001 priority=1 dr-address=0.0.0.0 
   backup-dr-address=0.0.0.0 state="Full" state-changes=4 ls-retransmits=0 
   ls-requests=0 db-summaries=0 adjacency=47s 

 1 instance=default router-id=10.69.83.1 address=10.69.83.1 interface=mesh 
   priority=1 dr-address=0.0.0.0 backup-dr-address=0.0.0.0 state="Full" 
   state-changes=4 ls-retransmits=0 ls-requests=0 db-summaries=0 
   adjacency=11m26s 

    image.png


  10. The last step is to remove RouterA's physical port from the mesh bridge, so that traffic can only route through the new VLAN interface you have created. Note that we do not do this step on RouterB, because we want the untagged traffic from other connected antennas to continue to enter the mesh bridge.
    Router A: /interface bridge port set 0 disabled=yes
  11. Now, OSPF Neighbors will only show the VLAN interface and you are done!

    image.png

Scenario 4: Point to Point with switches on one end

Router A > untagged > Antenna A <> Antenna B < untagged < Switch B < tagged < Router B

This is another common scenarios you'll encounter in the NYC Mesh network, where two hubs connect to each other with dedicated antennas. Similar to Scenario 2, we have to configure the switch to pass the VLAN traffic to the appropriate switch port; but on router A, we do not need a VLAN since the antenna is plugged directly into Router A.

  1. Reserve an unused /31 IP range to be used for the OSPF network (ask in the #architecture channel for access to the IP ranges sheet for more information on this)
    1. For this example, I will again use 10.70.253.200/31
  2. Create a vlan interface with a VLAN of your choice on the physical port the switch is connected to; if there are switches on both ends of the link, do the same on the other end (the VLANs do not have to be the same, but can be if you choose). For this example, let's assume that RouterB uses the bond1 interface to connect to the Switch, and we will use VLAN 2101.
    RouterB: /interface vlan add interface=bond1 name=bond1.2101 vlan-id=2101

    image.png


    Note: If there is a bonded/LAG interface (where multiple SFP/ethernet ports are used for uplink), then you should put the vlan interface on the bonded interface

  3. On each router, add the reserved IPs and network to the appropriate physical or virtual interfaces. Because RouterA connects directly to AntennaA, we add the address to ether1; for RouterB, we use the bond1.2101 interface we just created in step 2.
    RouterA: /ip address add address=10.70.253.200 network=10.70.253.201 interface=ether1
RouterB: /ip address add address=10.70.253.201 network=10.70.253.200 interface=bond1.2101
  4. On each router, add the applicable interface as an OSPF interface with the correct cost, set for "PTMP", and enable BFD. For this example, I will again use cost 9.
    RouterA: /routing ospf interface add cost=9 interface=ether1 network-type=ptmp use-bfd=yes
RouterB: /routing ospf interface add cost=9 interface=bond1.2101 network-type=ptmp use-bfd=yes
  5.  On each router, add the OSPF Network so the OSPF interface is routable. 
    RouterA: /routing ospf network add area=backbone network=10.70.253.200/31
RouterB: /routing ospf network add area=backbone network=10.70.253.200/31
  6. Here is where we diverge from Scenario 1 & 3: we must configure Switch B to pass the tagged traffic to the correct egress port and untag it on egress. We again assume the trunk port to the router is also bond1, and AntennaB is on ether8 on a Mikrotik switch. We need to set the trunk port to RouterB as tagged, and the access port to AntennaB as untagged
    SwitchB: /interface bridge vlan add bridge=bridge tagged=bond1 untagged=ether8 vlan-ids=2101

    Additionally, we need to instruct the Mikrotik switch to tag ingress traffic with the appropriate VLAN ID. If not already set, enable vlan-filtering on the switch bridge. Then, on ether8 set the PVID to 2101, enable ingress filtering, and set the port to only allow untagged traffic on ingress (optional, but recommended for security purposes).

    SwitchB: /interface bridge set 0 vlan-filtering=yes ether-type=0x8100 pvid=1 frame-types=admit-all
SwitchB: /interface bridge port set 7 vlan-filtering=yes pvid=2101 frame-types=admit-only-untagged-and-priority-tagged ingress-filtering=yes
    If the switch is a Ubiquiti S16, you only need to set the tagged and untagged port, and exclude ether8 from whatever it is currently set as untagged for:

    image.png


  7. The last step is to remove RouterA's physical port from the mesh bridge, so that traffic can only route through the new OSPF interface you have created. Note that we do not do this step on RouterB, because we want the untagged traffic from other connected antennas to continue to enter the mesh bridge.
    Router A: /interface bridge port set 0 disabled=yes
    If, however, RouterA also had a switch in between, then you would not remove the interface from the bridge port and instead repeat step 6 on SwitchA.
  8. Now, OSPF Neighbors will only show the VLAN interface and you are done!

    image.png

Juniper Point-to-Point Guide

Note: This guide is even further in-depth than the Point-to-Point configuration guide. This is only intended for a technical audience who is looking to create an NYCMesh P2P, when one or both sides of the link is routed by a Juniper router.

Overview

To create a mesh-specific OSPF P2P link on the Juniper, there are 4 configuration changes you will need to make

  1. Create an irb interface - creating an irb interface is the Juniper equivalent of creating a MikroTik VLAN interface, or a Brocade virtual interface (ve). It's an interface with an IP address, that is assigned to a certain VLAN. The only difference is that this VLAN is not limited to a port, and can be sent out any port. This is where you will assign your /30 P2P IP address.

  2. Create a VLAN - the VLAN and the irb interface are be linked together, and the VLAN is assigned to the switch port that the P2P traffic will come from. This will either be a switchport that the antenna is directly connected to, or a switchport that goes to another switch, which is connected to the antenna.

  3. Configure OSPF on the irb - this is where you configure the OSPF cost of your P2P.

  4. Add the VLAN to a switchport - this is where you assign the VLAN to the port where P2P traffic will be coming from on the Juniper

Prerequisites

Choosing a VLAN ID

In order to create a P2P, we need to choose an unused VLAN on the Juniper. 

  1. Log into the Juniper with ssh root@ip_address
  2. Enter cli at the initial prompt to enter the switch configuration
  3. Enter show vlans and press enter. This will display a list of all the VLANs on the Juniper. Note down a tag that is unused (this can be anything between 1 and 4094, but you should keep it close to other existing VLANs)

Configuring the Juniper

  1. At the main Juniper CLI prompt (where you should be after entering show vlans above), enter configure to start configuring the router.
  2. First, we'll create the irb. Enter the following commands, replacing <P2P_NAME> with the name of your P2P link, and <IP_ADDRESS> with the IP address of the Juniper side of the link.
    1. set interfaces irb unit <VLAN_TAG> description <P2P NAME>
    2. set interfaces irb unit <VLAN_TAG> family inet address <IP_ADDRESS>/30
  3. Next, create a VLAN and link it to your irb interface, replacing variables as needed.
    1. set vlans <P2P_NAME> vlan-id <VLAN_TAG>
    2. set vlans <P2P_NAME> l3-interface irb.<VLAN_TAG>
  4. Next, configure OSPF on the interface. Note the PTMP setting, meaning the OSPF configuration is Point to Multi Point and not Point to Point, NBMA (Non-Broadcast Multiple Access), or Broadcast (more info on the different types here)
    1. set protocols ospf area 0.0.0.0 interface irb.<VLAN_TAG> interface-type p2mp
    2. set protocols ospf area 0.0.0.0 interface irb.<VLAN_TAG> metric <OSPF_COST>
  5. (Optional: if the MikroTik side of the link is using Bidirectional Forwarding Detection (aka BFD, a faster way of detecting when a link is down than the built-in OSPF method), configure that here. If you don't know, disregard these steps)
    1. set protocols ospf area 0.0.0.0 interface irb.<VLAN_TAG> bfd-liveness-detection minimum-interval 200
    2. set protocols ospf area 0.0.0.0 interface irb.<VLAN_TAG> bfd-liveness-detection multiplier 5
    3. set protocols ospf area 0.0.0.0 interface irb.<VLAN_TAG> bfd-liveness-detection full-neighbors-only
  6. Now add the VLAN to the switchport where the P2P is coming from (usually a switch)
    1. To figure out what interface goes to which switch, enter the command run show interfaces description. This will list all of the ports and their descriptions. Note the interface name (example xe-0/0/4) that the switch or antenna is connected to. Note: if the switch is connected to a bond (known as ae interfaces in Juniper) be sure to add the vlan to that port.
    2. Add the vlan with set interfaces <INTERFACE> unit 0 family ethernet-switching vlan members <P2P_NAME>
  7. Type commit to save your configuration. Once the commit succeeds, type exit to leave configuration mode.
  8. If there are upstream switches that your P2P VLAN needs to be added to, add them normally according to the guide listed in prerequisites.
  9. To confirm OSPF comes up on the Juniper, enter show ospf neighbor, and the router will give you a list of neighbors, and their connected interfaces.