Zabbix

Zabbix lives at http://zabbix.mesh.nycmesh.net

Zabbix is used primarily for historical data collection and Slack. There are a handful of dashboards configured for a few devices, but for the most part, the rest of its configuration is unused.

Data Collection

Zabbix is fed through the following sources:

• Data gathered via SNMP from various OSPF devices (mainly OmniTiks) discovered through OSPF2Zabbix
• Data forwarded from the UISP API by UISP2Zabbix

Custom Templates

We have a variety of custom templates, some of which were set up manually at one point, the rest either auto-generated or managed by one of the above tools.

Alerting

The main purpose of Zabbix is Alerting. Alerting can be found in the #zabbix-alerts channel. Alerts need to be tuned to what we really care about, such as the antennas on the larger links.

To make a trigger show up in Slack, add the slack tag to it.

The trigger can be any severity level. By default, many triggers are straight-up disabled. Alerting is, unfortunately, a manual process. We're still figuring out what is important and what isn't. 

Weekly reports of noisy triggers are published in #zabbix-reports, where the top 20 noisiest triggers are aggregated. This can help us identify problems over time.

Todos:

• There is a plan to use certain triggers to automatically switch over links. For example, we'd like to disable the AF60xr on Vernon and use a backup link when it rains.
• (Willard): I was working on a service to generate Zabbix templates from MIB files using the Zabbix API. I'd like to tailor it towards specific Ubiquiti devices and use it + the UISP API to discover compatible antennas and use the SNMP data to enrich our DataLink data.
• (Willard): Expand UISP2Zabbix to cover more than just DataLinks. It would be cool to get all kinds of data out of it and into Zabbix for analysis
• (Willard): Problem heatmap. If I could overlay problems on top of Andrew's Node Explorer, we could see problem areas within the mesh.
• (Willard): Integrate Grafana with Zabbix. I know this is possible, the question is what's the best way to do this? And, if we're primarily doing this for UISP, then why not build something that integrates with UISP? (Couldn't be that hard to just query UISP's database directly, right?)

More Info

For (outdated-ish) information on how this was set up, including how Slack alerting was configured, refer to this doc: https://docs.google.com/document/d/1mJI8DWe882P6GCEGdT0xazxwrrCQZD7qEBcsDEjDU7Q/edit?usp=sharing