Skip to main content

Adding New Devices to Grafana

Most prosumer to professional networking equipment utilizes SNMP in order to pull statistics (and in some cases, set configuration) into a centralized system such as LibreNMS or Grafana.

Our statistics stack for non-Ubiquiti equipment consists of Grafana, ingesting data from Prometheus, ingesting data from SNMP Exporter (it also has other modules, such as NUT Exporter for UPSes, Node Exporter for Linux boxes, etc.).

  • Server IP: 10.70.90.142
  • Prometheus: 10.70.90.142:9090
  • SNMP Exporter: 10.70.90.142:9116

In order to add new devices to the statistics stack, the device must be added to Prometheus for the records to be available for Grafana to be graphed.

Adding Devices to Prometheus for Existing Device Types

As described above, each of the component scraping services (not Grafana) lives on the same box. All of the services run as a systemd service. To access the server and the configurations:

  1. SSH into the Prometheus server (see above)
  2. cd into the /opt folder (where the binaries and configurations live)
  3. cd into the prometheus folder, and edit the prometheus.yml file.

In the file, there are several "scrape modules", which have specific parameters with a job_name. Each module has a static_configs section containing multiple targets, which is where new devices will be added.

For example, for Cambium 6GHz ePMP devices, you would add additional targets to its respective block:

  - job_name: 'snmp cambium_epmp'
    scrape_interval: 30s
    scrape_timeout: 20s
    static_configs:
      - targets: ["10.97.98.118"]
        labels:
          hostname: "nycmesh-1417-force6g-1340"
# the new target to be added
      - targets: ["10.12.34.56"]
        labels:
          hostname: "nycmesh-123456-newdevice"

Once the file is edited and saved, systemctl restart prometheus will restart the service, and after a couple of minutes, the Prometheus web interface will show the new target on its Targets page.

image.png

This device, and all of its parameters, will now be visible by its hostname label—much preferred than requiring a specific IP address to be referenced in Grafana.

image.png

Adding Custom MIBs to SNMP Exporter

If a brand-new device is being added to the statistics stack for the first time, the existing SNMP configuration may not support ingesting this information via SNMP. The SNMP Exporter's generator tool will aid in this objective.

  1. Clone the repository and follow the above linked instructions to build it
  2. Get your MIB file and put it in the mibs subfolder
  3. Edit the generator.yml file to have some important parameters*
  4. Run this generator command: ./generator generate -g ./generator.yml -o /tmp/snmp.yml 

*What are the important parameters?

auths:
  auth_name:
    version: 2  # SNMP version to use. Defaults to 2.
                # 1 will use GETNEXT, 2 and 3 use GETBULK.

    # Community string is used with SNMP v1 and v2. Defaults to "public".
    community: nycmeshnet

modules:
  cambium_epmp:  # The module name. You can have as many modules as you want.
    # List of OIDs to walk. Can also be SNMP object names or specific instances.
    # Object names can be fully-qualified with the MIB name separated by `::`.
    walk:
      [pmpMibTree]

    max_repetitions: 25  # How many objects to request with GET/GETBULK, defaults to 25.
                         # May need to be reduced for buggy devices.
    retries: 3   # How many times to retry a failed request, defaults to 3.
    timeout: 5s  # Timeout for each individual SNMP request, defaults to 5s.

    allow_nonincreasing_oids: false # Do not check whether the returned OIDs are increasing, defaults to false
                                    # Some agents return OIDs out of order, but can complete the walk anyway.
                                    # -Cc option of NetSNMP

    use_unconnected_udp_socket: false # Use a unconnected udp socket, defaults to false
                                      # Some multi-homed network gear isn't smart enough to send SNMP responses
                                      # from the address it received the requests on. To work around that,
                                      # we can open unconnected UDP socket and use sendto/recvfrom
  • community: this is the SNMP community string that the class of devices will use. Not that important as we typically use public and can be changed later.
  • modules -> cambium_epmp: this is user configurable to the string you will reference in Prometheus. Make sure it is unique and something you will be happy with to reference for that class of devices forever.
  • walk: this is very important. To "walk" all of the OIDs (effectively the SNMP codes) in your MIB, it needs to match the MODULE-IDENTITY line (notes on this)

One you have your /tmp/snmp.yml file, you now need to incorporate this into your SNMP Exporter configuration.

  1. cd into the /opt folder (where the binaries and configurations live)
  2. cd into the snmp_exporter folder
  3. Append the contents of the /tmp/snmp.yml file into the snmp.yml file by running /tmp/snmp.yml >> snmp.yml
  4. Edit the snmp.yml file to remove the duplicate auths and modules blocks, up to where your new module is defined:
# delete all this
auths:
  public_v1:
    community: public
    security_level: noAuthNoPriv
    auth_protocol: MD5
    priv_protocol: DES
    version: 1
  public_v2:
    community: public
    security_level: noAuthNoPriv
    auth_protocol: MD5
    priv_protocol: DES
    version: 2
modules:
# to this
  apcups:
    walk:

If the command does not work, and says something along the lines of cannot find oid, run this command and see if your configuration is missing any recursive MIBs. You can download and insert them from LibreNMS' collection here.

root@nycmesh-713-02-prom1:~/snmp_exporter/generator# snmptranslate -mALL -L o | grep "Cannot find module"
Cannot find module (SNMPv2-SMI): At line 34 in /usr/share/snmp/mibs/UCD-SNMP-MIB.txt
Cannot find module (HCNUM-TC): At line 37 in /usr/share/snmp/mibs/UCD-SNMP-MIB.txt
Cannot find module (SNMPv2-TC): At line 40 in /usr/share/snmp/mibs/UCD-SNMP-MIB.txt
Cannot find module (SNMPv2-TC): At line 7 in /usr/share/snmp/mibs/UCD-IPFWACC-MIB.txt
Cannot find module (SNMPv2-TC): At line 10 in /usr/share/snmp/mibs/UCD-DLMOD-MIB.txt
Cannot find module (SNMPv2-TC): At line 15 in /usr/share/snmp/mibs/UCD-DISKIO-MIB.txt
Cannot find module (SNMP-FRAMEWORK-MIB): At line 9 in /usr/share/snmp/mibs/NET-SNMP-VACM-MIB.txt
Cannot find module (SNMPv2-SMI): At line 8 in /usr/share/snmp/mibs/NET-SNMP-MIB.txt
Cannot find module (SNMP-VIEW-BASED-ACM-MIB): At line 16 in /usr/share/snmp/mibs/NET-SNMP-VACM-MIB.txt
Cannot find module (SNMPv2-TC): At line 25 in /usr/share/snmp/mibs/NET-SNMP-VACM-MIB.txt
Cannot find module (SNMP-FRAMEWORK-MIB): At line 10 in /usr/share/snmp/mibs/NET-SNMP-PASS-MIB.txt
Cannot find module (SNMP-FRAMEWORK-MIB): At line 10 in /usr/share/snmp/mibs/NET-SNMP-EXAMPLES-MIB.txt
Cannot find module (SNMPv2-TC): At line 12 in /usr/share/snmp/mibs/NET-SNMP-EXAMPLES-MIB.txt
Cannot find module (INET-ADDRESS-MIB): At line 13 in /usr/share/snmp/mibs/NET-SNMP-EXAMPLES-MIB.txt
Cannot find module (SNMP-FRAMEWORK-MIB): At line 9 in /usr/share/snmp/mibs/NET-SNMP-AGENT-MIB.txt
Cannot find module (SNMPv2-TC): At line 21 in /usr/share/snmp/mibs/NET-SNMP-AGENT-MIB.txt
Cannot find module (SNMPv2-TC): At line 15 in /usr/share/snmp/mibs/LM-SENSORS-MIB.txt
Cannot find module (SNMPv2-SMI): At line 8 in /root/.snmp/mibs/cambium-epmp-5.10.0-mib.txt
Cannot find module (SNMPv2-TC): At line 12 in /root/.snmp/mibs/cambium-epmp-5.10.0-mib.txt
Cannot find module (IPV6-TC): At line 14 in /root/.snmp/mibs/cambium-epmp-5.10.0-mib.txt

No comments to display

Back to top